Search Engines and Algorithms: MetaSearch Engines, A Tool for SEO

In the previous articles, we looked at the algorithms of a few top search engines. In this article, we are going to explore specialized search engines: MetaSearches. Keep reading to learn how they can help you improve the efficiency of your SEO.

In search engine optimization, it is inefficient and not very wise to only optimize your site for one search engine, as you will likely miss almost half or more of the search queries that arise in the course of a day. Cross comparing search engine results, however can become very tedious. If you were to compare results from Google, Yahoo, and MSN alone, you can easily triple the amount of work you will have to do for SEO. MetaSearches can help consolidate your work in this area.

What are MetaSearch Engines?

A MetaSearch Engine is a type of search tool that will allow a user to conduct a search across 2 or more search engines and directories in one step. These tools do not index sites and compile their own databases; instead they are programmed to search across many of the top search tools simultaneously.  MetaSearches provide a quick way of finding out which engines are retrieving the best results for you in your search.

How MetaSearch Engines Work

Most MetaSearch Engines display multiple-engine search results in a single merged list, from which duplicate entries have been removed; although some do not collate search results but display them instead in separate lists as they are received from each engine. Duplicate entries may appear.  Every MetaSearch Engine has its own method of determining relevancy, so it’s difficult to lump them all into a general category of how they operate, although they follow some of the same basic principles.

MetaSearch engines don’t have their own database of web pages, they merely act like a middle agent between you and other search engines. Once you enter your search query into a MetaSearch engine, they then pass this on to the major search engines and provide you with results from all the databases surveyed quickly. Because the major search engines often produce very different results, MetaSearches are especially useful when you are short on time and want to gain a general overview of the information available. 

Types of MetaSearches

There are a few different types of MetaSearch Engines, and they all work a bit differently.  Here, we will examine some of the types of MetaSearches, and look at some examples of each type.

1.  The Raw MetaSearch Engine

This MetaSearch engine presents raw results from queried engines, presented in separate groups.  Usually, in this type of MetaSearch engine, no results are being stored in a database; rather, results are simply being scraped from other top search engines, like Google, Yahoo, MSN, AOL Search, and AskJeeves.

Dogpile used to be a good example of the Raw MetaSearch engine, although it has been newly remodeled to become more of the next type of search: database MetaSearch.  Mamma.com is also another Raw MetaSearch.  In showing results side by side, it calculates site popularity from duplicates it returns.  Even though it eliminates the duplicate entries, it counts the duplicates as “votes” for the results, then placing the site with the most duplicates at the top of its SERPs.  Mamma.com is one of the oldest MetaSearches known to the Internet.

2.  Database MetaSearch Engine

This is a type of search engine where results are processed, filtered and merged together, kept in a database of the MetaSearch Engine, either textual or virtual.

MetaCrawler is a good example of this type of MetaSearch engine. MetaCrawler is one of the oldest search engines, began in July 1995 at the University of Washington. MetaCrawler was purchased by InfoSpace, an online content provider, in Feb. 1997.  Dogpile is now considered a Database MetaSearch engine, as it merges the results of its scans, and keeps them in a virtual database.

3.  The Classification Engine

A Classification MetaSearch Engine is one that provides results that are processed and automatically categorized into specific categories.  

Vivisimo has an example of a Classification Engine called Clusty.  It uses the term “Cluster Engine”.  Vivisimo tells us that the problem of not being able to find information in a search query has now been replaced by the presence of too much information to sort through.  They claim “…categorization of all this data has been the obvious solution to enable users to better deal with this ‘information overload’. Traditional approaches to categorization involving taxonomies, however, are too expensive, time-consuming and complex for most organizations.”  Clustering enables users to have the ability to view categorized results without the time and expense of taxonomy building.

Infonetware RealTerm Search, another Classification Engine, is primarily designed to demonstrate its “Infogistics” classification technology. While it’s a meta search engine, and it does topical categorization of results, like Vivisimo; it is unique in that you can select several different topics, then choose to see results from all of them, rather than being restricted to the results from only one topic. 

4.  The Semantics Engine

The Semantics MetaSearch Engine is one that attempts to make sense of the results of a search query in a contextual format.  One could argue that a Semantics Engine is nothing more than a complicated version of a Classification Engine.  Currently, the use of the sense engine is being used to provide advertising, but eventually we’d like to see this concept used in a free search engine. 

Textonomy Advance, patented by Crystal Semantics, is one of the few Semantics Engines to date.  This search tool “…operates by applying human “senses” and concepts that current algorithms, semantic systems and other statistical techniques cannot match. Carefully built by human experts, Crystal Semantics’ unique semantic network provides an understanding of the “senses” of words and terms and the true linguistic relationship between them,” as they say on their website.

5.  The Bias MetaSearch

The Bias MetaSearch is an engine that can skew results based on importance of a particular search engine by presenting bias points or values.

Myriad Search is a prime, and maybe the only, example of a Bias MetaSearch.  Myriad allows you to compare results from the four top web search engines: Google, MSN, Yahoo, and AskJeeves; then tweak their relative importance in the overall results by adding to or subtracting bias values.  It is designed primarily to help search engine optimizers to view competitive intelligence research on keywords. While this is similar to what you’d see with other meta search engines, Myriad Search also displays the result description for each page found, sums up the “authority” value of a web page as determined by its position in all of the search engines, and votes accordingly. By default, #1 ranking pages get a score of 10; if all four search engines checked by Myriad Search return #1 rankings for a page it gets a score of 40, and an authority value of 100%. Its simple algorithm gives the SEO an interesting perspective on the relative importance that search engines assign to web pages, and possibly their relevance, as well.  You can also override the search engine algorithms by adding or subtracting bias points that will cause results from a particular engine to become more or less important in the results pages.  Because this is fairly new technology, and no one knows exactly how each search engines algorithms weighs individual factors, it may require some playing around to come up with the results you need.

6.  The Specialty MetaSearch

Specialty MetaSearch engines are types of MetaSearches geared towards a particular result, such as news, documents, images, digital media, or topics.

The prototype, AllInOneNews, is a News MetaSearch engine.  It only compiles news in its return of results for a particular search query.  It’s currently connected to about 50 news servers, but anticipates that number to become in the thousands fairly soon.  Because it is on a standard commercial server, its response time is very slow, but I have achieved fairly relevant results with this MetaSearch Engine.

7. Visualization MetaSearches

These MetaSearch engines compile results in such a way as to provide a unique experience to a search.  Many can compile the results into Flash movies, maps, or give you previews of the webpage with the relevant keywords.

SurfWax is a MetaSearch Engine that gives you visual results, by allowing you to preview your relevant site matches and the web page to see exactly where in the page your keyword phrase is listed.  Another example of a Visualization MetaSearch is KartOO.  KartOO uses Flash to draw interactive maps, and to show how websites are interconnected with keywords.  Other similar MetaSearch Engines are Jux2 and Ujiko.

8.  All In One MetaSearch Engines

Unlike the other MetaSearch Engines, All-in-one searches allow you to query any number of search engines from one page, individually.  While this technically isn’t a true form of the MetaSearch, these types of engines do allow a more concise search pattern without having to open multiple browsers, as you can do your searches from one page.

Proteus demonstrates this searching ability.  It lets you easily send your search to any one of several search engines, while also linking to that search engine’s help pages.

Other MetaSearch Engines

There is probably not enough room here in this article to give you a definitive list of all the MetaSearch Engines, and I’ll probably accidentally leave out your favorites, but here is a brief list of some of them, and what they do.  I wanted to list those that have unique features or attributes.

  • ProFusion.com – Uses over 1,000 specialized search engines to pull results from, including from the “Invisible Web”
  • ZapMeta.com – Provides interesting visualization tools and other features, by giving you the option of previewing the site before visiting
  • Family Friendly Search – geared toward kid-oriented searches, and returns results based on major kid-friendly search engines and directories
  • Search.com – One of the first specialized search engine guides
  • Excite – Formerly a crawler-based search engine, Excite was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but maintains its own portal features.
  • CurryGuide.com – Has ability to save your results for easy reference at a future point
  • Pandia.com – Has the option to search directories in addition to search engines
  • Ixquick.com – Ranks results based on the number of “top 10″ rankings a site receives from the various search engines
  • iZito.com – Allows you to park your listings to customize your results; also offers the ability to show listings in up to three columns across the screen, letting you see more results at once
  • Meceoo – Has the ability to create an “exclusion list” to block pages from particular web sites being included
  • Webcrawler – Offers fast and ad-free interface
  • Fazzle – Formerly called SearchOnline; offers a highly flexible and customizable interface to a wide variety of information sources, ranging from general web results to specialized search resources in a number of specific categories.
  • Info.com – Offers shopping, news, eBay, audio and video search, as well as a number of other interesting features, compiled from 14 various search engines and pay-per-click directories.
  • Copernic – A PC based search utility

There are so many other MetaSearch Engines, but hopefully you have a good idea of what kind of options are available to you for search.

Using MetaSearches for SEO

Just as a website designer knows that developing a website to be compatible across multiple browsers is a good idea, so in the same way does an SEO know that cross referencing search engines for the median results for keyword phrases is vital to doing their job correctly.  It doesn’t really matter which MetaSearch Engine you use in your cross referencing, but it does matter that you are optimizing for multiple search engines.  MetaSearch Engines are good tools to use to accomplish this task.  There are many reasons for using MetaSearches in your SEO efforts.  While this list is not all-inclusive by any stretch, you will probably see some overlap in the following points.

1.  Varying Results

While it doesn’t matter which engine you use, I would recommend using a Raw MetaSearch Engine for this task, since you will see the unadulterated results a person employing a search query would see. No two search engines return the exact same results of any search query.  When you see the information available in SERPs across several search engine platforms can be an effective teaching tool to help you learn about the individual search engines, and the way they perform in the course of a search query.

A study by the University of Pittsburg, Pennsylvania State University and Dogpile.com took a look at how diverse search engine results actually are.  In an article written by Mike McEwan, entitled “Search Engine Overlap and Divergence”, he references this study.  The results indicate that 1.1% of results for a search query are shared by the top four search engines (Google, Yahoo!, MSN Search, and AskJeeves).

A similar study in May of this 2005 showed still a different figure.  With a sample of more than 10,000 queries, 86% returned a different #1 ranked algorithmic result in each search engine, and 32% of these queries did not return any overlapping results in the top three algorithmic positions.  So in a few months time, search engine results have diversified greatly, and show no signs of stopping the trend.

Using MetaSearch Engines to give you an overview of your search query can better help you optimize your website for multiple search engines.

2.  Keywords and Key Phrases

Another good reason to use MetaSearches in your SEO is when doing keyword research and analysis.  Choosing keywords for your website is sometimes a time consuming task, but being able to have an overview of potential results in the information available over several search engines is highly useful. 

3.  Becoming Familiar with Individual Search Engines

Being able to see how different search queries are handled over multiple engines is a great way to not only decide what keywords you should be using, but it is also helpful to see what kind of competition you will have.  Because no two search engines are alike, as every search engine has its own algorithms, it is important to know how each search engine operates.  For an SEO, this is a full time job; especially since search engines are changing constantly.  By using MetaSearch Engines, you can help reduce the monumental task of learning about each search engine’s behaviors.

4.  Minimizing Your Work Load

As I mentioned before, cross comparing search results in multiple search engines can be time consuming.  MetaSearch Engines can help you by compiling the results of multiple search engines into one query, versus having to do the same search over and over in different search engines. 

The Disadvantages of MetaSearch Results

Unfortunately, some MetaSearch engines seem to be casting smaller nets by relying on subject directories and pay-per-click engines for their web results. I, for one, do not like searches that included sponsored listings.  Anyone can pay for a search keyword to produce results.  Take Google’s AdWords Program for instance.  If I wanted to utilize the Pay-Per-Click program for my SEO website, but decided I wanted to have my ad appear for the word football, there is nothing to stop me from doing so.  So including sponsored listings in with organic search results seems to propagate irrelevant search results far more than it would help curb them.

Many MetaSearch engines don’t offer advanced searches, or the variety of search options that individual search engines do. When you initiate a keyword or phrase search on a MetaSearch engine, you are usually at its mercy as far as how the search is configured and conducted.  I should mention, however, that more and more MetaSearch engines are adding advanced search options.  Dogpile, for example, now includes search options like word exclusion, language filtering, date range, and more.

When a MetaSearch query is made, the MetaSearch engine usually takes into account a large number of the popular search engines.  Although MetaSearch engines query a number of individual search engines, not enough search Google, one of the largest and most popular search engines on the Web.  Part of this problem could be related to Google’s requirement of an API key.  Anyone can apply and receive Google’s API key, but you are limited to 1000 searches per day.  To apply for a commercial API, I am sure it is quite pricey.  Myriad Search, one of the Bias Search Engines I referred to above, give you the option of entering your own API key.  This allows you to include Google’s results along with the results from the other individual search engines.  Otherwise, even though the MetaSearch engine includes Google as one of the search engines it compiles results from, unless you have an API key, it always returns no results from Google.

Conclusion

Regardless of MetaSearch Engines’ drawbacks, overall, they are still a good way to perform cross search engine referencing in your SEO efforts.  The upside is they are constantly seeking to better themselves overall in their usefulness.  Familiarizing yourself with the search engines and the way they display results can be a huge undertaking unless you utilize the tools at your disposal, like MetaSearches.  With the many diverse search engines, being able to compare results from two, three, or even more of them at a time, you are well on your way to performing efficient SEO.  By effectively using MetaSearches for cross-platform performance, you have one of the means to minimize your workload and the amount of research it takes for good SEO.

Google+ Comments

Google+ Comments