Search Engine Overlap and Divergence

A lot of people like to think search engines aren’t that different. They index the same internet, and they return mostly the same results, right? Not at all. A new study shows once again that search engines index mostly unique pages and rank common ones completely differently. This should carry a heavy message for optimizers, and a smaller one for searchers.

Pennsylvania State University and Dogpile.com studied the overlap in results that the four major search engines (Google, Yahoo!, MSN Search, and Ask Jeeves) provided.

The study used 485,460 unique searches in July 2005. The researchers only looked at the total results and found this information:

  • 84.9% results were unique to one search engine.
  • 11.4% results were shared by any two search engines.
  • 02.6% results were shared by three search engines.
  • 01.1% results were shared by the top four search engines.

It’s actually amazing how little agreement there is between search engines. It has a lot of implications. Primarily, people should be concerned because searchers and marketers can miss out on huge market segments. By focusing on only one search engine, you can miss a lot.

  • Google misses 70.8% of first page results on other engines.
  • Yahoo! misses 69.4% of first page results on other engines.
  • MSN Search misses 72.0% of first page results on other engines.
  • Ask Jeeves misses 67.9% of first page results on other engines.

This shows how completely different search engines are. Each has its own personality when it comes to evaluating and delivering the web’s most important content. Relevancy of results is also questioned with a study by comScore Media Metrix, which found that 31 to 56% of searches on these top search engines elicit a first page click. If half (or more) of searchers are not clicking on a first page result, did the search engine deliver irrelevant or useless results?

The study also found that, “Only 4.7% of Yahoo! and Google sponsored links overlap for a given query.” This means that companies paying to advertise are only considering one search engine or another. As we will take a look at on the following page, this kind of marketing is short sighted and misses huge numbers of potential customers.

For the source of all this information, and to read more interesting details, drop by Dogpile’s 30 page PDF at http://comparesearchengines.dogpile.com/OverlapAnalysis. It’s an interesting read, though mostly written to show that searchers may find Dogpile useful.

To searchers, the results simply mean if you didn’t find what you are looking for in the first few pages or SERPs move to another engine rather than digging down into lower less-relevant pages. This does not mean meta searches are useful, as you will read on following pages. Meta searches may eventually become much more useful gives some development.

To optimizers, the news should mean more. You should already know that not everyone uses Google. Studies show users spread out over many search engines. Just over a third of searches are Googled, 35%. Yahoo! accounts for 28% of searches. AOL and MSN follow with 16% and 15% respectively. All others make up 6% of the search market. (figures from SearchEngineWatch.com)

By only optimizing for Google, you miss out on 65% of the search market. That’s not a pretty thing to think about. That’s the majority of web traffic. The search engines don’t really cater to definitive demographics, so demographics shouldn’t even be an excuse.

Still shouldn’t optimizing a page for one search engine show all sear engines it is relevant? Are search engines flawed? The purpose of a search engine is to deliver the most relevant content on the internet. To any one particular search, this means there are 10 sites out there that should end up at the top of the results pages, right? Well, it seems none of the search engines really agree on top 10s, and don’t often have much consensus on top 100s either. So does this mean search engines have failed to deliver the right content to us?

As some of the top search engines prove, there can be multiple right answers. To begin, all search engines have different sites indexed that can be searched through. The portion of the internet they scour differs, though it also overlaps in places. They are pulling from different databases, but they are also pulling with different search algorithms. The algorithms are the “brains” of the search that decide what kinds of results a search should receive. How algorithms determine what information you want and provide it is a long discussion for another article. Suffice it to say, even different search results can provide equally relevant sites. Also, when some users search “mustang,” they may expect a car or construction equipment or a horse; search engines can only guess.

In publishing the results, Dogpile is doubtlessly trying to show the usefulness of meta search. Their service compiles the top results of the four major search engines (Google, Yahoo!, MSN Search, and Ask Jeeves) into one result set, ranking overlapping results higher and dropping unique ones to the bottom. It sounds like a good idea. So is Dogpile a superior service for finding the most relevant sites?

Hardly. First, picking crossover results and compiling top ranking sites will not automatically deliver a better result set than any singular search. It doesn’t actually mean the search is displaying popular links for searchers, just ones that the different program algorithms like rather well. It will treat those that play the SEO game especially well. It still has a selective first page of results, and potentially relevant links still get shoved down to subsequent pages. In fact, with giving favor to sites that all engines agree on, the area of the internet that you are effectively searching decreases in meta search to only what is most indexed. Trusting a compilation of all search algorithms is not better than trusting one.

Dogpile did claim they delivered more relevant results, though. The study used click-throughs to gauge result relevance of results, showing:

  • 62.9% of Dogpile searchers clicked something on the first page
  • 55.6% of Google searchers clicked something on the first page
  • 50.0% of Yahoo! searchers clicked something on the first page

This actually speaks very poorly of their results when you consider Dogpile displays 20 results per page and all the other searches show 10. They would rank far more poorly if their front page was half the size, like Google’s and Yahoo’s. Scientific results need fewer variables, and this does not succeed. This also doesn’t even address how click throughs are not indicative of relevance, just the appearance of the SERPs.

Moreover, the idea of meta search is by no means a holy grail, but it may prove to be useful if Dogpile can iron out problems. Dogpile seems to have a problem searching for phrases. Phrases are very important to a targeted search, and Dogpile results from phrase searches sometimes lacked any semblance to the search criteria.

Perhaps the site’s biggest sin is mixing sponsored links into the organic results, something other search engines rightfully avoid. Dogpile should take a hint from Google and Yahoo! in this regard and display them separately. Thankfully they did mark them in light print as sponsored, but it will be hard to claim relevancy when combining organic results with those that paid to be there.

Because of the low amount of pages common to all engines, this study still shows the difficulties in optimizing a site for multiple search engines. It does not prove meta search is superior, but it does show SEOs that you need ways to gauge how well your site is doing across the board.

One very interesting tool that allows anyone to see a visual mapping of search engine overlap is Thumbshot Ranking (http://ranking.thumbshots.com). You can compare the top 100 results in Google, Yahoo!, MSN, Altavista, Teoma, Alltheweb, and Wisenut. It displays the results as one horizontal line of dots for each search, and the dots are connected by a blue line if they are identical. The site also lets you highlight a site of particular interest, perhaps your own. It shows all highlighted results and their links in red.

To illustrate the variation in search results, I did searches on the service. I compared Google and Yahoo! result sets for the most part. The first was a search for the term “seo,” and I set the search to highlight all links to seochat.com in red. Here are the results:

The dots on the far left are the top ranking results, and they decent as they move to the right. Since the left dot on the top row (Google) is red, you know SEO Chat is in Google’s top rank for that term. It didn’t do quite so well on Yahoo!, but still on the first page of SERPs. While the site may rank as number 1 on Google, it actually populates more of the higher results on Yahoo!. If you are optimizing, either is a good goal.

The web program also tells you how much overlap there are in those top 100 results. For the “seo” search, we saw 20% of the results were the same. That means the vast majority of results (80%) are unique to each search engine.

You can be sure “seo” is a keyword that many websites are fighting for. Most people would rather trust an SEO site or service that ranked well, since rank is their job. My point is that despite tons of optimization and competition, the search engines picked mostly different top results. Optimizing for one engine will not suffice, and different techniques may only add to ranking on one search or another.

As a side note to further demonstrate how Dogpile is not yet useful, let’s see this site’s rank on their engine. SEO Chat is on the first page of Google, Yahoo!, and MSN Search for the keyword “seo;” it appears in rank 12 for Ask Jeeves, the second page of results. It is common and highly ranked on all Dogpile sources, yet the site appears all the way down in Dogpile’s 69th rank (as of 8/15/05). Even if every other page in the top 12 of all these search engines ranked better than SEO Chat, that would only put it at rank #44. The meta search simply fails at this stage of its development, and it is not functional for search engine users. Optimizing for it may not be entirely predictable either.

I tried many experimental searches, and one with high overlap was “browser.” Even though it was relatively high, the overlap was only 40%. Sixty percent of the results were still unique. That’s a lot; ranking in Google’s top 100 for this keyword gives your site only a 40% chance of showing up on Yahoo!.

For the fun of it, I made the results from Microsoft red. It looks like the guys who dominate the browser market don’t care about search rank. They were beaten by Netscape, Opera, Firefox, and Safari. No Internet Explorer page was even ranked well enough to show up consistently between the search engines; you’ll notice no lines connect the red dots, showing they are all unique. At least Microsoft can celebrate having beaten the Lynx browser.

The point of this is that using Thumbshot Ranking will help show strengths and weaknesses in search optimization. While most sites wouldn’t mind being on the first and second pages of Yahoo and Google, this is Microsoft. The most popular software manufacturer and browser developer has been shoved away by better sites.

Poorly ranking websites need to take hints from higher ranking sites. In particular, use Thumbshot to easily find sites that rank highly across all search engines. Take these sites; check out their code, layout, and URL structure. Find patterns and optimize with your competitors’ help. To see what kind of back-links they have gotten, you can always use a “link:” query in most search engines. This will show how much back-linking contributes to their rank, as well as showing useful sites you can try to get back-links from. You may also want to note what techniques these sites avoid.

It may be a headache to think about Google, Yahoo!, MSN Search, AOL, Altavista, and Ask Jeeves. But only optimizing for Google is a mistake. Tuning your site to rank highly on any one search engine guarantees you nothing on another. All SERPS and optimization methods are not created equal, and as much you hate to think it, the other search engines do have an audience. Only a third of searches are carried out on Google. I, for one, change search engines by mood and level of frustration.

Google+ Comments

Google+ Comments