Google`s Quality Rater Guidelines Leaked

Someone at Google has leaked quality rater guidelines. The document is full of information that is highly desirable to anyone trying to rank a site high in Google’s results pages. This article will take a close look at the guidelines and cull the most important points for your use.

Document Review

The document circulated on private forums but was made public by Search Engine Land.

This document details how Google quality raters score pages, treat spam and classify search queries. It has screen shots of Google’s “raters hubs” and forums, some of which we’ll feature here.

This is a review of the most crucial and useful points from this document. If you are in the SEO game I highly recommend that you download this file, put it into a safe place and read it at your own pace.

Raters only rate “pages” and not entire domains. The document refers to “landing pages” as opposed to domains. If a page is excluded by a rater, it is only excluded for a specific query, not for all queries. There are people like Matt Cutts who analyze sites on the domain level, but Quality Raters of the lowest ranks do not have this responsibility (at least, that’s what I learned from the document).

Here’s an interesting leaked video of the inside of the raters hub. http://www.searchbistro.com/secretlab.html

Importance of Country to Relevance

This is a key factor in Google’s scoring system. Throughout the entire document, raters are presented with example queries that specify: [query], language (country).

For example “[camera], English (US)” or “[camera], Russian (Russia).”

If the page in question provides relevant information to the query, but isn’t relevant to the country the user is in, then the page may be discounted.

You will be given a Task language and a Task location for each query-page Task. You must evaluate each Task in the context of its language and location.

In many cases, when there is a mismatch between the Task Location and the country domain of the page, you will need to lower the rating for the page.

Raters are advised to understand the query:

You should understand each query before you evaluate it. If the meaning of the query is unclear, you will need to do web research to learn about it. You can do this by entering the query in the search box of one or more search engines and looking at the results returned by them.

This implies that Google invests heavily in understanding a searcher’s intention instead of matching words and phrases on websites. There are definitely a number of algorithms tasked with this. Here’s more proof:

Sometimes you may feel unsure which of two ratings to give: Relevant or Useful? Not Relevant or Relevant? When you are unsure, select the lower rating.




The types of queries are not as important and are relatively known, so we’ll spend just a few seconds covering each.

Navigational

This is a basic request, such as ebay or amazon.com. Relevant results are official websites.

Informational:

The goal is to find information and read something on the topic.

Transactional:

The user is looking to buy something, download or interact with the resource.

Rating Scale

This is a revealing section. Google classifies pages as:

  • Vital

  • Useful

  • Relevant

  • Non Relevant

  • Off topic


Vital

Query: The query has a dominant interpretation. The dominant interpretation is navigational.

Page: The page to evaluate is the official web page of the query.

Examples:


Query

Vital

Vital Page

Description

Singapore airport

Yes

http://www.changi.airport.com.sg/ 

Official home page

toyota camry

Yes

http://www.toyota.com/camry/index.html

Official product page on the correct site

apple

Yes

http://www.apple.com/

The dominant interpretation of this is Apple Computer, Inc., and this is the official home page.


Vital queries must have an official site in the search results. Whenever you search for a celebrity name, Google tries to show the official website as a “Vital result.” Think of vital queries as brand searches.

As a webmaster, I wouldn’t be concerned about vital queries unless I had a strong brand name. In that case it is important to fall in into “Vital” category for “brand” searches (by brand search I also refer to names like “Toronto Pearson Airport” or “Tom Cruise”).

Official blogs such as those on MySpace and Face book of individuals (like Britney) are also considered vital.

Useful

I believe Google builds 80 percent of its search results with this measurement:

A rating of Useful is assigned to pages that contain some or all of the following characteristics: highly satisfying, comprehensive, high in quality, and authoritative. Useful pages answer the query just right; they are neither too broad nor too specific.

The document also lists types of useful pages:

  • a page that is highly informative.

  • a timely and informative article.

  • a page that allows the user to complete the intended transaction.

  • an important subpage on a correct site.

  • a home page of a correct site for a specific product. If the query “asks” for a list, then a directory (a collection of links) can be Useful, e.g. [ fudge recipes ], [ books about sharks ].

A search engine optimization campaign should aim towards getting pages classified as “useful.”

We can come to a conclusion with which you’re already familiar: ”Content is King.” I personally spend a lot of time developing the best content in the industry for personal and internal projects, aiming to be classified as:

  • “a page that is highly informative”

  • “a page that allows the user to complete the intended transaction"

The duo listed above forms a powerful alliance and makes both Google and users happy. A good example is Aaron’s article on search engine algorithms.

Relevant

This rating is lower than useful and doesn’t get as many kudos from raters as a “useful” page does.

A rating of Relevant is assigned to pages that have fewer valuable attributes than were listed for Useful pages. Relevant pages might be less comprehensive, come from a less authoritative source, or cover only one important aspect of the query.

If you read blogs and SEO articles, you are probably familiar with the words: “Provide useful and relevant content to users.” This is what most decent websites do, and I believe this is not enough to set you apart. I come back often to the wonderful examples by SEOmoz, wiep.net, and Aaron Wall, who kill in the area of “useful and relevant content,” providing super industry standard articles. This is the content that gathers links.

Relevant content is easy to produce, so aim higher.

Sometimes you’ll be classified as "merely" relevant because there are few sources available and you’re the closest thing. I wouldn’t worry about those.

Non Relevant / Off Topic / Didn’t Load / Foreign Language

This is common sense. Pages that don’t load or that are off topic are removed from search results. I believe Google also automatically removes 404s and other error messages.

Refer to the document for a full list.

There are three more sections: Unratable, Pornography and Malicious, but they are of little use (unless you’re in the porn business or attach viruses to your pages).

There’s an interesting post that claims Google pays $20 an hour to university students to review search results: http://www.searchbistro.com/index.php?/archives/19-Google-Secret-Lab,-Prelude.html

There’s also an official Quality Rater job posting on the Google board: http://groups.google.com/group/ba.jobs/browse_thread/thread/bc4f1dec633b9120/84689e434c38a1fb?q=%22search+quality%22+google+jobs&rnum=3&hl=en&pli=1

A search for EWOQ gives the official EWOQ link, which cannot be accessed without authorization.

There are around 15 pages in the document devoted to guiding new Raters inside Google’s Raters Hub.

Raters Workflow:

  • Acquiring Tasks

  • Starting to rate

  • Submitting your initial rating

  • Re-rating unresolved Tasks

  • Commenting


And here’s an interesting image of a Raters control panel:

Here’s where Googlers rate content:



In the control panel, raters can select between “Live Page” and “Cached Page.” If the live page doesn’t work, but the cached page does, raters have to rate the cached version. If both links do not work, the page is assigned to the ”Didn’t load” category.

Moderators resolve “unresolved tasks” or tasks that raters refuse to rate. Before flagging a task as unresolved, raters must provide comments.


Here’s how comments look:



And here’s the lingo Google raters use in comments:

-V (Vital) – DL (Didn’t Load)

- Usf (Useful) – FL (Foreign Language)

- Rel (Relevant) – MAL (Malicious)

- NR (Not Relevant) – PPC (pay-per-click)

- OT (Off Topic)

Google’s Human Secret Weapon

In the next article we’ll cover how Google raters identify and treat spam, and what is considered spam.

Danny Sullivan in “Search 4.0” stated that search engines are putting humans back in search via human rater programs such as the one discussed in this article, and behavioral data collected from users.

I believe it makes common sense to use human raters to filter out search results, but the biggest question is: to what extent? Aaron Wall of SEO Book claims that Google has close to 10,000 quality raters. How true is this? I have huge doubts, but when the two best known SEO figures speak of large human influence in search, you’d better listen. Danny hangs out with Eric, Sergei and Larry, while Aaron has engineer friends on the inside, so they ought to know what they’re talking about.

As for the guidelines, make you content the best you can and you won’t have to worry about raters. In fact, they’ll prop you up.

Google+ Comments

Google+ Comments