Google Sitemaps Review - Error Reporting from Google
(Page 4 of 6 )
Errors
If Google fails crawling some of your pages, you would probably want to know about it. It means that your pages might not be linked or set up correctly, and they aren't getting indexed. Before Google Sitemaps you had to guess what the Googlebot what doing by looking at your webserver log files. Now, Google gives you these failures so you can troubleshoot them.
The first section gives the basic HTTP errors of pages that were linked within the site, in the sitemap you submitted, or linked from another site. You can look for the errors of the first two, but obviously can't do much about the latter. Unreachable URLs shows when Googlebot encountered network issues, such as when it finds that it is overloading your server. URLs not followed are pages that Google could not reach because of your redirects. If you have pages in this category, you really need to work on troubleshooting them.
URLs restricted by robots.txt is pretty self explanatory, and it can help you to be sure that your robots file is blocking the right pages. They are not necessarily errors, but they explain why Google is not indexing the pages. We have an image of our robots blocking page below to demonstrate what all the error pages basically look like.
Robots Error Sheet
The details show the URL, the detail (in this case "URL restricted by robots.txt") and the last date that Google tried to follow a link to the page.
In detailing robots, this much information is fine. However, with a section like HTTP errors, it would be more helpful to know the link referrer. Without knowing where the bad link is-or even if it is on your own site-you are left stabbing in the dark to solve the error unless there is something obviously wrong with the linked page.
Our robots information leads us onto the next section of information that Google Sitemaps gives us.
Next: Robots.txt Troubleshooting >>
More Google Optimization Articles
More By Developer Shed
/ 72 
