Study Results: Search Engines, Meta Robots Tag and Robots.txt
(Page 1 of 5 )
Welcome to the second part of a two-part series that tests search engine reactions to the meta robots tag and robots.txt. In this part, the results of the experiment and testing we set up in the first part will be presented. For details about the experiment background, setup of the test pages and user-agents, please refer to the first part. You need to have read it very carefully to easily understand this one.
Data collection, as mentioned in the first part, will be done by Crawl Track, a web analytic script used to detect search engine bot visits to the test pages. The time frame of the experiment covered October 17, 2009 through November 20, 2009, more than a month of data collection and observation.
After logging in to the Crawl Track webmaster dashboard and visiting the "Crawler" section, you can see the complete list of bots that visits the website:
So for example, if I want to check the pages crawled by the Slurp Inktomi (Yahoo bot crawler), all I have to do is click; the detailed URLs that were visited will be revealed. Since the test pages have some crawl track code embedded in them using PHP, they will be traced using the above technique also.
All tests and experiments are hosted at www.php-developer.org . If you are ready to see the results, keep reading.
Search Optimization 
/ 8 
