Polite Bots

If you’ve ever wondered how to get a little better control over what parts of your web site get crawled by the search engines, how they crawl your pages, and how to encourage them to visit, keep reading. This article will explain the various protocols that the search engine robots (particularly Google’s) follow. It will also touch upon ways to help you guard against scraper bots.
Continue reading