I, Robots.txt

In this article, we will take a look at the Robots Exclusion Standard. It sounds like something straight out of a science fiction book, but is really nothing more than a tool to prevent web spiders and robots from accessing a particular section of your website, or even your entire website if you so desire, that you don’t want indexed. The standard goes by many names, like the Robot Exclusion Protocol, but you most likely have heard of it as the robots.txt protocol. No matter what you may call it, it is a handy tool that, when used properly, can help increase your ranking with the various web pages.
Continue reading