In this article, we will take a look at the Robots Exclusion Standard. It sounds like something straight out of a science fiction book, but is really nothing more than a tool to prevent web spiders and robots from accessing a particular section of your website, or even your entire website if you so desire, that you don't want indexed. The standard goes by many names, like the Robot Exclusion Protocol, but you most likely have heard of it as the robots.txt protocol. No matter what you may call it, it is a handy tool that, when used properly, can help increase your ranking with the various web pages.
The standard was created in June of 1994 to handle robots that were accessing deep virtual trees, attacking servers with a succession of rapid requests, and downloading certain files over and over again. Despite its name, the Robots Exclusion Standard is not backed by any acting body or organization. Nor is it enforced by anyone, and there are no guarantees that any present or future robots will comply with it. There is a movement involving what is known as ACAP, or Automated Content Access Protocol, that is seeking to update the Robots Standard, and perhaps govern it, but that is beyond the present scope of this article.
Search Optimization 
