Write a Robots.txt File - Leaving Comments in the File
(Page 5 of 5 )
You can also use comments in your robots.txt file, but you need to be careful of where they are used.
Disallow: /images/ #comment send googlebot away
We could run into a problem if a search spider bot attempts to disallow /images/#comment, which is a not a folder on the server and would more than likely tell the bots to just leave the website altogether.
It is better to leave your comments on their own separate line. See the example below.
#keeps googlebot out of my porn
User-agent: googlebot
Disallow: /images/
So as we can see there is a very valid and legitimate reason to use the robots.txt file. There are also numerous other times to use the file. In some cases it could stop a large company from looking like fools for not protecting their intellectual property, and in others it would stop sensitive data from being crawled and indexed over the internet, and also to help a site increase positions in the natural organic search results listings.
After you have written your robots.txt file and placed it on your server, you should validate it with one of the robots.txt validation tools online.
| DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware. |
/ 32 
