I, Robots.txt - Limiting Bots

(Page 4 of 5 )

So far we have focused primarily on how to limit files. Now we will work with limiting specific bots from accessing our files.

If we want to tell one specific bot to stay out of all of our directories, we input the following code into our robots.txt file:

User-agent: Google-Bot

Disallow: /

Now Google should stay away from all of our directories.

We can also tell a specific bot to ignore one or more directories or files, like so:

User-agent: NerdBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

And finally, if we want to specify that several bots are not allowed to access a directory or file, we can do so in this manner:

User-agent: NerdBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

 

User-agent: FatBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

Disallow: /tmp/

 

User-agent: HedonismBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

 

User-agent: Bender

Disallow: /images/

Disallow: /secrets/globaldomination.html

Disallow: /cgi-bin/

Whenever you add a bot, you must include a space between your line, which tells the interpreter that this is a new record.

Next: Commenting and Loosely Supported Extensions >>
 

More Search Optimization Articles
More By Jamesp

Comments On: I, Robots.txt

· Thanks for stopping by to read my article on the robots.txt protocol. In it, I...    · Thanks for providing such an informative post. It is always great to see issues...    · Thanks, glad you enjoyed it!   >>> Post your comment now!