I, Robots.txt - Limiting Bots

So far we have focused primarily on how to limit files. Now we will work with limiting specific bots from accessing our files.

If we want to tell one specific bot to stay out of all of our directories, we input the following code into our robots.txt file:

User-agent: Google-Bot

Disallow: /

Now Google should stay away from all of our directories.

We can also tell a specific bot to ignore one or more directories or files, like so:

User-agent: NerdBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

And finally, if we want to specify that several bots are not allowed to access a directory or file, we can do so in this manner:

User-agent: NerdBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

User-agent: FatBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

Disallow: /tmp/

User-agent: HedonismBot

Disallow: /images/

Disallow: /secrets/globaldomination.html

User-agent: Bender

Disallow: /images/

Disallow: /secrets/globaldomination.html

Disallow: /cgi-bin/

Whenever you add a bot, you must include a space between your line, which tells the interpreter that this is a new record.

Next: Commenting and Loosely Supported Extensions >>
 

More Search Optimization Articles
More By Jamesp