Write a Robots.txt File - How Do I Build a Robots.txt?

(Page 2 of 5 )

How do you build a robots.txt file for your website? I am glad you asked. One thing you do not want to do is use an HTML editor to build this file. The easiest way to create the file is with a text editor like Notepad. After opening Notepad (or another text editor), save the blank file as robots.txt. This file will be placed on the root level of your web server, or in other words the same folder as your index page, once it is complete.

Now I will cover several different methods of efficiently using a robots.txt file to direct the robot to crawl the correct directories and and avoid others.

First we will discuss how to format information. The text file is actually a list. Its directions consist of two fields, or lines of instruction.

The first line is the User-agent line. This is the line where you can specify which search spider bots are allowed to index your site(s). The second line is the directive line or disallow field. This is the line you will use to block folders or files blocked from spiders.

Of particular note: if the publishers of Perfect 10 magazine (an online porn magazine suing Google for linking to their images) had used the robots.txt file, they could stop search spiders from indexing their images. Is it Googles' fault the magazine hired incompetent IT staff? I don't think so. To me it's another adult webmaster looking for more free publicity.

To write the robots.txt file, you would start by addressing specific search engines. The User-agent line would start as:

User-agent:

Adding a specific search engines spider name here will give the search spider notice that it is to follow the next line for instruction, i.e.:

User-agent: googlebot

This tells googlebot that it is to follow the next line's directions on how to proceed through your website, or to leave altogether. You may also employ the use of an asterisk (*) as a wildcard for all search spiders.

Next: Writing the File, Using Disallows >>
 

More Search Optimization Articles
More By Clint Dixon

Comments On: Write a Robots.txt File

· Important, but simple, that's the robots.txt and this article is similiar, important...    · i don't know anything about robot text really so this article was great, i noticed...    · Search engine spiders will ask your web server for the robots.txt whether it is...    · I agree. Even if you don't know what to put in your robots.txt file, uploading one...    · 1st i have to say that is a great site it took me for ever gather bits and bits of...    · Hello, I need a little help with my robots.txt file. I know how I need my code to...    · "Hello, I need a little help with my robots.txt file. I know how I need my code to...    · robots.txt is one of those things I'd forgotten on a couple recent projects. Great...    · This article doesn't mention Allow (which is NOT Google-specific, by the way) or the...   >>> Post your comment now!