Write a Robots.txt File - Writing the File, Using Disallows
(Page 3 of 5 )
The second line known as the directive is written as:
Disallow:
By adding a folder after the Disallow statement, the search spider should ignore the folder for indexing purposes and move to others where there is no restriction.
Disallow: /images/
This is a special example, just for Perfect 10. This one minute bit of instruction could have saved a ton in wasted legal fees on a frivolous lawsuit. As this is a basic step in building websites, it is incumbent on website owners to protect their intellectual property, and not a 3rd party search engines duty.
You can also disallow specific files this way
Disallow: cheeseyporn.htm
One way I recommend using this all the time is to keep robots out of you cgi bin directory
Disallow: /cgi-bin/
If you leave the Disallow directive line blank or not filled in, this indicates that ALL files may be retrieved and or indexed by specifiedl robot(s). This would let all robots index all files.
User-agent: *
Disallow:
And vice versa you can keep all robots out easily.
User-agent: *
Disallow: /
In the example above, the one forward slash (/) equals your root directory. Since the root directory is blocked, none of the other folders and files can be indexed or crawled. Your site will be removed from search engines once they read your robots.txt and update their indexes.
Next: More uses for Disallows and User-agents in Robots.txt >>
More Search Optimization Articles
More By Clint Dixon
/ 32 
