Blocking Complicated URLs with Robots.txt - Benefits of Proper Robots.txt Usage (Page 2 of 5 ) As discussed, robots.txt is one of the most powerful web server file for the following reasons:
Since it provides instructions to search engine robots and other bots that tells them which directories and files in your web server are not to be crawled, you save a lot of bandwidth for the web site. Thus, you can divert that saved bandwidth for other purposes, such as improving the experience of your visitors, multi-media applications and other uses. Bandwidth is expensive for a web site, especially if you have a lot of visitors. Also, since search engine robots know what parts of your website are to be crawled, they will index your site in a very efficient manner. Crawling efficiency is very important for big sites with frequently-updated content. E-commerce websites that add new products on a daily basis can benefit from frequent crawling. This results in pages that will appear early in the search engine index, thus helping to increase the number of visitors to your web site.
No one will steal your protected content in the search engine results. For example, let us suppose you are a professional photographer with a lot of photos saved on your web server. If you are not using robots.txt, search engine bots can crawl every part of your web site, and it is highly possible that they might crawl and index your protected pictures. Then, when someone searches for images using a search engine (like Google or Yahoo), they might see your pictures and use them elsewhere -- such as their own web site -- or even alter and/or sell them without your permission!
Next: Google Webmaster tools Robots.txt Analysis Tool >>
More Search Optimization Articles
More By Codex-M |
/ 4 
