Use this tool to generate a simple robots.txt file for your website. This file allows you to hide files or directories that you don’t wish the search engine spiders to find. This generator tool is designed to create the text for the file for you; you can then make changes afterward should you deem it necessary.
Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.
To use the generator tool, enter the required information below and click the button. You will then be shown the text for the file. Copy this to a file called robots.txt and place on the root of your website (in the same place as your home page).
Imposing Restrictions
You may impose restrictions on which web pages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin/ directory, which commonly holds scripts, and their images directory /images/. To enable all web pages, select Yes to "Enable All Webpages," then enter each web page or directory path in the exclusion box, one per line.
Example: "http://www.yourdomain.com/cgi-bin/" (Excludes the /cgi-bin/ directory)
Example: "http://www.yourdomain.com/images/" (Excludes the /images/ directory)
Example: "http://www.yourdomain.com/welcome.html" (Excludes the /welcome.html web page)
White Papers
|
IT organizations have been conflicted between two backup approaches: local and centralized. While centralized backup requires less hardware, reduces administration time, and solves the security problem associated with loose tape media, it can introduce greater bandwidth consumption and longer backup/restore windows. Because of these issues, centralized backup has been a leap some managers have not been willing to make.
Request Your Free White Paper!
|
|
|
|
We are all aware of the tremendous costs of e-discovery. Those costs are compounded when the necessary documents are scattered throughout an organization in both paper and electronic versions. Despite advances in capture tools, tremendous amounts of paper are still used for keeping records in many organizations—even by organizations that have already eliminated some of their paper!
Request Your Free White Paper!
|
|
|
|
Complexity is expensive. Managing the ever-growing complexity of IT infrastructures is one of the top challenges faced by enterprise IT organizations. Static thresholds, rule-based systems, and manual data correlation techniques are indispensable tools for monitoring IT infrastructures, but these tools and the staff needed to manage them are rapidly becoming too costly and inadequate.
Request Your Free White Paper!
|
|
|
|
The Telework Fact Sheet details benefits for employers, employees and the larger community.
Request Your Free White Paper!
|
|
|
|
Enterprises that want to realize the benefits of virtualization to help decrease operating costs face a complex set of technologies that often slow or prevent adoption, potentially leading to missed opportunities. Read this technology report to see how Novell, IBM and Intel provide a game-changing virtualization solution.
Request Your Free White Paper!
|
|
|
|
Learn how to create your own winning technical support system by providing immediate and convenient solutions for common computer problems like spyware and viruses; diagnosing and repairing computer problems while the user watches and learns; and reducing the need for expensive movement and travel.
Request Your Free White Paper!
|
|
|
|
In this eSeminar, Bob Laliberte - an analyst with ESG - will present the results of ESG`s validation of HP`s Disaster Proof testing.
Request Your Free White Paper!
|
|
|
|
This white paper explains the benefits of protocol licensing for independent hardware vendors (IHVs), independent software vendors (ISVs), original equipment manufacturers (OEMs), and customers; describes the various protocols available for licensing; and highlights scenarios in which the use of these protocols can speed the development of interoperable products.
Request Your Free White Paper!
|
|
|
|
You must ensure that your .NET applications and .NET-based Web Services deliver the expected return on investment (ROI), while minimizing problems. This white paper presents best practices for .NET application management and support to increase ROI.
Request Your Free White Paper!
|
|
|
|
This technology brief details the top 10 ways to secure and accelerate your business by thwarting many web attacks with protections at the Web Gateway.
Request Your Free White Paper!
|
|
|
|
All White Papers.
|
