Use this tool to generate a simple robots.txt file for your website. This file allows you to hide files or directories that you don’t wish the search engine spiders to find. This generator tool is designed to create the text for the file for you; you can then make changes afterward should you deem it necessary.
Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.
To use the generator tool, enter the required information below and click the button. You will then be shown the text for the file. Copy this to a file called robots.txt and place on the root of your website (in the same place as your home page).
Imposing Restrictions
You may impose restrictions on which web pages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin/ directory, which commonly holds scripts, and their images directory /images/. To enable all web pages, select Yes to "Enable All Webpages," then enter each web page or directory path in the exclusion box, one per line.
Example: "http://www.yourdomain.com/cgi-bin/" (Excludes the /cgi-bin/ directory)
Example: "http://www.yourdomain.com/images/" (Excludes the /images/ directory)
Example: "http://www.yourdomain.com/welcome.html" (Excludes the /welcome.html web page)
White Papers
|
Warburton's the largest independently owned bakery in the UK faced a number of difficult challenges in providing the most robust yet efficient IT infrastructure for their organization's success. IBM's services combined with their xSeries servers created the perfect platform for their SAP environment with sufficient flexibility, and did so in very time effective fashion.
Request Your Free White Paper!
|
|
|
|
This white paper describes the benefits you can expect with SOA, and how IBM can help take your business there.
Request Your Free White Paper!
|
|
|
|
Gartner summarizes its view on Application Delivery Controllers, evaluates strengths and weaknesses of solutions, and provides Magic Quadrant reporting for a quick comparison across all vendors. Learn from Gartner how you can benefit from an all-in-one device like Citrix NetScaler that delivers the highest levels of availability, performance and security.
Request Your Free White Paper!
|
|
|
|
What you don't know can hurt you, and is likely costing you money and increasing your security risks during an era of scarce resources. This white paper proposes six key strategies that enterprise security managers can use to improve their network defense posture.
Request Your Free White Paper!
|
|
|
|
The rationalized multi-tool approach is flexible, scalable and cost effective. It provides the necessary input to the IT service management business processes. It preserves prior investments in monitoring tools, empowers technologists to select the best tools with which to do their jobs, and enhances effective response to incidents.
Request Your Free White Paper!
|
|
|
|
This Migration Guide reviews the approach that Principled Technologies and Dell recommend for a tested and validated migration of Oracle 9i databases to Oracle 11g on the Dell PowerEdge R900 server and RHEL 5.2.
Request Your Free White Paper!
|
|
|
|
This video is part of a 3-part series that focuses on organizations in varied industries – Hilton Grand Vacations, Duke Corporate Education, and Intersil - that have solved application performance issues over the WAN with wide-area data services (WDS) from Riverbed. Benefits to the organizations included application acceleration, consolidated IT infrastructure, optimized bandwidth, and accelerated backup and replication. Watch part 2 of the series featuring Intersil.
Request Your Free White Paper!
|
|
|
|
Large-scale deployments of personal computers are complicated and time-consuming. The costs associated with deploying and maintaining fleets of computers have become a large portion of many companies overall information technology budget. Download this white paper to learn more.
Request Your Free White Paper!
|
|
|
|
During this eSeminar, Michael Vizard will discuss the current state of the storage industry including the challenges and opportunities afforded by Web 2.0 technologies.
Request Your Free White Paper!
|
|
|
|
One of the most common ways to start your Unified Communication solution strategy is to build an approach that starts with a Pilot Program. This approach has many aspects that make it a very good way to proceed.
Request Your Free White Paper!
|
|
|
|
All White Papers.
|
