Using Xenu Link Sleuth to Spot and Improve External Links in a Domain
(Page 1 of 4 )
Xenu Sleuth is one of the most important open source tools in web development and search engine optimization. Xenu Sleuth performs the task of crawling and checking links in a domain, just as other bots such as Googlebot and Yahoo! Slurp do. This article explains how to use Xenu Sleuth to improve the external or outgoing links on your web site.
Xenu Sleuth is free and can be downloaded here. However, this is a tutorial focusing on Xenu Link Sleuth 1.2j. I believe that the developers released a more recent version. If this is the case, please find a link in their site to download the recommended (older) version.
External links are the types of links in a website that go out to other domains. Search engine rankings in major search engines like Google, Yahoo and MSN give importance to the quality and relevance of external links in your website.
Properly-checked external links can help search engine rankings a lot. The objective of this article is to show to you how you can spot and improve the external links found in your website using Xenu Link Sleuth.
The Basics about Xenu Sleuth: Important information
Xenu has its own file type, .XEN. After it crawls your site, you can save the results to your desktop. This is highly recommended and useful because:
- You can re-open the Xenu file in the future for further analysis of the links in your site.
- When you have a Xenu file of a certain domain, say in month X, and then you let Xenu re-crawl the site after six months, you can compare the two Xenu files to easily see what links appear and what links are gone in between these months.
- On big sites, a Xenu crawl can last an hour, so it is important to save it as .xen so that you will not need to re-crawl the site again.
Not all sites are crawlable with Xenu; some sites have disallowed it in their robots.txt because Xenu can consume a lot of bandwidth and slow down the site. This means that you have to use Xenu with caution.
Also, if the site uses session IDs and their shopping baskets are crawlable, Xenu can add several items into the basket without checking out. Re-crawl the site with caution, as this can cause some kind of DOS (Denial of Service) attack.
To start Xenu, you must configure things correctly. After successful installation, launch Xenu, navigate to File -- ? and then Check URL. (see below)
Important rules:
- Type the root URL of your domain, starting with http://. This will ensure that Xenu will start crawling at the top most part of your domain and continue all the way down.
- Uncheck External links, so that you can distinguish external links in the exported report later and save time in the crawling process.
- Leave other settings as is (default).
See the screen shot above for what should be filled in at Xenu's Starting Point.
Okay, let's go to preferences and options. This can be found under Options ? Preferences. Then make sure Xenu has been set using the following values:
Parallel Threads: 30
Ask for password or certificate when needed: (check this one)
Treat Redirection as errors: No (do not check this one)
Maximum Level: 999
In the report section check this all options, starting with Broken Links, ordered by links until Orphan files
It is recommended that during the crawling process you minimize all Internet activity; otherwise, you might experience several timeouts in the results.
Next: The Process of Crawling >>
More Search Optimization Articles
More By Codex-M
/ 5 
