The Missing Link - Making the Spider Happy
(Page 4 of 9 )
On the first visit the spider starts on the homepage and finds the sitemap page and then the pages linked from the sitemap page. In other words, it finds all 8 pages on the first visit.
Figure 3 Site using a sitemap
So what have we accomplished by using this model where the homepage links to a sitemap, and sitemap link to our pages? Hopefully the answer is fairly obvious. We get our site spidered quickly, while maintaining a clean homepage. There are in fact other PR related benefits (see Appendix 1).
What About Big Sites?
Now, suppose that you have a site with 1000 pages. Does that mean you need a sitemap page with 1000 links on it? The answer to this one has to be NO. A spider will only read a certain number of links on a page, probably governed largely by the length of the page. It has been suggested on forums that the maximum number of links that will be spidered on a page will be around the 100 mark. So where do we go with our idea of a sitemap?
My solution is to have a mini-site sitemap. A group of sitemap pages all linked together. The homepage will link to the first sitemap page, which links to every other sitemap page. For a 1000 page site, you could have 10 sitemap pages, each with links to 100 site pages. I personally go for a smaller number of links per page, say 20 pages of 50 links each, but that is personal preference. Each of my 20 site map pages will have links to 50 pages on my site PLUS, links to the other 19 sitemap pages. That way, every sitemap page is linked to every other. For a 1000 page site map, using this strategy, how long would it take to get all pages found by the spider?
Example of a Mini-Site Sitemap
The homepage links to sitemap page 1. Sitemap page 1 is linked to all other sitemap pages as well as to a number of normal site pages.
Figure 4 A Site Map Mini-site
Next: Can You See What The Spider Will Do? >>
More Link Trading Articles
More By Andrew Williams
