Web Pages to Include in Your Site - Give Instructions

Most of these pages are designed to cover your assets legally and/or help your human visitors in one way or another. But you play host to several visitors that aren't human – the indexing bots sent out by the search engines. You want to help them, too. You want to tell them what content they should index, and what content they should avoid. You can do this with a robots.txt file, known more fully as a Robots Exclusion Standard directive.

There are other ways to hide content from the search engines. The easiest is to simply not put it on your web site. But all of the major search engines obey robots.txt. If you choose to use it, make sure you are using it correctly. James Payne, our editor-in-chief, wrote an article detailing how to use robots.txt files; you might want to check it out.

Finally, we come to the site map. Actually, there are two kinds of site maps. One is for the navigation of your human users, though the search engines benefit from it as well. What you show on your site map depends on a number of factors, such as how dynamic your site is, and whether you want users to know that particular areas exist if they don't have permission to use them. The amount of content you have on your web site, and what kind, may play a role in how you organize your site map; some particularly large sites will use a scheme that puts pages in alphabetical order. For some reason, I've noticed this more with university web sites, though large corporations will use it as well. It seems to be more common, however, for the site map to be divided into categories, with the appropriate pages under each category.

The other kind of site map is specifically for the search engines to crawl. You can find out more about it on the official Sitemap page. As the site explains, “In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.” You can find the protocol for this kind of Sitemap at the link I included above; going into detail about it is beyond the scope of this article.

It's going to take a good bit of work to create these pages for your web site, and even after you've created them, you'll probably need to revise them as your web site and your company grows and changes. But they will give your site a more professional look and feel to your visitors, and could indirectly increase conversions. Good luck!