Search Directories For the Invisible Web

Google indexes billions of Web pages, but there are many more billions of Web pages and websites that the major search engines cannot reach. In fact, by some estimates, this “Invisible Web” is 500 times the size of the visible one. Fortunately for searchers, there are ways to access this treasure trove. This article explains how search directories work, and what can be found by using some of the major ones.

In a recent article I discussed the existence of a vast number of documents that, due to a variety of reasons, cannot be retrieved using the major search engines. But the so-called Invisible Web exists, and many treasures are hidden from easy access in various ways. Some sit behind the thick “firewalls” of sites that require registration to be searched. Others have complex database structures that search engine crawlers cannot technically deal with. Still others are hosted on a “black list” domain, or are simply skipped because search engines cannot index every single page on the Web the moment it appears.

Still, these pages exist, and their number significantly exceeds the number of pages that are accessible via the major search engines. According to rough estimates, the Invisible Web is 500 times bigger than what is served by search engines. And what is more important, often the information you search for cannot be found (or at least not easily) using the major search engines, while at the same time it is just a couple of clicks away from you. And if you do not know how to search for it, you might never find it.

There are special tools for searching this vast ocean of information. True, they are not as convenient as searching with Google, and they take much more time and effort, but when you need very specific information this might be the only way to get it. The two major tools for searching the Invisible Net are the specialized search engines and the search (known also as subject) directories. Additionally, using the Invisible Web databases and virtual reference libraries often can deliver you the information you need, but the two major tools are the search engines and the search directories for the Invisible Web.

The major difference between search engines and search directories is the way in which they obtain data. Search engines use spiders and index pages automatically, while search directories are manually fed with data. Also, search directories are often maintained by humans, so in a sense the content of the site is verified. As a rule, it is the responsibility of the submitter to give a short description of the site and to classify it under the appropriate category.

In search directories, as the name implies, sites are arranged hierarchically by topics and subtopics. Unless it is a very specialized directory, usually the top-level topics start with broad categories such as business, technology, science, entertainment, and so on. Each of these topics has its subtopics, which in turn have their subtopics, and so forth. Subtopics may continue branching off to a depth of several levels, thus providing a very targeted approach to position the site in the category where it fits best.

The hierarchical structure of Web directories sometimes poses difficulties regarding the right category in which a site belongs, because it could fit in more than one category. Attempts to submit it in all of them could lead to being classified as a spammer, and as a result, a particular directory might totally reject the site for inclusion.  So, as a SEO expert, consider which category fits best, and as a user, if you do not see what you are looking for right away, consider alternative categories under which the information you are looking for could be classified.

Very often search directories are not just a vast collection of links. For making information retrieval easier, search directories could provide a search function. It is this visual similarity between search directories and search engines that misleads some into thinking they are just the same. But when you hit the Search button and retrieve the results that match the search query, you see the list of categories as well — and what is more important, usually it is possible to search only within a particular category, which speeds the search a lot.

Since search directories are manually fed with data, it is logical that sites do not always appear immediately after they are submitted. The delay could vary from a couple of days to several months to a year. As a SEO expert you should keep this in mind when planning submissions. As a user, searching for information, you should bear in mind that search directories cannot always include the latest links, and sometimes you will encounter broken links in directory listings, but in general, especially with human-maintained directories, this is rare because irrelevant information is periodically removed.

I would like to say one more time that search directories are not the fastest way to retrieve information. Often one will have to browse endless pages of link collections, and there is no guarantee that you will find what you need. In some cases you will be more successful when looking in all-purpose directories, while in other cases a topical directory will be a better choice. While all-purpose search directories in general contain many links to all kinds of sites on the Internet, topical search directories are targeted to a specific area –- academic research, news, sites from/about particular countries or regions, medical, educational, financial information, and so on.

It is not possible to visit all search directories on the Internet, but it is also hardly necessary. Depending on what kind of information you are looking for, you can start your search with the most established all-purpose directories, such as:

  • The Open Directory Project (DMOZ – http://www.dmoz.org) is probably the biggest and best-known search directory on the Internet. It has more than 600,000 categories (just imagine the number of sites there!) that cover even items you may have never thought could exist. There is a good chance that you will find what you are looking for in this directory.

  • The Invisible Web Directory (http://www.invisible-web.net) is another directory listing that classifies many sites in categories and subcategories.

  • The official directory of the sites in the world (http://www.123world.com/) contains a vast amount of links in different categories. It also does not include absolutely everything you might think of (for example, there are many countries not listed in the Countries category) but it is another good starting point.

  • The Internet Public Library (http://www.ipl.org) is more of a reference source than a general search directory, but it contains many links covering different subjects, organized in many categories and subcategories.

  • Librarians’ Index to the Internet (http://www.lii.org) is also one of the big search directories on the Web. Since it is compiled by librarians, it has the advantage of human-maintained directories in respect to content quality.

  • About.com (http://www.about.com) is an old but golden source of reliable information. About.com has many categories and subcategories, complemented with articles by its journalists and external authors.

  • The Scout Report (http://scout.wisc.edu) contains about 17,000 cataloged sites. It allows users to browse a catalog by using Library of Congress Subject Headings.

  • Many university libraries have vast collections of resources, which often are not strictly academic but cover a wide range of topics. Often these resources are publicly available on the Web and can also be used for searching the Invisible Web. Such a resource is the Best Information on the Net site (http://library.sau.edu/bestinfo/) and the above mentioned Scout Report.

  • Two of the major search engines – Yahoo (http://dir.yahoo.com/) and Google (http://directory.google.com/) maintain large directory listings of sites, although most of them are accessible via their search engines as well, so these cannot be qualified exactly as Invisible Web search directories.

All-purpose search directories offer many resources, but sometimes a specialized topical directory, which offers much less, can be a faster way to find what you are looking for. Sometimes the search directories themselves contain links to other topical search directories. These topical search directories are known under different names in addition to “directory.” They can be called “Web directories,” “gateway pages,” “guides,” and other designations. If as a SEO practitioner you are responsible for sites that are not oriented towards the general public but are targeted at a specific audience, such topical directories are a highly recommended place for submission.

Most such collections are written and/or maintained by experts in the particular topic or field, and the resources they contain can be really valuable. There are specialized search directories based on different criteria that cover topics like:

  • Country-oriented information. For instance, there are many directories that concentrate on sites from a particular country mainly, like http://www.alephsearch.co.uk/, which is UK oriented. If you are interested in resources oriented toward a particular country, the biggest search directories offer collections of sites arranged in directories by country.

  • Legal resources. One of the best online research resources for lawyers and jurists in general is FindLaw (http://www.findlaw.com). This site contains real treasures for legal professionals and for the general public.

  • Academic information, which is one of the major groups of information on the Web. If you are looking for academic research, courses, or simply educational resources for students, visit places like Infomine (http://infomine.ucr.edu) and Academic Info http://www.academicinfo.net/. Both sites contain many resources, plus links to more specialized search directories and engines that delve into a particular branch of academic study (for instance, medieval history, microbiology, etc.)

  • Medical and health information, which is another group of topics that can be located by specialized search engines. There are many directories available for medical and health information, and it is difficult to name only one or two. The Best Medical Internet Resources site (http://www.meduniv.lviv.ua/links/) is a good starting point for research.

I hope this article gave you at least a brief idea about the diversity of resources hidden in places like search directories. Before the advent of Google, directories used to be a major tool for finding information. The ease of use and the reliability of Google made users slow to search for information outside the search box of google.com. I do not mean that Google does not do a wonderful job in making the Web a better place. I just want to point out that there are additional places worth visiting, when one is searching for really quality information.

Google+ Comments

Google+ Comments