Search Engines For the Invisible Web
(Page 1 of 4 )
There are many websites and resources on the Internet that cannot be reached by querying the major search engines. Fortunately, there are other ways to reach this Invisible Web. This article discusses the various kinds of search engines and databases that can be used for exploring this hidden gold mine of information.
In a recent article I discussed the existence of a vast number of documents that, due to a variety of reasons, cannot be retrieved using the major search engines. But the so-called Invisible Web exists, and many treasures are hidden from easy access in various ways. Some sit behind the thick “firewalls” of sites that require registration to be searched. Others have complex database structures that search engine crawlers cannot technically deal with. Still others are hosted on a “black list” domain, or are simply skipped because search engines cannot index every single page on the Web the moment it appears.
In another article I described search directories as one of the tools for retrieving Invisible Web information. Now I am going to give more ideas about which search engines, in addition to the major ones like Google, Yahoo, MSN, and so on, can be used to find resources that are hidden among the vast amount of noise on the Internet.
Probably it is important to clarify that the Invisible Web is also an entity on the move. For example, a site that is not accessible via Google today can be included by the search engine tomorrow, when Google's spider visits the site. But it is also possible that, if the whole site or certain pages are accessible only after registration (even if it is free), the site will never appear on Google.
No company works to find only the pages that are not available through searches performed on Google, Yahoo, and MSN, so the search engines for the Invisible Web do not provide only the results that are missing in the major search engines. On the contrary, they provide mixed results that reflect what their indexing algorithms have found. These results include both pages that can be found via Google and pages that cannot. It is the second group that is more interesting.
Next: Searchable Sites for the Invisible Web >>
More Search Optimization Articles
More By Tsvetanka Stoyanova
