Navigating the Invisible Web
(Page 1 of 4 )
What do the majority of people do when searching for information on the Web? They fire up Google (or occasionally another search engine) and rely on the returned results to find what they are looking for. In many cases you are lucky enough to find among the first 10, 20, or 30 results the stuff you are after but sometimes even browsing the next hundreds of search results does not lead you an inch closer to what you want. It looks like the thing you are looking for is not out there on the Web although your internal voice tells you that this is impossible; it must be there!
Although it is true that search engines have revolutionized the way we use the Web and have made vast amount of content reachable by everybody, it is also true that search engines (even the most powerful ones) are not almighty. They cannot index to include in their databases every single page on the Web. And what is not in the search engines' databases you simply cannot retrieve it in the form of search results.
People learn quickly that even if search engines are the easiest way to search the Web, they are by no means the only one. What is more, it seems that in particular cases (for very specific searches) major search engines are no good. They waste much more time and drown users in so much irrelevant information that it is inevitable that one gets furious at them and starts looking for alternatives. But don't get angry; search engine really do a very nice job on the Web, at least on the visible part of it. If it happens that what you are looking for belongs to the other part, the Invisible Web, just leave Google for a while and follow the other paths to navigating the Invisible Web.
What is the Invisible Web and Why Does It Exist?
First, let's clarify some terms which are interconnected but identify different things. The portion of the Web that is indexed by search engines like Google, Yahoo, MSN, etc. is often called the Surface Web because it is the topmost, visible layer of all documents on the Web. Estimates are that currently the Surface Web consists of over 20 million Web servers (not to mention the number of documents on the each of them) and the portion that is not covered by general search engines is up to 500 times greater! Of these resources most belong to the Invisible Web, the portion of the Web that due to variety of reasons does not appear in the search results of the major search engines but can be accessed by special search tools. There are also other portions of the Web, referred to as the Opaque and Dark Web which are respectively content that is not linked and cannot be accessed even by special search tools and content that is not for general use (i.e. corporative Intranets). Having in mind the speed at which the Web grows, one can expect that the non-visible pages will increase – both in absolute numbers and as a percentage.
The above clarification on the parts of the Web was necessary because there are different ways to navigate the different portions of the Non-visible Net. For resources that belong to the Opaque and Dark Web, the only way to find them is if someone tells you their URL (and to provide you with the login credentials, if needed). For Dark Web sites, visitors from the general public are not always welcome, and this explains why site owners might explicitly ban search engines from indexing their sites.
Next: Reasons Pages Turn Invisible >>
More Search Optimization Articles
More By Tsvetanka Stoyanova
