So what are the reasons pages or sites go Invisible? There are variety of reasons but the most common ones are:
No matter how refined and powerful search engines and their indexing algorithms are, they still do not have the technical possibility to index every single page the moment it appears on the Web.
The page has appeared after the latest crawl of the spider and it will not be indexed until the next time the spider returns to the site. Chances are that the next crawl of the spider will include many results currently excluded from the search engine's database.
The site requires registration before use or is fee-based and this prevents the search engine from indexing it. While these conditions hold true, there is no chance to index the site.
The search engine has excluded the site from its database. Sometimes Google (and other search engines) exclude from their listings sites that violate the rules and misbehave by trying to fool the search engine in order to get a top listing. Do not take it for granted, but chances are that (hopefully) one day, the site will return in the search results. Until that day comes, welcome to the (invisible Web) club!
A particular site is locked inside a database and cannot be searched, because it cannot be indexed by Web crawlers. Very often the particular site's database is completely searcheable but only after you personally go there and perform the search.
Although generally dynamic pages are indexed by search engines, it is not realistic to expect that the last-minute news will top Google. Instead, if you are looking for news, stock quotes or similar items that change often and quickly become old, go directly to a known source: a news portal, the site of a news media company, or a financial institution that provides such information online.
Due to the restrictions search engines have regarding the depth of crawl and/or page size, separate pages or whole sites could be skipped during indexing. For instance, if the URL of your site is http://www.domain.com/dir1/dir/11/dir111/.../mysite/index.html it might or might not get indexed but why use such URL when you can use http://www.mysite.com/index.html instead?
The domain has been classified as a source of spam and because of this all pages that are hosted there are excluded from search engines.
If you are looking for a page/site in a language that has special characters (accented characters, umlauts, etc.), which are sometimes substituted by ASCII symbols, try searching using all possible transcriptions of the keyword. Or go to a national search engine that does not have such problems.
Pages in unsupported format or pages that use question marks in links are generally not loved by search engines and even if the search engine does not have a policy of explicitly excluding such pages, it is less likely that they will be indexed.
The list of reasons is hardly exhaustive but I hope that it will be useful for both users desperately hunting for information and for SEO experts who, although most likely aware of all those reasons, are worried why some or all of their pages do not appear in the search results.
