This is a brief history of search engines and search technology. In this article you can learn the basics of how search engines work and how they got to where they are at the moment. We'll also look at important ranking factors, including the historical reasons for why they are important; near the end, we'll consider what factors may become more important in the future.
Parts of the Search Engine
There are three main parts to every search engine:
Spider
Index
Web Interface
Spider
A spider crawls the web. It follows links and scans web pages. All search engines have periods of deep crawl and quick crawl. During a deep crawl, the spider follows all links it can find and scans web pages in their entirety. During a quick crawl, the spider does not follow all links and may not scan pages in their entirety.
The job of the spider is to discover new pages and to collect copies of those pages, which are then analyzed in the index.
Crawl Rate
Pages that are considered important get crawled frequently. For example, the New York Times may be crawled every hour or so to put new stories in the index. Less authoritative sites with less PR are crawled less frequently, even as rarely as once a month. The crawl rate depends directly on link popularity and domain authority.
If many links point to a website, it may be an important site, so it makes sense to crawl it more often than a site with fewer links. This is also a money-saving issue. If search engines were to crawl all sites at an equal rate, it would take more time overall and cost more as a result.
More Spider Features
Spiders may check for duplicate content before passing page copy to the index, in order to keep the index clean (or at least cleaner).
Search Engine News 
/ 3 
