The Yahoo SLURP Crawler

(Page 1 of 4 )

As SEOs and webmasters, we're always looking for ways to get the search engine spiders to crawl our sites, and the deeper, the better. This article shows you how to target Yahoo's crawler and convince it to stop by regularly.

The search engine wars are fought with strategies, alliances, and robots. As Yahoo! primes itself to be the number one contender for market share after Google, websites that want to optimize for Yahoo must study how Yahoo ranks pages and how it indexes pages. The Yahoo web crawler SLURP should be studied; your site server logs should have recorded visits from various robots, including SLURP. If you do not have records of SLURP visiting your site, then this article will give tips on how to get SLURP to crawl (hopefully deep crawl) your site.

The Preamble

Yahoo SLURP evolved from Inktomi SLURP. The Yahoo SLURP robot is an upgrade from Inktomi’s SLURP. Yahoo used Inktomi’s search engine to replace Google, which used to take care of its search results. This officially triggered the second search engine wars (the first was won by Google without it declaring hostilities).

Yahoo has at least 130 million registered users on its network. Granted, Google is the definitive search engine, but Yahoo is large enough that it should not be ignored.

Next: The Robot >>
 

More Search Engine Spiders Articles
More By Akinola Akintomide

Comments On: The Yahoo SLURP Crawler

· robots inability to understand context, and their susceptbility to being fooled by...    · wonderful article!although i have a question about Slurp and underscores. How...    · I was looking for alot more information than this article gave me. There was...    · ouch, do u have any paricular questions? i would be more than happy to answer them,...    · I would agree that it is a fairly generic article.What factors may contribute to...    · I know hyphens are not seen by some robots, so it probably sees the whole word...   >>> Post your comment now!