What is Yahoo! Brewing in its Lab?

We hear so much about Google that even its close competitor Yahoo! can get lost in all the noise. Sure, Google has been coming up with some great things, but Yahoo! has certainly not been sitting idle. What have the search geniuses been working on over there? We’re glad you asked.

Google is well-known for its Google Labs. This is where the search engine giant features projects that it is trying out. They are usually in beta, so not quite ready for prime time. Making them accessible to the public affords Google the chance to see how they perform when real users get their hands on them. Many of these projects have user forums connected to them, which lets Google receive even more input about how well (or poorly) these experiments fill user needs.

What people tend to forget is that Yahoo! maintains a similar area on its own website. Well, more than one, to be precise. There is an area called Yahoo! Next (http://next.yahoo.com/). It follows the same pattern as Google Labs. The company describes it as “a showcase of some of Yahoo!’s newest and coolest projects – the cutting edge of what Yahoo!’s doing today and working on for tomorrow!”

Yahoo! Next is explicitly connected to another area of the site called Yahoo! Research (http://research.yahoo.com). While there doesn’t seem to be anything to play around with here, there are tantalizing hints of projects that Yahoo! is working on, apparently not even in beta yet, which could change the way we search. That part of the site also lets you check out the wide range of science publications written at Yahoo! Research Labs. You can click to read the abstract; if you want to browse the publication itself, however, you need to send an emailed request (clicking on the title of the publication in question automatically pops one up for you if you use Outlook; all you have to do is click “send”).

Even for just the abstract of a paper, you need to be able to translate some pretty dense computer science lingo – or possibly jargon from other disciplines; Yahoo! researchers come from a wide variety of backgrounds. But patience can be rewarded. Take one of the more recent papers on the site: “Discovering large dense subgraphs in massive graphs,” published in September of 2005. The authors talk about their new algorithm, which is “based on a recursive application of fingerprinting via shingles, and is extremely efficient, capable of handling graphs with tens of billions of edges on a single machine with modest resources.” The authors apply their algorithm to hosts on the World Wide Web, and discover that a lot of the “dense subgraphs” are link spam. In short, these clever researchers just might have created another potential tool to use in the battle against search spam.

{mospagebreak title=Just Looking, or Looking to Buy?}

It is on Yahoo! Research that one can find information about Mindset, which Yahoo! describes as “intent-driven search.” Mindset uses advanced algorithms explained in a research paper published by Yahoo! researchers in March to train a machine-based classifier. The classifier separates a page based on whether it is mainly commercial (trying to sell you something) or not. The idea was to create a classifier which is accurate but still fast enough to classify web pages on the fly, so the user won’t notice any effect on the speed of search results returned.

The demo of Mindset, which is available on Yahoo! Next (but also linked to appropriately on Yahoo! Research) takes advantage of a slider. Leaving the slider in the middle returns results in the “usual” order for a Yahoo! search. Move the slider to the right, for “researching,” and your results will be ordered with the ones that score highest as “informational” at the top. Move it all the way to the left, and your results will be ordered with the most “commercial” ones at the top. There are, of course, continuums between each position.

For instance, leaving the slider in the middle while searching for “CPU” turned up a review site first, followed by the Freedom CPU website (apparently an effort to bring open source to the world of microelectronics), followed by some sites that defined the term CPU, more that reviewed CPUs, and others that explained how a CPU works. Moving the slider all the way to the right bumped definition and “how does it work?” sites right to the top, with CPU-World at the top followed by Wikipedia’s definition of CPU. In this case, it’s a subtle difference, but not entirely insignificant.

Moving the slider all the way to the left, however, gives us very different results. We start getting results from Newegg, Computer Geeks Discount Outlet, and many other stores. It isn’t perfect by any means; the fourth result was for California Pacific University, which probably wouldn’t have turned up if I had typed in “central processing unit” rather than just the initials. Still, it is a very nice little beta, and could become quite helpful to the creation of customized search.

{mospagebreak title=Predict the Future of Technology}

Some folks like fantasy football leagues; Yahoo! has a fantasy stock market in beta. You don’t buy and sell fantasy shares of companies here – or at least, that’s not all you buy and sell. You can buy and sell stock in hurricanes, software distributions, gadgets, web services, and more. It’s called the Tech Buzz Game, and it’s a joint venture between Yahoo! Research and O’Reilly Media.

The game’s submarkets pit certain rival technologies, each represented by a stock, against each other. For instance, seven stocks currently make up the Browser Wars market, including Internet Explorer, Firefox, and Opera, among others. Players have access to the “buzz” around each technology. “Buzz” is measured, as you might expect, by the number of Yahoo! visitors who use the search engine to find information on that item. It’s probably more accurate to call this a “futures market” than a “stock market.”

The game gives a light-hearted example of a player being bullish on podcasting, and buying shares of the PODCAST stock. Britney Spears then announces that her next single will be available only via podcast. Curious young fans flood Yahoo! with searches about podcasting. The player’s stock shoots up. 

What is the point of the game, however? It’s not played with real money, after all. Well, there are two goals. First, O’Reilly wanted to study the power of prediction markets to forecast high-tech trends. This has been done before in a rather different way, and for entertainment rather than research purposes; you may recall the website F**kedCompany, whose users predicted how much longer various technology companies would last, based on news they heard (in some cases from disgruntled workers within the companies themselves). If such markets proved to be accurate predictors, the results could be used in a variety of ways, from advising companies about what technologies to deploy (and avoid deploying) to helping a publisher decide what topics and authors to pursue.

Yahoo!, in the meantime, wanted to field test the dynamic pari-mutuel market, which is a trading mechanism created by Yahoo! Research and designed to price and allocate shares. The paper it is based on speaks of it as a “mechanism for risk allocation and information allocation.” Despite having something of the appearance of a stock market, it doesn’t quite work that way. “Since the mechanism is pari-mutual (i.e., redistributive), it is guaranteed to pay out exactly the amount of money taken in.” Since companies are capable of creating new value as well as losing value, the actual stock market is not a zero-sum game. On the other hand, the fact that The Buzz Game is zero-sum probably makes it easier to distinguish predictions.

{mospagebreak title=What’s Next at Yahoo!?}

I’m going to focus here on what appears to be the most recent beta. Yahoo! Maps is one of the many services in beta you can find at Yahoo! Next. It gives you a place to put in two addresses to the right, and (by default) a map of the United States to your left. Putting in two addresses gives you text directions and marks your route on the map.

You can manipulate the map, but it is not quite as intuitive as Google Earth. Yahoo! shows you a small box. This box includes a smaller version of the map that is showing in the larger screen, plus a significant part of the surrounding area that is not in the screen. This box contains a grey rectangle, which you can move over any portion of that smaller map. Moving the rectangle changes what is showing in the larger screen (indeed, the area covered by the grey rectangle is the exact area that is shown in the screen).

You need to really watch where the grey rectangle is when you zoom in and out. Fortunately, this is not difficult. There is a slider control on the right side of the box, with a plus sign at one end and a minus sign at the other end. Zooming in and out is done by dragging the slider in the appropriate direction. Needless to say, these are maps; there is no satellite photography involved.

One thing that is nice is that you can click on a “Live Traffic” link, and the map will show you where all the incidents are. It will tell you what time the map was last updated, and if you run your mouse over any particular incident, it will give you the time of the incident, exact location, whether there are any injuries, whether any lanes are blocked…in this particular case, the same information that the Florida Department of Transportation has.

I didn’t like the fact that you apparently couldn’t move the box to other parts of the screen, though keeping it on the upper left did seem to keep it out of the way. Fortunately, there was an easy way to tell Yahoo! about it; a good-sized, clearly labeled button on the bottom right takes you to a feedback form. It asks for your name and your email address, but the latter, at least, is optional.

In short, Yahoo! may approach things a little differently from Google, but it should not be underestimated. The company has some really bright ideas. It will be interesting to see what it gives us in beta next.

[gp-comments width="770" linklove="off" ]