Google`s Latest Moves in Information Indexing - Webmasters Unprepared
(Page 2 of 4 )
Several people who made comments to the Google blog itself pointed out a variety of issues. One big one is that webmasters may not be prepared for the googlebot's changing approach. SuperJason noted that "I may have to exclude googlebot from my error list. I'm guessing it's pretty easy for it to put in some bogus content." He also wondered whether Google didn't already have enough to index.
Another poster noted that it wouldn't entirely solve the problem of missing a lot of content, since so much of it is now being published using Ajax programming, which Google can't crawl yet. Olaf Lederer, yet another commenter, said that he feared the opposite problem: Google would find tons of "duplicate content" that isn't really duplicated, but merely appears to be because of this technique. That's a major concern, since Google penalizes duplicate content.
But Olly raised one of the strongest objections to the new crawling technique: "Inserting random text into form fields is just plain wrong. That's what spammers do." Since many webmasters block that kind of behavior for just that reason, some might end up blocking the googlebot altogether, which will hamper Google in its continuing mission to completely index the world's knowledge.
Danny Sullivan, however, lauded Google's new technology. "The move is potentially good for searchers, in that it will open up material often referred to [as] being part of the 'deep web' or 'invisible web' as it was hidden behind forms...It should be noted that Google's not the first to do something like this. Companies like Quigo, BrightPlanet and WhizBang Labs were doing this type of work years ago." Google is the first major search engine to do this kind of exploration, though.
Webmasters need to be aware that the googlebot just might explore past HTML forms, and act accordingly. If you have content that you do not want indexed, you need to take the appropriate measures to specifically block it; you can't use the shortcut of putting it behind an HTML form. And you should probably watch your logs a little more closely for unusual activity if this concerns you.
Next: The Second Search Box >>
More Search Engine News Articles
More By Terri Wells
