If you did start to click on later pages, you would find out that the results they say they found aren’t actually there. Instead, you get a message saying that they have omitted around 90-99% of the websites that came up, and that message will be repeated on every page you try after the initial results they chose to include. What’s going on? Does that mean they don’t really have those results?

What Search Query Results Really Mean

One of the reasons that engines like Google or Bing can find so many results is that they don’t bother to collect them for your use. The chances that you will need them are slim to none, and even if you did by chance require a deeper web page, you wouldn’t be able to find it. There are just too many to sift through. Instead, you would have to go back and narrow down your search terms to get a better list of choices, something we are all pretty used to doing by now.

But those results still do exist. The milliseconds it takes to conduct a search are that quick because it has sorted the most relevant sites based on your search terms. These use a ranking system to put them higher on the list, while other sites are left behind. If they were to offer you the billions of websites they have in their database that contain your search term, it would take a very long time to load them, indeed. Just imagine the less-than-one-second loading time for around 700 results. Not try to imagine applying that same time frame to billions.

Yeah, it wouldn’t work out very well for the average user that doesn’t want to spend months getting an entire internets worth of websites they will never need, never use and never be able to search through.

In other words, what the search results found number really means is how many mentions of that keyword they have in their database. Not how many results have been cultivated for your personal use from the search engine. Which is something I am sure we can all be thankful for.

Image Credits: 1

Author information

Ann Smarty

Community Manager at Internet Marketing Ninjas

Ann Smarty is the pro blogger and guest blogger, social media enthusiast.

TwitterFacebookGoogle+LinkedIn

The post The Number of Google Results Found: What It Really Means appeared first on SEO Chat.

Perhaps a better question to ask is this: how do you know that Google has crawled and indexed all of your website’s pages? If you’re a little guy, that’s easy; you can count that high. But those with bigger sites can easily lose track, as Mike Moran points out. And if you don’t at least have some kind of an estimate for the number of pages on your website, you don’t know if the search engines left any of them out of their indexes – or, indeed, just how many didn’t make it in.

I’ve been working at SEO Chat now for nearly eight years, and I can’t begin to even guess how many pages we encompass. I’m not proud of this, especially since I’ve written so many articles during that time (and edited most of the rest). I know I’m not alone in facing this kind of problem. Fortunately, Moran mentions four ways to get an answer to this burning question.

Perhaps the easiest way is to ask your webmaster how many pages your website contains. The fine folks in IT are pretty much required to know this information because of what they do every day pertaining to your website. Indeed, they’ve probably been asked this before, and have the answer close to hand.

If you don’t want to go to your webmaster, you can try going to your content management system (CMS). Most of these will tell you how many pages they are currently handling. If you use more than one CMS, make sure you query them all. Moran notes that “Even a free CMS such as WordPress can do this,” leaving you with few excuses for not collecting this information.

But suppose for some reason you can’t do this. You still need to know how many pages your website includes. There are programs that can help you with this. They will spider your site, rather like the search engines spider your site for indexing. A program such as Xenu will even turn up pages that you might otherwise overlook.

And if worse comes to worst? Well, you can take a guess. Before you stare at me in shock, remember that “it’s better to  hazard a guess than to just throw up your hands,” according to Moran. Why? Well, I have to assume that you still care at least somewhat about your finances if you’re running a business with a website – right? So, don’t you want to know what you’re spending? Of course you do. Unless you’re running some kind of hobby site as a labor of love with free hosting and unlimited bandwidth (and even that isn’t truly “unlimited” these days), then every website page costs money. If you can estimate the number of pages your website includes, you can better understand how much money you’re spending on it. As Moran points out, “A guess is better than nothing.” 

Author information

Terri

The post Get a Page Count for Your Website appeared first on SEO Chat.

 Below is a list of five reasons search engine spiders may not crawl your site:

•   Flash or Java Links
•   robots.txt
•   Too Many Link on One Page
•   Links in Forms
•   Links in frames

Flash or Java Links

While Google assures us that Flash is becoming more crawlable, for the time being, you will probably want to avoid relying on embedding links in Flash. If you do it, be sure to have a link outside of Flash on the page as well. The same goes for Java, Silverlight, and any other type of plug-in.

Robots.txt

I wrote an article a while back concerning the uses of robots.txt that may be helpful to check out if you are experiencing a lack of indexing. Basically, the robots.txt file tells the spiders that they may not access the information on a given page (ie; crawl the page).

There can be many reasons why you would not want a page to be available to the engines, but still want it accessible to certain individuals. For instance, you may wish to allow only certain people to view your portfolio page via a direct link, and not have it available to anyone with access to Google.

Too Many Links on the Page

For what should be obvious reasons, piling a million links on a page is a bad idea. For one, it is too confusing to the user. Second, it is usually a sign that you are a spammer or trying to put one over on the search engines. Spiders crawl maybe  100 or so links on any given page. Any more than that puts you at risk of not getting crawled.

Links in Forms

If you have users fill out a form prior to viewing a page, and that is the only way to access the page, be forewarned — it is not likely that the page will be crawled. The reason for this is, bots do not like to provide personal information, nor do they like to click on submit buttons. Call it bigotry if you like, but always be sure to include a link to your content that does not require a form to be submitted.

Links in Frames

Sometimes when you use frames on your page, Google, Yahoo, and Bing get confused by the structure of your site, and that confusion can lead to a traffic jam. Besides ruining your SEO, frames, to me, are just hideous anyway. Avoid using them to avoid this common pitfall.

Author information

Jamesp

The post Search Engine Spiders: Why Don`t They Crawl My Site? appeared first on SEO Chat.

The search engine wars are fought with strategies, alliances, and robots. As Yahoo! primes itself to be the number one contender for market share after Google, websites that want to optimize for Yahoo must study how Yahoo ranks pages and how it indexes pages. The Yahoo web crawler SLURP should be studied; your site server logs should have recorded visits from various robots, including SLURP. If you do not have records of SLURP visiting your site, then this article will give tips on how to get SLURP to crawl (hopefully deep crawl) your site.

The Preamble

Yahoo SLURP evolved from Inktomi SLURP. The Yahoo SLURP robot is an upgrade from Inktomi’s SLURP. Yahoo used Inktomi’s search engine to replace Google, which used to take care of its search results. This officially triggered the second search engine wars (the first was won by Google without it declaring hostilities).

Yahoo has at least 130 million registered users on its network. Granted, Google is the definitive search engine, but Yahoo is large enough that it should not be ignored.

SLURP crawls websites, scans their contents and meta tags, and travels down the links contained on the page. It then brings back information for the search engine to index. Yahoo SLURP 2.0 stores the full text of the page it crawls in its memory and then returns to Yahoo’s searchable database. This is one of the semi-unique points of Yahoo SLURP; not all search engine crawlers store the entire text of the pages they crawl.

While SLURP has some features unique to it, it also obeys the robots.txt command. This command is very important since it ensures that you have control over which pages the crawler searches and indexes. This lets you protect the sensitive pages which you need to keep secure, pages which contain information you would rather not have in the hands of hackers (who regularly try and infiltrate search engines databases), or pages which you don’t want indexed at all (for whatever reason).

Another good thing about the robots.txt file is that it enables you to exclude specific robots, so you can inhibit the Googlebot but enable SLURP to crawl a particular page. This can be useful if you have optimized different pages for separate search engines. This may occur in order to give you flexibility, but a search engine may think you have duplicate pages and may penalize you. So careful use of the robots.txt file should definitely be on our list of how to make your website more search engine friendly. So how do you use the robots.txt file? You open notepad and type in the following lines:

  User-Agent: Slurp
  Disallow: whatsisname.html
  Disallow: page_optimized_for_google.html
  Disallow: credit_card_list.html
  Disallow: whatnot.html

Save it as robots.txt and upload it into your root directory. You can disallow as many pages for each crawler robot as you want, but to disallow certain pages for another crawler, you start a new line of code.

  User-Agent: Slurp
  Disallow: whatsisname.html
  Disallow: page_optimized_for_google.html
  Disallow: credit_card_list.html
  Disallow: whatnot.html
  User-Agent: Googlebot
  Disallow: page_optimized_for_yahoo.html
  Disallow: credit_card_list.html
  Disallow: whatnot.html

If you want to disallow all crawlers, you replace the name of the user agent with the wildcard command (*)

Robots.txt is useful for not getting banned on search engines and can also be used to pinpoint crawlers when they come calling. Only crawlers request Robots.txt, and these requests show up on the server logs.

Another way of shutting out SLURP is by using the noindex meta-tag. Yahoo SLURP obeys this command in the document’s head, and the code inserted in between the head tags of your document is

  <META NAME=”robots” CONTENT=”noindex”>

This snippet will ensure that that Yahoo SLURP does not index the document in the search engine database. Another useful command is the nofollow meta-tag. The code inserted is

  <META NAME=”robots” CONTENT=”nofollow”>

This snippet ensures that the links on the page are not followed.

Dynamic Page Indexing

This is the real charm of SLURP. Most search engine crawlers don’t bother crawling and indexing dynamic pages (.php, .asp, .jsp) since their content is subject to rapid change, which makes the process of indexing useless. Yahoo SLURP, however, does daily crawls in order to refresh the content on their indexed dynamic pages. It also does bi-weekly crawls which enables the search engine to discover new content and add it to its website incrementally. This enables a complex site’s URLs, generated by forms and content management software, to be indexed.

This frequent crawls show up in your server logs as frequent download requests, as the crawler moves, stops, and restarts. Yahoo says that these frequent download requests should not be a cause for alarm.

SLURP’s ability to index dynamic pages and to constantly refresh its content is a great relief to web designers (like me) who like having dynamic pages to enable fast loading and rapid updating. Websites which were not search engine friendly are suddenly in contention to be ranked number one.

However, the down side to this is that SLURP may never deliberately crawl your dynamic pages, unless you trigger the crawler via techniques which Yahoo encourages (to the benefit of their bottom line).

Getting Framed

Yahoo SLURP also has the ability to support frames, although it will not follow the SRC tag links to stand alone framesets; it only follows the HREF tags (as all good crawlers do).

After having said all this about Yahoo SLURP, there is now the little issue of getting your site crawled by this particular search engine spider. There are some ways to go about this task, and here we begin to see the inklings of what would be the order of the day in a search engine market dominated by Yahoo! (who seems to be very, very concerned about its bottom line).

Linking

The first strategy is good old linking; just get a link on a site on which Yahoo! regularly crawls, and voila. You have SLURP knocking on your door. This can be done by corresponding with a site which ranks well on Yahoo, or by submitting your web site to directories which SLURP regularly crawls (you can find these by searching for “directories” on Yahoo). If SLURP deep crawls (crawls lots of pages instead of just one or two) your site regularly, you have a high chance of getting a good ranking on the key word or topic for which you have optimized your site.

Yahoo Companion Toolbar

This is supposed to trigger the SLURP robot to crawl your site. And it also enables searchers to search within your site, offering value for your audience and attracting Yahoo SLURP as well.

Sitematch

This is done by paying Yahoo’s fees and submitting your site. This guarantees you will be added to the index (at a price) but is no guarantee of your website’s ranking in the SERPs.

This is a scary service, and some reviewers speculate that it is a foretaste of what site owners would face in a market dominated by Yahoo. It is carried over from Overture (which Yahoo purchased) and involves an annual fee for submitted pages. The URLs are submitted into Yahoo’s index and are then crawled by SLURP every 48 hours.

However, apart from the one off fee, there is a cost per click fee charged for each lead driven to your site (so you better have deep pockets)

Apart from SLURP visiting every two days, you also get listed on searches done on about.com, Excite, Overture and other Yahoo partners. However there is no guarantee of a high ranking, and frankly I do not like this method (because I absolutely love free stuff).

There is a way to submit  your site for free,  however Yahoo does not guarantee that websites submitted through such means will ever be crawled by SLURP.

By now you should know enough about SLURP to spot it, track it, attract it, and prevent it from crawling specific pages of your site.

Author information

AkinolaAkintomide

The post The Yahoo SLURP Crawler appeared first on SEO Chat.

You have a masterful website, with lots of relevant content, but it isn’t coming up high in the search engine results pages (SERPs). You know that if your site isn’t on those early pages, searchers probably won’t find you. You can’t understand why you’re apparently invisible to Google and the other major search engines. Your rivals hold higher spots in the SERPs, and their sites aren’t nearly as nice as yours.

Search engines aren’t people. In order to handle the tens of billions of web pages that comprise the World Wide Web, search engine companies have almost completely automated their processes. A software program isn’t going to look at your site with the same “eyes” as a human being. This doesn’t mean that you can’t have a website that is a joy to behold for your visitors. But it does mean that you need to be aware of the ways in which search engines “see” your site differently, and plan around them.

Despite the complexity of the web, and dealing with all that data at speed, search engines actually perform a short list of operations in order to return relevant results to their users. Each of these four operations can go awry in certain ways. It isn’t so much that the search engine itself has gone awry; it may have simply encountered something that it was not programmed to deal with. Or the way it was programmed to deal with whatever it encountered led to less than desirable results.

Understanding how search engines operate will help you understand what can go wrong. All search engines perform the following four tasks:

• Web crawling. Search engines send out automated programs, sometimes called “bots” or “spiders,” which use the web’s hyperlink structure to “crawl” its pages. According to some of our best estimates, search engine spiders have crawled maybe half of the pages that exist on the Internet.
  
  
• Document indexing. After spiders crawl a page, its content needs to be put into a format that makes it easy to retrieve when a user queries the search engine. Thus, pages are stored in a giant, tightly managed database that makes up the search engine’s index. These indexes contain billions of documents, which are delivered to users in mere fractions of a second.
  
  
• Query processing. When a user queries a search engine, which happens hundreds of millions of times each day, the engine examines its index to find documents that match. Queries that look superficially the same can yield very different results. For example, searching for the phrase “field and stream magazine,” without quotes around it, yields more than four million results in Google. Do the same search with the quote marks, and Google returns only 19,600 results. This is just one of many modifiers a searcher can use to give the database a better idea of what should count as a relevant result.
  
  
• Ranking results. Google isn’t going to show you all 19,600 results on the same page – and even if it did, it needs some way to decide which ones should show up first. Thus, the search engine runs an algorithm on the results to calculate which ones are most relevant to the query. These are shown first, with all the others in descending order of relevance.

Now that you have some idea of the processes involved, it’s time to take a closer look at each one. This should help you understand how things go right, and how and why these tasks can go “wrong.” This article will focus on web crawling, while a later article will cover the remaining processes.

You’re probably thinking chiefly of your human visitors when you set up your website’s navigation, as well you should. But certain kinds of navigation structures will trip up spiders, making it less likely for those visitors to find your site in the first place. As an added bonus, many of the things you do to your site that will make it easier for a spider to find content, will often make it easier for visitors to navigate your site.

It’s worth keeping in mind, by the way, that you might not want spiders to be able to index everything on your site. If you own a site with content that users pay a fee to access, you probably don’t want a Google bot to grab that content and show it to anyone who enters the right keywords. There are ways to deliberately block spiders from such content. In keeping with the rest of this article, which is intended mainly as an introduction, they will only be mentioned briefly here.

Dynamic URLs are one of the biggest stumbling blocks for search engine spiders. In particular, pages with two or more dynamic parameters will give a spider fits. You know a dynamic URL when you see it; it usually has a lot of “garbage” in it such as question marks, equal signs, ampersands (&) and percent signs. These pages are great for human users, who usually get to them by setting certain parameters on a page. For example, typing a zip code into a box at weather.com will return a page that describes the weather for a particular area of the US – and a dynamic URL as the page location.

There are other ways in which spiders don’t like complexity. For example, pages with more than 100 unique links to other pages on the same site can make them get tired with just one look. A spider may not follow each link. If you are trying to build a site map, there are better ways to organize it.

Pages that are buried more than three clicks from your website’s home page also might not be crawled. Spiders don’t like to go that deep. For that matter, many humans can get “lost” on a website with that many levels of links if there isn’t some kind of navigational guidance.

Pages that require a “Session ID” or cookie to enable navigation also might not be spidered. Spiders aren’t browsers, and don’t have the same capabilities. They may not be able to retain these forms of identification.

Another stumbling block for spiders is pages that are split into “frames.” Many web designers like frames; it allows them to keep page navigation in one place even when a user scrolls through content. But spiders find pages with frames confusing. To them, content is content, and they have no way of knowing which pages should go in the search results. Frankly, many users don’t like pages with frames either; rather than providing a cleaner interface, such pages often look cluttered.

Most of the stumbling blocks above are ones you may have accidentally put in the way of spiders. This next set of stumbling blocks includes some that website owners might use on purpose to block a search engine spider. While I mentioned one of the most obvious reasons for blocking a spider above (content that users must pay to see), there are certainly others: the content itself might be free, but should not be easily available to everyone, for example.

Pages that can be accessed only after filling out a form and hitting “Submit” might as well be closed doors to spiders. Think of them as not being able to push buttons or type. Likewise, pages that require use of a drop down menu to access might not be spidered, and the same holds true for documents that can only be accessed via a search box.

Documents that are purposefully blocked will usually not be spidered. This can be handled with a robots meta tag or robots.txt file. You can find other articles that discuss the robots.txt file on SEO Chat.

Pages that require a login block search engine spiders. Remember the “spiders can’t type” observation above. Just how are they going to log in to get to the page?

Finally, I’d like to make a special note of pages that redirect before showing content. Not only will that not get your page indexed, it could get your site banned. Search engines refer to this tactic as “cloaking” or “bait-and-switch.” You can check Google’s guidelines for webmasters (http://www.google.com/intl/en/webmasters/guidelines.html) if you have any questions about what is considered legitimate and what isn’t.

Now that you know what will make spiders choke, how do you encourage them to go where you want them to? The key is to provide direct HTML links to each page you want the spiders to visit. Also, give them a shallow pool to play in. Spiders usually start on your home page; if any part of your site cannot be accessed from there, chances are the spider won’t see it. This is where use of a site map can be invaluable.

I’ll assume that you are all reasonably familiar with HTML. If you have ever looked at the source code for an HTML page, you probably noticed text like this wherever a hyperlink appeared:

When a web browser reads this, it knows that the text “SEO Chat” should be hyperlinked to the web page http://www.seochat.com. Incidentally, “SEO Chat” in this case is the “anchor text” of the link. When a spider reads this text, it thinks, “Okay, the page http://www.seochat.com is relevant to the text on this page, and very relevant to the term `SEO Chat.’”

Let’s get a little more complicated.

Now what? The anchor text hasn’t changed, so the link will still look the same when the web browser displays it. But a spider will think, “Okay, not only is this page relevant to the term `SEO Chat,’ it is also relevant to the phrase `Great Site for SEO Info.’ And hey, there’s a relationship between the page I’m crawling now and this hyperlink! It says that this link doesn’t count as a ‘vote’ for the page being linked to. Okay, so it won’t add to the page rank.”

That last point, about the link not counting as a vote for the page being linked to, is what the rel=”nofollow” tag does. This tag evolved to address the problem of people submitting linked comments to blogs that said things like “Visit my pharmaceuticals site!” That kind of comment is an attempt by the commenter to raise his own website’s position in the search engine rankings. It’s called comment spam, by the way; most major search engines don’t like comment spam because it skews their results, making them less relevant. As you may have guessed, then, the “nofollow” tag in the “rel” attribute is specifically for search engines; it really isn’t there to be noticed by anyone else. Yahoo!, MSN, and Google recognize it, but AskJeeves does not support nofollow; its crawler simply ignores the nofollow tag.

In some cases, a link may be assigned to an image. The hyperlink would then include the name of the image, and might include some alternate text in an “alt” attribute, which can be helpful for voice-based browsers used by the blind. It also helps spiders, because it gives them another clue for what the page is about.

Hyperlinks may take other forms on the web, but by and large those forms do not pass ranking or spidering value. In general, the closer a link is to the classic <a href=”URL”>text</a>, the easier it is for a spider to follow a link, and vice versa.

Author information

Terri

The post How Search Engines Work (and Sometimes Don’t) appeared first on SEO Chat.

In the quest for that elusive nirvana of search engine friendliness, we frequently find ourselves searching for “instant fix” ways to improve a page’s ranking without considering the big picture; that is, without looking at the problem of optimizing a web page as a whole and instead looking at several separate optimization steps as part of routine markup development or copy writing. While SEO experts do not tend to fit this mold, the average web developer certainly does. How many web pages have you “optimized” by simply adding keyword and description meta tags, and stopped right there? I imagine a hand count at this point would supply a fairly substantial number. An even better question at this point might be, “How many of you have tried to provide SEO for a web page without having even a basic understanding of search robot logic or what it expects to see in your pages?” Once again, I suspect we would have a healthy hand count.

The steps to optimize a page are well known to the SEO community, and many articles by authors far more knowledgeable than myself on the subject are available to web developers. So with all this knowledge out there, why are there so many developers who lack a big picture understanding of the subject? One word: fundamentals. It is crucial to know how the technology behind the scenes works, but like any other skill, the bulk of people attempting to learn that skill do not start at the bottom. They start somewhere that makes sense in trying to solve a particular problem and then they build up from that point.

If a developer held a greater understanding of the fundamentals of SEO and how search robots went about crawling web pages, they would in turn have a greater understanding of how to populate those alt attributes and meta tags. The objective of this article is to provide a general overview of how search robots (also called spiders) go about crawling and indexing web pages.

There are a number of things that a spider expects to see when it looks at a web page, many of which are optional but still important in the big picture. The following is a list that describes the core elements considered by a typical search engine when calculating page relevance.

1. Title Tag – The title tag should contain a title relevant to the page, not just “Home Page” or “Contact Us”. The title should be used for up to five keywords.
   
   
2. Headings – Search engines view <h> tags as terms of emphasis, meaning additional weight is given to terms that appear inside them. Keywords should appear in <h> tags.
   
   
3. Bold – Also viewed as terms of emphasis, but with less weight than headings.
   
   
4. Alt Text – Brief descriptive sentences should be used in image alt attributes. At least one keyword should appear in each alt attribute.
   
   
5. Keyword Meta Tag – Some engines use the keyword meta tags directly, some use them as part of a validation process ensuring that the keywords closely match the page content. The latter is the more typical scenario for modern engines. Keywords should be chosen carefully and be specific to the page they appear on.
   
   
6. Description Meta Tag – Most search engines use this tag in a similar fashion as the keyword tags. Each page should have a unique description. The description should contain a few keywords and briefly summarize the content that appears on the page with a high degree of accuracy.
   
   
7. Keyword Placement – Terms that are higher up on a page are more heavily weighted.
   
   
8. Keyword Proximity – Terms that are close together are probably related, and thus the site will show up in searches for those terms.
   
   
9. Comment Tags – Some search engines use comment tags for content, particularly in graphics rich/text poor sites.
   
   
10. Page Structure Validation – Proper coding is likely to be of better overall quality, and thus rewarded.
    
    
11. Traffic/Visitors – Search engines keep track of how many people follow their links. The more a link is followed for a given search, the more relevant the link is assumed to be.
    
    
12. Link Popularity – Also known as PageRank, this is a measure of how many web pages on the Internet link to your site and the relevance of those pages to the page they are linking to. The popularity of the linking site is also evaluated.
    
    
13. Anchor Text for Inbound Links – This is a measure of the relevance of the anchor text from the referring site.
    
    
14. Page Last Modified – Newer content is regarded as “fresh” and is treated as more relevant.
    
    
15. Page Size – Engines tend to weigh content at the start of a document more than content further down. If a page is too long, typically more than 50k in markup only, then it should be broken up into multiple pages.
    
    
16. Keywords in URL – URLs are considered important by engines. Use of hyphens rather than underscores in filenames and using keywords in filenames and directories improves a pages potential relevance.

These elements are all poured into an algorithm by the search engine that produces a very specific result: a relevance score for a page based on a given set of keywords. Evaluating page relevance is a constant reciprocal process that involves crawling around all pages indexed by a particular engine and evaluating the relevance of their content and the relevance of references to that content. The items listed above are things search engines expect to find in a page as well as factors that are not necessarily expected, but are considered if available (such as inbound links).

The next set of variables weighed by search engines are negatives. These will negatively effect the performance of a page on a search engine without exception. Avoiding these items is crucial to reaching and, more importantly, maintaining a high rank on a search engine.

1. Broken Links – Internally or outgoing, search engines do not view pages with broken links as pages with fresh content, and are going to be scored as less relevant for their keywords.
   
   
2. Spam – This refers to any attempt to trick a search engine, such as using irrelevant keywords to draw extra hits, placing invisible content on the page to boost keyword density, and using meta refreshes (often in combination with irrelevant keywords) to draw a user in for an irrelevant search and then direct them to the page you want them to see. These techniques can result in a ban from search engines if they catch them.
   
   
3. Excessive Search Engine Submittal – Over submitting a site to a search engine will likely result in a ban. Submit no more than once every three months, according to Google.
   
   
4. Empty Alt Attributes – Empty alt tags is a major accessibility issue as well as just poor coding, and will affect a page negatively.
   
   
5. Excessive Punctuation – Excess punctuation in the Title and Description tags wastes valuable space and may cause a problem with the engine.
   

These negative factors could greatly effect an otherwise relevant page, of course some of them preclude the page actually being relevant, particularly spam. The biggest pitfalls for an otherwise optimized page is simple typographical errors, broken links (usually due to stale content) and oversight in markup. Simple mistakes could mean the difference between top ten and top fifty for a search on an engine, a difference that could mean thousands of dollars per day in lost revenue for many websites.

Imagine if a site like Amazon.com failed to use alt attributes and stopped using <h> tags (replaced by images, for example). Searches that would typically show the site as a number one result could start bringing the site up as a number fifty result.

Conclusion

Approaching SEO as a holistic process rather than simply a combination of steps is critical. It is simply not enough to use an effective Title tag on every page and stop, or to use keyword relevant URLs and then stop. To achieve and maintain top rating on all pages for the appropriate sets of keywords, a page must be optimized completely for the way search engines weigh content relevance, and that involves taking everything discussed earlier in the article seriously.

Author information

DavidFells

The post Spider Guts appeared first on SEO Chat.

Call them spiders. Call them robots. Call them bargain hunters (or one heck of a nuisance); they’re software programs with a mission: to hunt down information and bring it back. Many search engines couldn’t live without these electronic assistants to help them keep track of the proverbially explosive growth of the Web. Certainly they can save a lot of time when you’re trying to comparison shop online — just let a spider do the hunting and bring back the results. This is all well and good, unless the owner of the site doesn’t take kindly to spiders. eBay won an injunction against Bidder’s Edge that forced BE to stop sending spiders to eBay’s Web site in search of auctions. But a somewhat similar-looking court case was just recently decided in favor of the spider-wrangling plaintiff. Does this have wider implications for information aggregators who use spiders?

First, let me give my disclaimer: I am not a lawyer, nor do I play one on TV. So check with someone who eats, drinks, and breathes this stuff before you do anything drastic.  That said, let’s take a look at the cases at hand.

The more recent case was just decided early in April, in a district court in Florida. It involves two companies with Web sites that list yachts for sale…which may in part explain why this case did not attract the kind of attention that the earlier eBay case did. (There are more people interested in buying Beanie Babies than buying big boats.) Anyway, the older Web site in this case is owned by Boats.com, who, for the past nine years, has owned and operated Yachtworld.com, a Web site on which yacht brokers could post information about the big boats they have for sale — sort of like electronic classified ads, with more interactivity. Enter Nautical Solutions Marketing, in 2001, with their Web site, Yachtbroker.com — and two services that Boats.com complained blow them right out of the water.

The first service is the more interesting one for our purposes (though they’re both relevant) because it’s that service which directly involves the use of a spider. Cleverly named “Boat Rover,” this software program would connect to a targeted Web site, extract specific facts about a yacht for sale from a public yacht listing, collect those facts, and enter them in a searchable database. Boat Rover did not hold onto the HTML used in the listing; it copied it just long enough to get the facts, then discarded it. Yes, this matters — because this case was dealing with a question of copyright infringement. Specifically, did NSM infringe any copyrights when it sent Boat Rover to Yachtworld.com repeatedly between November, 2001, and April, 2002, to collect information which it then posted on its own Yachtbroker.com Web site?

Not according to Judge Merryday, who presided over the case. I’m sure you’d love to hear that it was a simple “no,” thus reaching a decision once and for all in favor of spiders running rampant, but anyone who’s ever had reason to consult a lawyer knows how unlikely that would be. Legal cases involving high technology, especially the Internet, tend to be less cut-and-dried. In this case, two important points were raised in consideration of whether any copyrights were infringed. First, what kind of information was Boat Rover collecting? In this case, it was just the facts: manufacturer, model, length, year of manufacture, price, location, and the URL of the Web page that contained the yacht listing. Well, fortunately for NSM (and Joe Friday), there’s an existing precedent that states that facts cannot be copyrighted; facts are part of the public domain, and thus there can be no question of copyright infringement. There’s no copyright to infringe!

The second aspect of this service that might have opened NSM up to charges of copyright infringement involved displaying these listings on its Web site. Remember when I mentioned that Boat Rover’s discarding the HTML information was important? That meant that NSM had to code the information without using Boats.com’s coding as a template — in theory, at least. And in fact, Judge Merryday found enough differences between the “look and feel” of a Yachtworld.com listing and a Yachtbroker.com listing to keep NSM in the clear over possible copyright infringement. The judge wouldn’t even grant Boats.com a copyright on its use of descriptive headings like “electrical,” “accommodations,” and “galley” to describe specific features of a yacht — because the terms were industry standards, and at least two other yacht brokering Web sites were using those terms in the very same way. The legal reasoning seems to go something like this: ideas themselves do not receive copyright protection. Expression of a particular idea does. However, in some cases, the ways to express a particular idea are so limited that the expression doesn’t receive copyright protection, because that would be just like protecting the idea. In this case, there’s only so many headings you can use for various areas of a yacht when listing it for sale, so — like facts — they don’t receive copyright protection.

The second service that NSM offered its customers was a “valet service.” With the permission of a yacht broker using the service, NSM would move, delete, or modify the yacht broker’s listing. Boats.com complained that NSM’s people were copying and pasting listings from Yachtworld.com over to Yachtbroker.com. The court found strong enough evidence that the only items being cut and pasted were descriptions and pictures, not HTML or anything that Boats.com could claim a copyright to. In fact, the court found (and both NSM and Boats.com agreed) that the yacht brokers themselves owned the copyrights to their own listings — and, since NSM was doing its thing with their permission, they weren’t infringing any copyrights.

Interestingly, this case was a “pre-emptive strike;” NSM was the plaintiff. They brought the suit seeking a declaration that they did not infringe any copyright owned by Boats.com. And they won, too.

I imagine you can all see how this has relevance for spiders and those who aggregate information about specific products…or even auctions. So, does this mean that spiders can go out and disregard those software-based warnings from Web sites telling them that they’re not welcome? Does this take eBay’s winning of an injunction against Bidder’s Edge and turn it on its head?

You’d think it would. On the face of it, the two cases look similar. Bidder’s Edge, an auction aggregation site, was sending spiders to eBay to collect information on eBay’s auctions. BE would then aggregate eBay’s auction information along with auction info from many other auction Web sites, making it a “one-stop shop” for that kind of information. eBay wanted them to stop — and, in fact, was able to legally force them to stop.

That is where the similarities between the two cases end. You see, eBay didn’t claim that Bidder’s Edge was infringing its copyright — or, at least, the court didn’t grant the injunction based on a claim of copyright infringement. Oh no. eBay claimed that Bidder’s Edge was trespassing! Bidder’s Edge admitted that it was sending 80,000 to 100,000 queries a day to eBay’s Web site, and eBay argued that that was akin to sending 80,000 to 100,000 robots into a bricks-and-mortar business looking for prices and not buying anything. Well, the court was skeptical of that argument, but not of certain other arguments. Specifically, eBay was able to point out that Bidder’s Edge’s spiders were using up eBay’s computer capacity, after eBay had told BE more than once that it was not welcome to use its spiders on eBay’s Web site. At that time, it was only using less than two percent of eBay’s capacity — but the principle in this case wasn’t the amount of capacity it was using up, but that it was using that capacity after eBay told it not to. A Web site might not actually be real estate in the same way that land is — but there is no question that computers and servers are property that can be owned. And Bidder’s Edge was using eBay’s computers in ways that eBay specifically told it that it wasn’t allowed to do. What would you do if you told someone not to use your computer in a particular way and they kept doing it? Uh-huh, that’s what I thought. In eBay’s case, they were granted an injunction against Bidder’s Edge in late May 2000.

So what does Nautical Solutions Marketing vs. Boats.com add to the sometimes-controversial issue of spiders? Well, first off, just because something was copied from a Web site — even by a robot — doesn’t mean that the person or company doing it is committing copyright infringement. Especially if it’s essentially factual information. On the other hand, that also doesn’t mean that they’re not in hot water. If you’re a spider wrangler, make sure your spiders check to see whether they’re welcome on the Web sites they visit — and don’t try to force the issue. It’s a big Web out there; there’s plenty of cyberspace for spiders to crawl without looking for trouble. 

Author information

Terri

The post Score One for the Spiders? appeared first on SEO Chat.

Despite recent improvement in tools and programs in the battle against SPAM, most of us cannot escape the menace that plagues most of our inboxes on a regular basis. Each day most of us probably receive more SPAM than actual real email, and with Spammers getting more and more creative in their ways to circumvent traditional anti-SPAM tactics, it’s vital webmasters empower themselves with some anti-SPAM tactics for their own websites.

In this article I will discuss a few ways to SPAM-proof your website against malicious SPAM robots that inevitably collect your email to be sold by the thousands to Spammers worldwide, whether it be for using your information inappropriately, or simply for no-good reasons.  These tactics are so effective that within a month of implementing them, you should see a dramatic drop in the amount of SPAM that makes it through to your website email addresses, not to mention a decrease in bandwidth.

How to Stop SPAMbots Dead in Their Tracks

1. Using JavaScript
2. Using Mod_Rewrite

Both of these techniques are effective in blocking SPAMbots and Spybots from finding your email address or other personal information on your website. While JavaScript is an easier solution, using mod_rewrite to block SPAMbots is more technical and requires knowledge of editing your .htaccess file. It’s best to try the JavaScript method first, and then venture into using mod_rewrite to further block SPAMbots from hitting your website.

Using JavaScript

To understand how to use JavaScript to block SPAMbots from harvesting your email, let’s examine the ways that they find your email in the first place.

1. Mailto: Links – these are common links placed in the HTML code of a website, offering a potential visitor the ability to send an email to the webmaster of the site.  A visitor clicks on the email link and it opens an email client with the To: field already filled in with the address specified in the code.  These links are the prime target of SPAMbots harvesting your email address, and simple use of JavaScript can cut down on email harvesters hitting your inbox with SPAM.  The main objective with using JavaScript is to change the appearance of your email address so that email harvesters do not recognize your email, but still retain complete functionality for legitimate visitors to send you an email.

2. Contact Forms - this is another prime location for SPAMbots to leave their tracks, steal your email address and be gone, ready to report back with fresh email addresses.  These forms are another common feature on websites, and the following is what most often causes SPAMbots to find your email.

<input type=”hidden” name=”recipient” value=”support@example.com”>

The following are examples of JavaScript that you can use to make your email address appear different in the code but still perform the same function as if it were regularly coded in HTML (ie: mailto:support@example.com).  To use these examples, just copy and paste the code into your HTML document and replace the required field(s) with your email address.

1. Basic Email Script

<script language=JavaScript>
<!–
document.write(“support” + “@” + “example.com”);
//–>
</script>

Result:  support@example.com

2. Basic Mailto: Email Script with Link Text

<script language=JavaScript>
<!–
var username = “support”;
var hostname = “example.com”;
var linktext = username + “@” + hostname;
document.write(“<a href=” + “mail” + “to:” + username +
“@” + hostname + “>” + linktext + “</a>”);
//–>
</script>

Result: support@example.com

3. Inline JavaScript

<a href=”#” onclick=”JavaScript:window.location=’mailto:’+'support’+'@’+'example’+’.com’”>Link Text</a>

Result: Link Text

The three scripts options above should give you some flexibility in how you choose to use these on your website.  Remember to insert your own email address into the fields where the support@example.com email address is located.

Problems Associated with JavaScript

There doesn’t appear to be many problems with using the above scripts in the HTML code of your documents.  The biggest issue may be incorrectly coding the scripts or issues with older browsers that do not support JavaScript. One last issue that may see its day in history is email harvester programmers being able to find email addresses among the JavaScript code.  While this may be a reality sooner than we expect, for the most part JavaScript should be SPAM-proof enough to block most malicious SPAM bots.

In this section, the use of mod_rewrite is very successful in blocking the SPAMbots and other spybots that visit the website with a mission to either steal your email address or grab information from your website without your permission. Consider this method as a step above using JavaScript, because it stops them before they ever read the webpage itself.  So if you are thinking of using JavaScript on the page to block bots from finding your email, consider the use of mod_rewrite as a primary defense weapon against SPAM and other malicious robots.

One note to readers: The use of mod_rewrite requires that you have it installed on your server, and you have the ability to edit the .htaccess file.  Below is a simple way to locate the .htaccess file while using a program such as CuteFTP (or a similar FTP client that performs the same functions).  If you are unsure whether you have mod_rewrite installed, you should first consult the server administrator with your primary hosting company.  Ask them if you have mod_rewrite and permissions to edit the .htaccess file.

How to Find .htaccess in a Common FTP Client

To locate the .htaccess file, most often you need to display all hidden files present when connecting to your hosting account.

To enable your FTP client to display all hidden files (.htaccess and many other files not normally seen by the user).

1. First locate your saved site properties.
   
   
2. Right click on the profile of the website you want to display hidden files. This is most often located in the “FTP Sites” section of most clients.
   
   
3. Once you right click on the FTP site, select “SITE PROPERTIES” from the menu.
   
   
4. An option box will load up displaying the site properties of your site. Look for a tab called “ACTIONS” and click on it.
   
   
5. It will display the actions of the site. Locate a gray box called “FILTERS” and click on it.
   
   
6. This will display the “Filters” properties of the site.
   
   
7. Locate the “Enable Filtering” from the options available. Make sure this box is checked.
   
   
8. Once you have checked the enable filtering box, a small box at the bottom of the options will be displayed.
   
   
9. It should say something similar to “Enable Server Side Filtering”. Make sure this box is checked as well.
   
   
10. Now enter the following into the “Remote Filter” box: -a
    

Once you have entered in the filtering options, make sure to click “Ok” or “Apply” in order to save your changes.  You should now be able to see all hidden files on the server.  Make sure you start a new connection to view all files.  If you are still having trouble viewing all your files and can’t seem to locate the .htaccess file, don’t give up, but consult the system administrator of your hosting account to assist.

How to Setup Your .htaccess File

Once you have confirmed that you do have a .htaccess file, and mod_rewrite is turned on, add the following lines to your .htaccess file:

Options +FollowSymlinks
RewriteEngine On
RewriteBase /

The robots that you will want to block will depend on your preferences, as well as any bots that frequent your website on a regular basis.  Cutting down on bandwidth costs, preventing robots from collecting your email address, and preventing robots from collecting information from you or your website are all good reasons to block a potential robot.

The best method of deciding which robots to block is to do some quick research about the robots that like to take residence on your site.  If you cannot find reliable information about a robot or its use of something you would not approve of, simply block the robot by using a robots.txt file.  If you find that a robot does not obey the robots.txt file, pull out the big guns and use mod_rewrite to stop them dead in their tracks.

Example Robots

There are several common bots that one might run into frequently such as “Microsoft URL Control” which is a robot that ignores the robots.txt file and fetches as many pages as it can before leaving the site.  This SPAMbot is used by many different people all using the same name. 

 The second robot that frequents websites is the NameProtect (NPbot) robot. This robot’s job is to collect information about websites that are potentially violating brand names of clients.  This robot does not obey the robots.txt file, responds to emails sent to the NameProtect company, and serves no good purpose as far as we have determined.

To Block the Microsoft URL Control Robot by User Agent:

RewriteCond %{HTTP_USER_AGENT} “Microsoft URL Control”
RewriteRule .* – [F,L]

To Block the Nameprotect Robot by User Agent:

RewriteCond %{HTTP_USER_AGENT} “NPbot”
RewriteRule .* – [F,L]

Furthermore, once you establish a good number of bots that you would like to block using mod_rewrite, you can compile a list and add comments as well, like so:

RewriteCond %{HTTP_USER_AGENT} “Microsoft URL Control” [OR] #bad bot
RewriteCond %{HTTP_USER_AGENT} “NPbot”
RewriteRule .* – [F,L]

One thing to note about using the examples here, make sure that you correctly know how to insert the script into mod_rewrite and that you do so in the proper rules required for this technique to be effective.  Additionally, one last thing to note is that mod_rewrite rules are not an ultimate solution to SPAM and malicious bot problems. You can, however, effectively block a good majority of bots out there and dramatically cut down on the amount of SPAM you receive. If you use the JavaScript methods and mod_rewrite then, not only will your website be one heavily guarded anti-SPAM site, but you may actually enjoy downloading your all email messages to find them SPAM free.

Author information

seo_admin

The post Protect Against Invaders by SPAM-Proofing Your Website appeared first on SEO Chat.

Author information

seo_admin

The post ROBOTS.TXT Primer appeared first on SEO Chat.

Author information

seo_admin

The post Designing Websites For Humans In A World Of Robots!! appeared first on SEO Chat.