How to Conduct Competitive Research

Every time Google updates its algorithm, it presents a new set of obstacles for webmasters and SEOs alike to overcome. Whether it is a change in SERPs (search engine results pages), position, PageRank, or inbound links, one thing is immediately clear: it takes a huge toll on us to live in fear that our website will drop into complete obscurity. Instead of scurrying to fix your website every time an update happens, there may be some things we can do proactively to make sure that our organic search optimization efforts aren’t entirely wasted every three or four months.

First and foremost, you should be aware that during the first week or so of an update, many things will fluctuate before evening out. So if your site suddenly drops 50 places or tanks completely, there is no reason to hit the bottle just yet.  There is something you should be doing in the meantime however, and that is research. Do keyword research, web statistics tracking, and continue to monitor position reports in other search engines to understand the entire picture. Because Google commands such a huge share of the search engine market, it’s hard not to be fazed when a shakeup or shakedown occurs, but you really shouldn’t panic.

While there are many different elements to good SEO, one element that doesn’t get enough credit on its own standing is competitive research. What is competitive research?  It is the study of your competition, more specifically, the competition that is beating you out of top position in the SERPs. In this article, I want to point out a few techniques you can use to concentrate on beating out your competition for top positioning.

Good research involves good research tools.  Successful SEO efforts are not necessarily about the project itself, but rather how well the website’s SEO relates to the other websites around it in the scope of its relevant keywords, and how well it receives popular “votes” from other websites. This is what creates the pecking order of SERPs, so to speak. Competitive research is very important when you are attempting to get a good position in a search engine because you have to know everything about your enemy, so to speak. Any marketer will tell you that strategic planning in regards to your competition is one of the first steps to determining what your target audience is looking for, especially in a highly saturated market.

I’m not necessarily talking about studying your particular industry as a whole, especially not initially, but your direct competition, as well as the top dogs. And as much as you might like to just blow everyone out of the water, more often than not it is a step by step stair-climbing process; sometimes it’s more like watching paint dry or your fingernails grow. If you target your immediate competitors first, then you have a better chance of not only working you way to the top but also feeling pretty good about your progress with measurable results. Biting off more than you can chew is not only inviting disappointment, but you could be setting yourself up for complete failure.

{mospagebreak title=Beginning Your Research: Where do You Rank?}

So what’s involved in competitive research? The first step in competitive research is to figure out where you are in the SERPs for certain core keywords. This sounds difficult, and if approached manually, it is almost impossible to track. So you need the right kinds of tools to help you here. One tool that’s available commercially is Internet Business Pro, or IBP for short, from Alexa. You simply enter your keywords and website, and it can find your site within the top 500 for particular keywords, if there. 

There are other Internet tools available for similar results, and even websites that can check within the top 500 to 1000 results, as long as you obtain a Google API key, which is free from Google and allows you to make up to 1000 searches per day. (If you have multiple email addresses, you could obtain multiple API keys).  One website with a tool which is useful for checking positioning in Google, Yahoo, and MSN is It’s free, and you don’t need to sign up for any type of account. You will need a Google API key, though.

You should be aware, however, that depending upon which data center you check in Google and where you are located, you could easily get conflicting reports. The search engine will route queries to the nearest data center from your location, and each data center has a different set of information. In a recent interview with Matt Cutts, he stated, “In fact, even at different data centers we have different binaries, different algorithms, different types of data always being tested.” If you would like to see where you rank in a different part of the world, consider using an IP proxy while searching.

Further, if you have signed up and are signed into a Google account, it will also affect your returned results. Aaron Wall from corroborates this point. “If you are logged into a Google Account they will bias your search results based on websites you have visited, especially those you have clicked through to from search results.

“If you visit a site or page frequently they will improve the positioning of that page in your personalized search results. If you visit a page occasionally just rank checking, and then sometimes clicking onto your result then clicking back nearly immediately Google will demote those pages in the SERPs.”

Once you’ve located your ranking, find the sites that rank in the top ten positions for those keywords. Now I know I said that it was a step by step process, and of course it is; you will still want to be aware of what the best of the best are doing.  The next few steps will help you understand why and how they are achieving top results, and will allow you to follow their lead so that you can do the same.

{mospagebreak title=Keyword Choices, Inbound Links}

Keyword Choices and Density – You should know what your core keywords are before you decide which sites you will tackle for competition on these keywords.  A good way to do this is to use a program that will literally extract all of the words from your web page, and choose your keywords from the resulting list. You should choose words and phrases that not only relate to the overall theme of the page, but also words and phrases that appear continuously and consistently.

If you are looking at increasing your keyword density, you can substitute your keywords into existing words and phrases within your content. For example, if my core keyword is “search engine optimization,” I can substitute “SEO” or “a project” with “search engine optimization.”  Ideal keyword density varies per search engine, so it’s best to average them among the top three: Google, Yahoo, and MSN.  To make it more complicated, it appears that keyword densities also vary depending upon what the keyword actually is!

Because of this, it’s difficult to give you a set number for your density percentage.  TIP:  If you don’t know what your keyword density should be, all you need to do is look at the sites in the top ten of your keywords, run the extraction tool on that site or page, then find the average density of your competitors by first dividing the number of keywords by the total number of words on the page, then average that among the top ten sites. Then you’ll need to go one step further and do this for the other two search engines, and once you’re done there, you should average them all together.  Make sure you don’t go into overdrive: over-optimizing keyword density can work against you, and can appear as spammy results to a search engine.

Inbound Links – The higher your inbound link count, the better your chances are of reaching the top of the SERPs.  If you look at the top ten sites for your core keywords, you’ll see that most of them have many inbound links.  Even more specifically, they will have many QUALITY inbound links, not just sheer numbers of them. It’s really the quality of those links that count more than the amount.  If you use Firefox, you can download and install a browser plug-in called SEO Open, which with a click of a button on your tool bar can check for backlink counts in Google, Yahoo, MSN, Alexa, and has other tools that are very useful for SEO.

If you don’t have Firefox, you can always find a different tool, or check the inbound links manually by typing into your browser address bar: for Google, Yahoo and MSN, or site backlinks with for Yahoo and MSN.  There are many tools available to do this for you; just do a search for backlink tools.

{mospagebreak title=Navigation, Site Structure, PageRank}

Navigation and Site Structure – While it’s very time consuming to visit a website and to click on all of the pages in order to determine structure and navigation, it’s going to be necessary to find out the way these sites are structured and how that affects their position in the SERPs. The best way to do this is with a tool that can crawl the site similarly to the way a search engine spider can. A good tool for this is GSiteCrawler, which crawls a site to extract links for developing Google Sitemaps, or Xenu, which crawls a site to check for broken links.

PageRank – Since PageRank is nothing more than a reflection of relevancy factors, which includes mostly backlink quality, you honestly don’t need to pay too much attention to this as far as SERPs go. A site with a PR of 5 can easily rank higher than a site with a PR of 7 or 8. It’s inconsequential what your PR is in regards to your competition, unless of course you are sitting at a PR of 0. Frankly, I don’t pay much attention to PageRank anymore.

Walk Away – Once you’ve been able to incorporate the elements that you need to beat out your competition, you should sit back and relax a bit and just watch and wait. The best advice I can give you here is to walk away for now. What?!  Yes, you heard me. You need to walk away. If you sit and continually check your positioning, then you will absolutely drive yourself nuts on a daily basis. It’s similar to investing in the stock market. While there are no guarantees with either, you will find that with some smart research, and knowing everything about your market and your competition will help as well as follow good SEO practices, you can know that your site will indeed climb. But don’t be surprised if your site dips from time to time. The important thing is to track is a steady climb overall, and not your daily rankings.

Hopefully this article gives you a great start to being able to beat out your competition in the search engines, and the next time the update happens, you won’t be nearly as freaked out as you could be.

[gp-comments width="770" linklove="off" ]