Most Web sites with a large number of pages are developed using dynamic server-side Web technologies such as Hypertext PreProcessor (.php), Java Server Pages (.jsp), Active Server Pages (.asp), ColdFusion (.cfm) and Perl. These technologies provide programmers with the tools to build sites so that adding product or pages does not require extensive HTML work. In reality, all high volume sites must use one of these technologies in order to maximize efficiency and stay profitable.
The power and flexibility provided by these technologies is outstanding. There are, however, issues that need to be addressed in regards to the way search engines crawl a dynamically driven Web site. These issues do not have to do with the pages that are generated, but with the URLs these technologies generate.
The solution we came up with was to program a Mod_Rewrite on the URLs to remove the stop characters from the URLs. We modified a URL that once looked like http://www.freshpair.com/catalog.php’formid=5&query=bra§ion=women to something more like http://www.freshpair.com/catalog_section_women_id_8.html. We replaced all stop characters with underscores and more friendly URL characters and names. Today, Google has indexed over 21,000 pages on Freshpair.com and sales have increased tremendously due to the Mod_Rewrite and other search engine optimization techniques applied to the site.
For more information on Mod_Rewrite please visit the Apache module mod_rewrite page at: http://httpd.apache.org/docs/mod/mod_rewrite.html