One of the most important objectives in search engine optimization is to ensure that the text content as intended to be shown to the readers is the same as seen by search engine bots. When search engine bots crawl text content which is not seen by normal visitors, this comprises what is called “hidden text.” At the opposite end, when search engine bots cannot read text that is seen normally by visitors, this introduces crawling issues.
At worst, you discover that search engine bots are not picking up your text content after all, which you never would have thought was happening, since your website looks very good to visitors. This is where using advanced tools to check “website crawlability” are important as part of the website design process and during the execution of onsite search engine optimization improvement.
One of the best of these tools is the Lynx browser. This is a text browser as opposed to normal browsers that read HTML, images and coded content. A text browser will display purely text and indexable text-based hyperlinks very similar to what search engine crawlers can see (such as Google bots). In fact, this text browser is recommended by Google to diagnose text crawling/search engine crawling issues.
You can read what Google has to say about hidden text. Likewise, if you’re looking for information on using Lynx, Google provides some technical guidelines. And if you’re still not convinced, you might want to check out a Google Webmaster blog entry on how spiders view the web.
As mentioned above, this is a two-part tutorial series on mastering the use of Lynx for search engine optimization. The first part discusses the installation, techniques and important features of Lynx. The second part will discuss the detailed interpretation of text content/crawling issues and troubleshooting using the text browser.
Using Lynx will ensure that your site is within search quality and technical guidelines for search engine spiders. If you are ready, let’s get started.
The best version of Lynx is the one you are going to run on your computer, not using online tools. There are Lynx versions on the web (used for online checking), but they are not real-time and active text browsers. So it is recommended that you use the stand-alone version.
Since the majority of computers are powered by Windows, we will use this as the scope of this article in terms of operating system. And you can download Lynx.
If you are using Linux, UNIX or other operating systems, use the Lynx web site.
After downloading the file, unzip it to your desktop. I have compiled it to run completely stand alone (as one file only) without needing you to install it or mess with its files in your computer. Once it is extracted to your desktop, double click it to run the application. The first thing you will see is the Lynx dashboard showing your files in “My Documents."
Here are some screen shots illustrating the processes of downloading, extracting and opening the text browser:
Once you see the dashboard, you are ready to use the Lynx text browser.
The first basic operation that you should know is how to maximize the DOS-based screen which acts as a text browser view in Lynx. It is relatively difficult to work without maximizing the Lynx browser window. To maximize the screen (full screen mode):
Step 1: Click any area in the Lynx browser.
Step 2: Press Alt -> Enter.
Step 3: The browser will occupy the whole screen in your computer monitor. This is important if you are working with details, especially troubleshooting suspected uncrawlable sections in your website.
To go back or close the full screen mode of Lynx browser, simply type the Esc key twice and press “Q” ; when you see the message “Are you sure you want to quit?” press “Y” for yes in your computer keyboard. In case you cannot see the Q key button or you cannot see the message above, try pressing the TAB key and then type ESC -> Q to repeat the procedure above.
The next thing you should know is how to actually surf web pages. To do this:
Step 1: Launch “Lynx Text Browser.”
Step 2: View in full screen mode (optional).
Step 3: In the dashboard, press “G” in your keyboard. After pressing this letter in your keyboard, you will see:
URL to open:
You can then type the exact URL of your website or specific web page; always include http:// if it uses the HTTP protocol. Example:
Below is a related screen shot showing the Lynx dashboard. It shows how your screen will look after typing the letter “G” in your keyboard (“URL to open” will appear), and finally how the URL is entered in the Lynx browser before pressing the “Enter” key in your keyboard to visit the website.
Finally, when you press enter, after typing your website URL like the one shown above, you will see some cookie-related warnings. This actually depends on the website cookie settings; for example, below are the cookie-related warnings you will see when visiting www.google.com:
www.google.com cookie: P=ID=7e1e51c1c0d02146: TM=125 Allow? <Y/N/Always/never>
Since you are only using the Lynx browser to diagnose search engine crawling related issues, you can press letter “V” to ignore and NEVER allow cookies.
After pressing “V,” the page will continue to load until you can see all of the text-related content on your web page.
In Lynx, you cannot use your mouse to click hyperlinks or navigate the web pages as you would with an ordinary HTML web browser (Firefox, IE, etc). In the Lynx browser, you only use your computer keyboard arrows and keys to navigate the web page. How?
After your web site has finally loaded in Lynx, you can scroll down the content by pressing the Arrow Down keys. To go up, press the Arrow Up keys.
It is also important to know the colors representing particular information in the content.
Violet or White represents pure text content.
Green represents crawlable hyperlinks.
Yellow denotes the highlighted/selected hyperlink, or form button.
Red represents bold text information.
Blue is for text in italics.
Every time you press arrow down or arrow up keys in your keyboard, your cursor will navigate though hyperlinks and buttons. HTML text links are in green when not being actively highlighted, and turn to yellow when they are highlighted.
For example, in the screen shot below, the link with the anchor text (this is in the navigation menu of the website) “PHP Web Programming” has been selected, and the color of the link has changed from green to yellow:
To visit that page, click “Enter;” Lynx will load the “PHP Web Programming” page. So for example, if I want to visit the home page again, I will just scroll up/down to highlight the “Home” link (which changes from green to yellow) and then press the “Enter” key.
If you want to view the source code of a page, press “” on your computer keyboard. If you want to reload a page, press Control – R on your computer keyboard.
Try using Lynx on your favorite websites. After at least three hours of practicing the above features, you will start to become familiar with how to navigate web pages on the Internet using the Lynx browser.
In part two we will illustrate actual examples of the use of Lynx in search engine optimization.