If you haven’t figured out what I’m talking about, and I’m sure most of you haven’t, I’m referring to software fuzz testing specifically for your website. It’s one of the best ways to test your website for bugs. Because even search engines will still return sites with errors, you have a better chance of a good ranking if there’s less randomness for them to process. And why wouldn’t you want the cleanest possible look for your site? You’ll get a lot more attention if you’re not bombarding people with fuzz.
Let me start off with a simple definition: fuzz testing is a technique that tests your code quality by supplying random samples of data, which we call the fuzz, as input to your program. Whenever your program breaks or crashes, the flaws can be marked for repair. Fuzz testing was created and developed in 1989 by University of Wisconsin-Madison professor Barton Miller and his students. The technique’s main purpose is to succeed where human testers fail by locating all the overlooked mistakes that even the most intricate of tests miss on occasion.
Fuzz testing is not to be confused with a fail safe method to ensure program quality. Because of its randomness, it is best suited for finding the bugs and the kinks that prevent program perfection and cause pesky errors. To say that your code is strong enough to handle the exceptions doesn’t necessarily account for the overall product.
Most people will probably recognize the flaws when certain file types are marked improperly and return as something else. These syntactic errors occur more than you think; in fact this paper cites research done by Eric Brewer that says at least 40 percent of web pages have one syntax error or more (keep in mind that this paper is from 1996). Holy frijoles, think of the randomness! Not to worry though, because fuzz testing processes all this randomness, allowing the search engine to return the "best" documents, not the least bugged.
That’s right, web pages offer just as many problems as the typical program file. Obviously, the fact that malware can be served up from basically anywhere on the web is evidence enough. The search engine alone probably won’t catch this, so you have to test the site yourself, specifically how you serve ads. Obviously, ads are essential for financing a website, but if your visitors and clients are being hit with malware, well I don’t have to tell you that’s bad for business.
The next section will tell you the basic techniques for fuzz testing…
Whether you’re testing a software program, a website, etc., these steps can be applied comparably to each in order to discover what issues your code has that need to be fixed.
First of all, you must take a correct or valid file or piece of data and change it to some kind of random data that will affect the program maliciously. It’s possible to replace the entire file or just part of it randomly. A "fuzzer" can do one of two things: they can generate their own data through a process called "generation fuzzing," or they can take the data from an actual source and simply alter it in some way.
During the fuzzing process, it’s important to make sure all fuzzing occurs in the right components. The altered data must be directly fuzzed under the specific document rather than the structure itself. You merely want to test the application that consumes the data under its specific format/filetype. Testing the entire format will only affect the format verification code, so you need to make the necessary adjustments.
The next step is to transfer the altered data to its specific destination. There are different ways that this can be done; large corporations that do vast amounts of programming might find it necessary to develop their own fuzz testing tools that transfer data to a wide range of applications. It has become widespread practice to make a record of the test data before initiating the fuzz test.
The last step (this one is for the master problem solvers) is to see what happens. The most common disaster occurrences are crashes, CPU usage spikes, and memory spikes. A crash indicates a bug. This is dangerous because they are the most likely to be exploited. Instead of taking a chance and waiting see if it is exploitable, it’s best to just fix the bug. A spike simply means that any malicious data might possibly discontinue service, so fix it!
It is important not to try to go overboard with the fuzzing; stick with trying to find random crashes and spikes. Anything more would require customizing the test to specifically monitor whatever changes are being made to the data and/or software application. However, because most of the programs are being used by third parties, fuzz testing for bugs is still crucial.
As you might expect, some of the most recent fuzz testing news has to do with the Internet, specifically web browsers. Not too long ago, HD Moore, a security researcher, along with a set of colleagues used fuzzing to test the major browsers, mostly notably Internet Explorer, and found several hundred ways to make it crash. All in all, there were at least 50 defects, some of which could allow someone to gain control of a website user’s Windows system. Moore and Co. found it easier to check the user applications (i.e. web browsers) than test the server itself. This also helps protect the user from the malicious websites that I mentioned earlier.
Of course, attackers have become more innovative when it comes to attacking a network’s internal system through browsers. And apparently the internal systems are more vulnerable than the well-maintained external applications. Microsoft has done its part in fixing some of the most crucial vulnerabilities in Internet Explorer, but it would be interesting to see just how far they’ve come to this point.
Earlier this year, the company SPI Dynamics released a web fuzz testing tool specifically for web applications called Web Fuzz. The simplicity and ferocity with which web applications are made makes fuzz testing the ideal choice for detecting vulnerabilities, especially since they account for half of all defects. This was good news for developers because the vendors aren’t likely to have their back when bugs start coming up in their apps. SPI Dynamics employee Michael Sutton describes the way fuzzing finds the simple vulnerabilities through a process called FUGGLE (Fuzzing Using Google Gets Low-hanging-fruit Easily):
… you can Google for sites that are going to be vulnerable to attack. Then you make a request for them using Google fuzzing to see if they could find indicators of what vulnerabilities [are there].
The new developments in fuzz testing grow by the day, it seems. The real innovations are in the field of customized tools designed to test specific software protocol. Some companies are providing libraries of reusable code that make customizing easier without having to start over. Yes, it’s a fledgling industry now, but as the web grows and developers need ways to test their apps, so too will the fuzzing industry.
As I conclude this article, I want to again draw on some of the advantages and disadvantages of fuzz testing. The biggest problem, as I mentioned earlier, is that fuzzing, for the most part, only finds the simplest defects and bugs. Different fuzzers will find different bugs, but these bugs are often of a severe nature, meaning they are exploitable, so don’t automatically discount the fuzz test. And don’t forget that the attackers will use fuzz testing themselves to locate the ripest vulnerabilities.
A lot of people think that the randomness of fuzz testing would also be a disadvantage because it might miss the more valuable flaws. There is a technique called "robustness testing" that tests the input space that the format specifically defines, thereby making the test less random. However this goes back to the value of fuzz testing results in general, and I think we’ve discussed that already.
The one thing you should take from this article is that if you’re going to fuzz test, you must think like the enemy. After all, there are attackers deliberately trying to infect a program’s code. Your program should be able to process any stream of data sent in to it by yourself or a third party. Obviously fuzz testing isn’t fool proof, but it does improve the program’s security against unexpected input.
In short, you need fuzz testing to make your program look its best. It will also make those who visit your web site or use your web application much happier. So help yourself and everyone else by doing a fuzz test.