Fuzzing for Software Security Testing and Quality Assurance, Second Edition

Book Preface

It was a dark and stormy night. Really. Sitting in my apartment in Madison in the Fall of 1988, there was a wild midwest thunderstorm pouring rain and lighting up the late night sky. That night, I was logged on to the Unix systems in my office via a dial-up phone line over a 1200 baud modem. With the heavy rain, there was noise on the line and that noise was interfering with my ability to type sensible commands to the shell and programs that I was running. It was a race to type an input line before the noise overwhelmed the command. This fighting with the noisy phone line was not surprising. What did surprise me was the fact that the noise seemed to be causing programs to crash. And more surprising to me was the programs that were crashing—common Unix utilities that we all use everyday.

The scientist in me said that we need to make a systematic investigation to try to understand the extent of the problem and the cause. That semester, I was teaching the graduate Advanced Operating Systems course at the University of Wisconsin. Each semester in this course, we hand out a list of suggested topics for the students to explore for their course project. I added this testing project to the list. In the process of writing the description, I needed to give this kind of testing a name. I wanted a name that would evoke the feeling of random, unstructured data. After trying out several ideas, I settled on the term “fuzz.” Three groups attempted the fuzz project that semester and two failed to achieve any crash results. Lars Fredriksen and Bryan So formed the third group, and were more talented programmers and most careful experiments; they succeeded well beyond my expectations. As reported in the first fuzz paper1, they could crash or hang between 25–33% of the utility programs on the seven Unix variants that they tested.

However, the fuzz testing project was more than a quick way to find program failures. Finding the cause of each failure and categorizing these failures gave the results deeper meaning and more lasting impact. The source code for the tools and scripts, the raw test results, and the suggested bug fixes were all made public. Trust and repeatability were crucial underlying principles for this work. In the following years, we repeated these tests on more and varied Unix systems for a larger set of command-line utility programs and expanded our testing to GUI programs based on the then-new X-window system2. Windows followed several years later3 and, most recently, MacOS4. In each case, over the span of the years, we found a lot of bugs and, in each case, we diagnosed those bugs and published all of our results. In our more recent research, as we have expanded to more GUI-based application testing, we discovered that classic 1983 testing tool, “The Monkey” used on the earlier Macintosh computers5. Clearly a group ahead of their time.

In the process of writing our early fuzz papers, we came across strong resistance from the testing and software engineering community. The lack of a formal model and methodology and undisciplined approach to testing often offended experienced practioners in the field. In fact, I still frequently come across hostile attitudes to this type of “stone axes and bear skins” (my apologies to Mr. Spock) approach to testing.

My response was always simple: “We’re just trying to find bugs.” As I have said many times, fuzz testing is not meant to supplant more systematic testing. It is just one more tool, albeit, and an extremely easy one to use, in the tester’s toolkit.

As an aside, note that the fuzz testing has not ever been a funded research effort for me; it is a research advocation rather than a vocation. All the hard work has been done by a series of talented and motivated graduate students in our Computer Sciences Department. This is how we have fun.

Fuzz testing has grown into a major subfield of research and engineering, with new results taking it far beyond our simple and initial work. As reliability is the foundation of security, so has it become a crucial tool in security evaluation of software. Thus, the topic of this book is both timely and extremely important. Every practitioner who aspires to write safe and secure software needs to add these techniques to their bag of tricks.
Barton Miller Madison, Wisconsin April 2008