Random Testing and "App Compat"

In 1996, I posted my version of the Ten Commandments of software testing on a Web site. I wrote these commandments in the middle of the night when I couldn't sleep. I was thinking about what things I could do to try to repro an elusive bug. As most testers will attest, sleep can be hard to come by in such a circumstance! Somehow in my slightly warped tester brain, these "things" took the form of commandments. It was a very spiritual moment.

I took the time to write the commandments down and then promptly went into a very deep sleep that made me late for work the next day. I was determined not to make my sleepless night-and late arrival-completely without benefit and decided to post the commandments to my Web site. I wondered if anyone would notice.

Well, I can now say for certain that, indeed, they noticed. It took a while, but I started getting mail about them. At first, the mail trickled in. Then, months later, it began pouring in, at the rate of two or three a month. It steadily increased until I would be surprised if a week would go by without getting at least one note about my commandments.

The mail was neither complimentary nor derogatory. In typical testing fashion, it was inquisitive. Could I please interpret one of the commandments? Would I settle a bet between two colleagues about commandment number four (except I thought "settle" meant "pay" and I told them "no")? And the most frequently asked question: "Why are there only nine commandments?"

Well, after hundreds of private e-conversations (some of them months-long in duration) with individual testers around the globe, I have finally decided to come clean, publicly, and get these commandments "off my chest" once and for all. But it will take three of these columns to accomplish the task. This column explains the first two. The second will explain numbers three through six, then seven through nine (and the lack of the tenth) in the third column. I hope you enjoy them, as I have enjoyed them over the years.

First, I'll just list the commandments so you can see them as original visitors to my old Web site saw them, listed but not explained:

1. Thou shalt pummel thine app with multitudes of input
2. Thou shalt covet thy neighbor's apps
3. Thou shalt seek thee out the wise oracle
4. Thou shalt not worship nonreproducible failures
5. Thou shalt honor thy model and automation
6. Thou shalt hold thy developers' sins against them
7. Thou shalt revel in app murder (celebrate the BSOD!)
8. Thou shalt keep holy the Sabbath (release)
9. Thou shalt covet thy developers' source code

And now, here are my interpretations of numbers one and two. I just hope I remember all the things I included in my earlier interpretations! Any of my old correspondents are welcome to chime in at the end of this column.

1. Thou Shalt Pummel Thine App with Multitudes of Input
One of the first things that any tester learns is that the input domain of almost any nontrivial software application is infinite. Not only are there lots of individual inputs, but inputs can be combined and sequenced in so many different combinations that it is impossible to apply them all. One of the second things testers learn is that the trick is to apply the right set of inputs so that infinity doesn't have to be faced head-on.

Well, of course I agree with this approach. My own writing and teaching is full of advice on how to select the right set of inputs. But I also counsel testers to buck up and face infinity anyway. The method of doing this: massive-scale random testing. It is a tool that should be in every tester's toolkit and few, if any, testing projects should be without it.

Massive-scale random testing must be automated. Although it isn't easy to do the first time, it does get progressively easier with each project and eventually becomes rote. It may not find a large number of bugs, but it is an excellent sanity check on the rest of your testing: If you are outperformed by random testing, you may have a problem on your hands. And I am always pleased with the high quality (albeit few in number) bugs that random testing manages to find.

Another reason to apply massive-scale random testing is that setting up such tests requires a healthy knowledge about the input domain of the application under test. Testers must really get to know their inputs and the relationships that exist among the inputs. I almost always find bugs and get good testing ideas just from the act of planning massive-scale random testing.

2. Thou Shalt Covet Thy Neighbor's Apps
This commandment sounds a bit perverted, but I can assure you it has a "G" rating. The idea I am trying to get across here is not to test your application in isolation. Otherwise, you might run into the nightmare scenario of "application compatibility," or more specifically, lack thereof. Application compatibility, or "app compat" as it is widely abbreviated, means that one application does something to break another one. In case you are not sure, that is a bad thing.

One way to combat this problem is to keep a cache of apps (old ones, new ones, borrowed ones, blues ones-the more diverse the better) and make sure that you run your app at the same time each of these are running. Of course, we want to do this with operating systems as well. A user should not have to tell you that your application won't run with a specific service pack installed; that is something you should find out in your own testing.

So covet those apps and those service packs: The more the merrier!

References

• Massive-scale random testing is a subdiscipline of model-based testing. Learn more about both at  www.model-based-testing.org
• Oracles, Failures, Models, and Sins: The Ten Commandments of Software Testing, Part 2
• Test-Physicians, Source Code, and BSODs: The Ten Commandments of Software Testing, Part 3