The Value of Requirements-Based Testing

For the last twelve years, I have been the co-owner of a restaurant in Tampa, Florida. A couple of years ago I had a meeting with a group of local restaurateurs and we discussed (among more important topics, like how to sell more beer and wine) the software we used to run our restaurants. Here's a little anecdote told to me by one of my fellow restaurateurs.

"We used to have a clunky old application: slow, awkward, really just plain ugly, and full of bugs. When we got new hardware we also "upgraded" to new state-of-the-art software. We fell in love immediately with its look and feel: it was slick with touch screens, icons, and other nifty features.

We quickly learned though, that the builders had obviously never worked in a restaurant-maybe they'd never even eaten in one. One particularly annoying problem was that after a customer's check was closed out it couldn't be reopened. So, when the customer yelled out, "Oh, just give me one more beer after all," we had to delete the old ticket, create a new ticket, and re-enter all the old data plus the new order. (If you simply opened a second check the customer would have to pay two separate checks.) Meanwhile the customer is getting very thirsty and consequently unhappy.

When we complained to the vendor, they assured us that the code did exactly what it was designed to do. Finally, after complaints from other users, the vendor issued an "enhancement" that allowed an existing ticket to be modified. Yea!!! Unfortunately though, when we ran our daily and weekly reports, we eventually discovered that the original ticket and the modified ticket were both added into our sales summaries, which played havoc with our management metrics and caused us to overpay our state sales taxes.

The vendor of this system had created a very attractive product that appeared to work seamlessly. Unfortunately the product was built upon a flawed concept. The developers and testers didn't seem to understand the restaurant business. Even though the code did what it was designed to do, the customer was unable to use the software to do some of the most basic functions needed to effectively run a restaurant. As important as it is to employ techniques that excel in finding bugs (think orthogonal arrays, state transition diagrams, structural testing, etc.) it is also important to ensure that the systems we build satisfy the user's actual requirements.

Notice that I didn't say requirements specifications. Please note the distinction drawn between a requirement and a requirement specification. The requirement is a need of the user; the requirement specification is our imperfect attempt at documenting this need.

A complaint that many test professionals make is that it is difficult, if not impossible, to test a requirements specification. This is because by their very nature, requirements specifications are inaccurate and constantly changing. They change because the users change, the business changes, some requirements turn our to be infeasible or too expensive, the users may not know exactly what they want, or we maybe we just got 'em wrong in the first place. Very few, if any, organizations will ever successfully create a complete, accurate requirements specification prior to the commencement of design and coding.

But just because we can't document the requirements perfectly doesn't change the fact that all systems have requirements. These requirements may or may not exist in a requirements specification, but they do exist, otherwise why are we building the system in the first place?

In my view, the key is to have a good starting point-a reasonable view of what the user wants or needs and to document this need in some fashion that will help the builders and testers do their job. This document, or requirements specification, does not always have to be perfect because it will change as the design and code are created. Still, we should make every attempt to clarify ambiguities and omissions to avoid building in defects from the start.

Testers can play an important role in testing the requirement specification before the design and coding begin. Some people use word pictures, use cases, inspections, or some other technique. The method that we find effective for many organizations is called "preventive testing." We admit, this is a bit of a misnomer since preventive testing does not really succeed in preventing bugs, but helps find them very early in the development process.

The concept is deceivingly simple, and no doubt already in use by many of the readers of this article. High level test cases are written to "test" the requirement spec prior to the commencement of design and coding. These test cases will probably never be executed, their only role is to discover ambiguities, errors, and omissions in the specification that will ultimately be used to create the system. Some of the test cases will identify flaws in the requirements specification which will cause it to be updated, rendering the original test case obsolete. Yea, those test cases have already done their job.

Here's an example:

Take a typical (inaccurate) requirement specification-this one is for an ATM:

R1: A valid user must be able to withdraw up to $200 or the maximum amount in the account.

This is obviously a weak attempt at writing a requirement specification, but this is certainly not atypical. Developers try to solve problems. Testers are also problem solvers, but we also try to look for things that can go wrong. What can go wrong with the spec above? What is a valid user? Hopefully that is documented in another requirement, but if not we could create some test cases to challenge the term "valid" user. What does the word "or" mean? If I have one million dollars in my account can I withdraw it all? How many withdrawals can I make in a day? Can I withdraw an odd amount, say $158.27?

A few sample test cases:

TC01 Attempt to withdraw $800 from an account with $800 in it (i.e. the max amount in the account)

TC02 Attempt to withdraw $200 10 times in one day.

TC03 Attempt to withdraw $158.27

Notice that these "test cases" are not elaborate enough. They don't even have expected results since they will probably never be executed. Their purpose is merely to challenge the voracity of the requirements spec which will ultimately form the basis for the development effort.

If you had written the above test cases, you would have discovered defects in the requirements specifications that no doubt would have resulted in a clarification of the requirement specification. Oh, I know what you're thinking: Do we really need to do this? Surely no one would ever try to create code to withdraw pennies, nickels, dimes, quarters, etc. from an ATM. Unfortunately, things like that happen more often than we would like to think.

So far, we've talked about test cases based upon the requirement specifications that are used to validate the accuracy of that particular document. Do we also need to create test cases based upon the requirements specification that will be executed? I think so. I believe that it is important to create tests that demonstrate that the user can actually perform their day-to-day functions using the system. As important as it is to find defects, at the end of the day, our users don't care how many bugs you find as a tester if they can't use your system to perform their job.

Do we really want to build systems without requirements specifications? Do we really want to build the system and then "learn" about it to figure out if this is what the user wanted? Are the developers and testers truly the subject matter experts or should we do our best to figure out what the users want in advance, even if we are not perfect in our endeavors? Can we use preventive testing or some other similar technique to help flesh out some of the ideas of what the system will do before we launch into coding? Even though requirements-based testing will probably only touch a fraction of the possible interactions of the code, we can execute enough requirements-based tests, at the very least, to ensure that the user can perform the basic tasks required for them to run their businesses.