Cem Kaner once defined testing as “a technical investigation done to expose quality-related information about the product under test.” If you agree, then a tester is a technical investigator—which is just what you need to be when testing the back end of a program. But is that enough? How else can we think about testing?
Imagine a little application that pulls all the rows for the last month out of a database, formats them in a manner the government can understand, and sends them off to a government agency for accountability.
For me, it’s not too hard to imagine; I’ve done that sort of work many times in my career. Here, I’d like to talk about a way to test it, using a real example, simplified just enough to cover in one article.
It’s time to get specific.
I once worked on a project that helped the government manage the guest worker process, and the software reported who was visiting the country and the state of their passports and visas. A simple maintenance change might be a new government code: People on an expired visa from the United Kingdom now get an output of “7” for visa_type, and people from any other country who have expired visas continue to get a “6.”
The obvious test methods for this are to find or set up some example data, run the extract, and check the results. When we tested this, we had both programmer tests to check the new functions and also looked at the results. But is that really enough?
The Good News
Once a day, the extract enters maintenance mode and a powerful new tool suddenly becomes available: using the previous version as an oracle. In testing, an oracle is a way to recognize a problem once you encounter it. By comparing the previous results to the current results, anything not directly related to the requirements of this change is worth looking into as a potential problem.
Showing no changes outside the requirement for millions (or even tens of millions) of data sets based on real transactions doesn’t proveanything, strictly speaking. But it does show that none of our real, live data we used lately would cause a problem, and that’s nothing to sneeze at.
Here’s an example of how we did it.
Testing Even More
Over my first few weeks working for the government client, I had to test these sorts of changes. I’d start with the obvious spot check, but I would also run the extract twice, creating two data files: one against the database with the current build, and another against the previous production version. I also built small programs in Ruby and Perl to compare those two files.
In this case the files were in XML format; each transaction was a different blob of XML, looking something like this:
<transaction id=“11101”>
<visa_status>1</visa_status>
<entry_date>01012015</entry_date>
<expireson_date>01012017</expireson_date>
<fullname>Matthew Robert Heusser</fullname>
<countrycode>USA</countrycode>
</transaction>
My little diff tools would spit out all the IDs that had different information and what those differences were. At the end, the tool would list all the IDs that were different so that I could cut and paste the results into a database query.
Once I had that, I could write a query:
SELECT DISTINCT EXPIRED(visa_date), visa_county, count(*)
FROM
Transactions t
WHERE T.ID IN (11101,11233,11433,11753,11540,12050)
GROUP BY EXPIRED(visa_date), visa_country
You’ll note the function called “EXPIRED” compares visa_data to today’s date and returns Y or N, so what I expect to see is a single line with “Y,” “UK,” and the same count as the number of IDs. That tells us that every row that was changed should have been changed. After that, I can flip the query around to check if there were any results that should have changed but were not:
SELECT DISTINCT EXPIRED(visa_date), visa_county, count(*)
FROM
Transactions t
where t.id NOT IN (11101,11233,11433,11753,11540,12050)
AND EXPIRED(visa_date)=‘Y’
GROUP BY EXPIRED(visa_date), visa_country
Running that process shows that for a month of live production data, the only thing that changed was the UK expired visa_status moved from a code of 6 to 7—and that there were no expired visas that did not change.
The change generates a high degree of confidence. It also requires me to run a half-dozen steps, to cut and paste, and to look at the results by hand and decide for myself if they are right.
Wait a minute. Is that automated testing? Manual testing? “Actually” testing, or “just” checking?
What Testers Do
Cem Kaner, a professor of software engineering at the Florida Institute of Technology and lead author of Testing Computer Software, once defined testing as “a technical investigation done to expose quality-related information about the product under test.”
If we assume he is on to something, then a tester is a technical investigator—which is exactly what I was doing creating those diff programs and SELECT statements. That investigation, though, had much less value on the next build. I might reuse the queries, or I might throw them away. I certainly didn’t have a goal of getting a green bar automatically without looking at the result.
Those two extremes—the private investigator at one end, and something I once dismissively called the “cult of the green bar” on the other—represent ways of thinking about testing. They present a tension, but it’s not so much an issue of right or wrong; most teams I work with could benefit from more of both. Instead, think about which we should have more of right now.
I’ve told you my story of one team. Where do you want to improve in 2016?
