Are You Quantifying Your Value with Test Metrics?

As a provider of test management service, I frequently hear stories from our clients faced with the unpleasant question of “How exactly does your team provide value?” We laugh at the question, and say, “Try shipping without us!” But, put on your CEO hat for a moment and consider what they are asking. The bottom line is time and money; shipping earlier means saving time and money. The test or QA groups don’t add new features and slow the project down. Isn’t it fair for CEOs to ask what they are getting for their money?

At the very least they may ask, “How can we go faster?” Here’s two of the more common ways a CEO might say this:

• What's holding up the release of this project?
• Why can't your team run faster tests?
• Are these tests really that important that they should delay the release?

QA management has to field these sorts of questions and others like them on a fairly regular basis. It can be downright tough to come up with an answer that will satisfy the non-tech crowd. We all know that running a solid QA effort and making sure that the final software product is released with the highest quality possible takes time and patience, but try telling that to an antsy executive.

So what are we, the quality professionals, supposed to do? How can we show our higher-ups that the line of work is important and, moreover, a cost-saving machine? Testing metrics can be extremely useful in this regard, but it all comes down to which ones you use. For instance, if you have an irate executive grilling you about the amount of time production has been locked into running tests and you respond with a list of problems you’ve found, you are actually providing a list of things causing even more delay. “We add value by slowing things down!” might be the message you want to send, but I’d advise a little more discretion.

I'll let you in a little secret about software development: The movers and shakers of this industry couldn't give a rip about how pristine a product is when it's launched. They could argue that if a flaw in the coding never manifests itself when the software's in the hands of the end user, who cares if it exists? To get executives off your team's back, you need to scrounge up some testing metrics that demonstrate value.

Choosing the Right Metrics
A client of ours once recounted the time he found himself having to deal with a particularly bad-tempered upper-management type demanding an explanation for why his testing was holding up a release while they ran some automated test scripts. This executive was receiving some heat himself from his own superiors, and I was the benefactor of his misplaced anger.

Our client had to calmly explain to this executive that his team wasn’t out to test everything; they were already running the critical tests. In order to ease his superior’s mind that his team wasn't going to be stuck in a perpetual state of evaluation and analysis, he showed the executive their test execution progress, clearly indicating what his team had done so far and how much longer he could expect testing to take if they found no release-blocking defects. Now, could the irate executive make sense of all the information in the report? Certainly not. What he needed was to see a plan with an end date and visible steps to track progress. With a plan in hand, he could explain to his superiors what was holding things up and when they could expect things to be done.

After showing the plan, we were still left with that perception that testing doesn’t add value and slows down release. The next step is to show how testing benefits the bottom line. The obvious answer is “We stop disasters,” and yes, even the most impatient executives will concede that some testing needs to be done before a product gets shipped out. They do realize how harmful a total disaster would be, but it's still up to you to show that you're not wasting time or money during that process. In this instance, the test manager was able to demonstrate that the automated scripts his team was running resulted in a build-test-deploy process that was much quicker than a manual process. Also, the automate scripts were reused from earlier projects, which QA members could access through their test management system. By reusing old scripts that had proven effective before, our client’s team could have saved the company time and resources.

In The Long Run
Even non-technical senior managers get the value of testing; they just don’t want their money to be wasted. Choose metrics that provide insight into what you are doing instead of obscuring it, and help tell the story in a compelling way. Doing so will help you move from being an antagonist to being a partner in shipping the software. I’ve been both; being a partner is better.