In this interview, Mike Trites, a senior test consultant, talks about his upcoming presentation at STAREAST 2014, the future of metrics, the importance of improving the efficiency of your metrics, and even an interesting take on the old phrase that numbers never lie.
Mike Trites will be presenting a presentation titled "Making Numbers Count: Metrics That Matter" at STAREAST 2014, which will take place May 4–9, 2014.
About "Making Numbers Count: Metrics That Matter":
As testers and test managers, we are frequently asked to report on the progress and results of our testing. The question “How is testing going?” may seem simple enough, but our answer is ultimately based on our ability to extract useful metrics from our work and present them in a meaningful way. This is particularly important in agile environments, where clear, concise, and up-to-date metrics are potentially needed multiple times per day. Mike Trites identifies a number of ways metrics can be used to measure progress during a test cycle and, ultimately, to determine when to consider testing complete. Learn the common pitfalls that metrics misuse can lead to and how you can avoid them by giving proper context when communicating metrics to your stakeholders. Take back key metrics for measuring the effectiveness of your testing and discover how to use what is learned on one project to improve your testing process on future projects.
Cameron Philipp-Edmonds: Today we have Mike Trites, who will be speaking at STAREAST 2014, which is May 4 through May 9. He is giving a presentation titled "Making Numbers Count: Metrics That Matter." He is the senior test consultant with Professional Quality Assurance Ltd. Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an advanced-level testing manager certification from the ISTQB. As a consultant, Mike has worked with clients both small and large in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems.
As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve the internal testing processes. Did we catch everything?
Mike Trites: Yes. Sounds about right.
Cameron: Because you're doing a session titled "Making Numbers Count: Metrics That Matter," I'd like to ask you a couple of questions about the value of metrics and knowing how to make the most of metrics that you have. First question, and it's a broad one, is why are metrics really all that important?
Mike: The thing about metrics is they help us answer a lot of the questions that people often ask, like how long is this going to take, how are we progressing, are we on track, and where are the problem areas? They allow us to deliver the information stakeholders are looking for in order to make the important decisions that they need to make. Furthermore, stakeholders are looking for this information regardless of whether or not we plan to capture it. It's helpful to have the information ready to have the ability to deliver it in a clear and consistent manner. Metrics are also important for process improvement. They allow us to identify any deficiencies or simply areas for improvement and address those on the fly or from one project to the next. Of course, we can measure the results of those adjustments and then continue to improve further.
Cameron: Why are metrics even more important to agile environments?
Mike: It's funny. I think that actually most people would argue that metrics aren't important at all in agile environments, but I think there's still a place for them. After all, stakeholders need information about the status and progress of testing regardless of which methodology is being employed. I think what's more important in agile environment is the timely delivery of any metrics that are being reported on. Things happen really quickly in agile environments, and that's why the process and documentation is so much more lightweight compared to other methodologies, but despite this, I think we can get some benefit out of metrics. With the right tools, we can easily make metrics available to stakeholders without much overhead.
Cameron: You mentioned some common metric misuses in your presentation. Can you give us an example of one?
Mike: Probably the biggest misuse of metrics that I can think of is using them as a basis for performance evaluation. Too often, companies judge the worst testers by looking at the number of test cases they execute or the number of defects they logic. While that may seem a good way to motivate teams or individuals, it can actually have a number of unintended side effects. Long term, it can actually be demotivational and lead to an overall lack of faith in metrics and the value they provide.
Cameron: There's an old expression, an old adage out there, that numbers never lie. How does this apply to metrics?
Mike: Metrics don't lie, numbers don't lie, but saying that is a lot like saying, "Guns don't kill people, people kill people." When they're not used properly, metrics can be left very much open to interpretation. That also means that they're open to manipulation and abuse. I think that's part of the reason that more companies don't invest more heavily in metrics. Unfortunately, metrics, like a lot of other things, only provide real value when they're used properly. By the same token, you don't get all the thrills you're looking for out of a Lamborghini if you keep it in first gear the whole time, but that doesn't necessarily mean that all Lamborghinis are junk, now, does it?
Cameron: No. That's a good analogy. I like that. What is some advice you would give people who are unable to really change the method of metrics they have in place but can adapt how they use that method?
Mike: A big factor in successfully implementing a metrics program lies simply in how they are presented. Even when a lot of things aren't under your control, improvements can be made to how metrics are reported to stakeholders. The key is to make sure that the message behind the metrics is consistent and clear to everyone by providing the necessary context. Because metrics is so open to interpretation, there is a risk that people will draw the wrong conclusions even if there's no intent to deceive. This can have the effect of triggering false alarms or, even worse, hiding problem areas and giving a false sense of confidence when things are not going well. Situations like these can be avoided by making sure that metrics are always provided with context in one form or another.
Cameron: You have more than a decade of experience in software testing and you've worked with clients both big and small. Does the importance of having good metrics or being able to use metrics correctly increase the bigger the project or the bigger the team working on it is?
Mike: I think that metrics have an important role on all projects. With bigger projects, though, things like communication become more important but also more difficult. Since metrics are a communication mechanism at their core, they can be a real necessity on big projects where there isn't always time for everyone to talk to everyone else. The key is to make sure that they are used consistently so that everyone is effectively speaking the same language. At the same time, bigger projects offer more opportunities to accumulate historical data, which actually increases the potential payoff in the long run. Longer timelines means there are more opportunities to learn lessons from the metrics that are captured and make course corrections along the way.
Cameron: You'd like the attendees of your presentation to really step away with a number of different things, but what is one thing, or the main takeaway, you'd like the attendees to come away with?
Mike: More than anything, I want to stress the point that metrics are important and that there is the place for them on most if not all projects. The key, really, is finding the right approach that works for each individual in their organization.
Cameron: This is a broad hypothetical question and there's no wrong or right answer to this: How do you think the use of metrics will change in the future?
Mike: That's a bit of a hard one. I guess if there's more people like me out there, they will be a little more widely adopted, I suppose. That depends on how many people come to the presentation, but with the onset of agile methodology, the trend at the moment seems to be steering away from standardized documentation and standardized process and, unfortunately, at the same time, standardized metrics. We often see things like this as cyclical. I really do hope that it's not something that falls off entirely because, again, I do see there being value in the process.
Cameron: I agree. I agree. Is there anything you'd like to say to the delegates to STAREAST before they attend the conference and before they attend your presentation?
Mike: I'm really excited to be part of this year's STAREAST conference. Not only am I looking forward to the opportunity to present and share my own perspectives on metrics, but I'm also really looking forward to the opportunity to attend the other session and learn from them. Thanks so much for this great opportunity, and I'll see you all in Orlando, I suppose.
Cameron: All right, and if you haven't signed up yet, sign up for Mike Trites's presentation, which is "Making Numbers Count: Metrics That Matter." And he will speaking at STAREAST, which is May 4 through May 9. Thank you so much, Mike.
Mike: Thank you.
Senior test consultant with Professional Quality Assurance Ltd., Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an advanced-level test manager certification from the ISTQB. As a consultant, Mike has worked with clients, both small and large, in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems. As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve their internal testing processes. Find Mike on LinkedIn at ca.linkedin.com/in/miketrites.
Podcast Music: "Han Solo" (Captain Stu) / CC BY-NC-SA 3.0
