Are You Too Obsessed with Objectivity in Quality?

Delivering objectivity in a test effort and its effect on overall product quality are key areas of focus for a test management team. This objectivity and the associated measurable outcomes are important for driving quality upstream—both in representing the quality health index to senior management and in directing individual test teams on a test execution effort. The greater the objectivity in a quality initiative, the easier the decision on whether a product is ready to be shipped.

Test teams spend a lot of time and effort building objectivity into the overall test effort—be it building dashboards, tracking test and product quality through a set of metrics, working based on service level agreements (SLAs), or exit-criteria gating a product’s release based on meeting specific numbers.

While there is clear benefit in working toward these numbers, sometimes teams focus too much on the numbers element while losing sight of the bigger picture that these numbers help achieve. There is no supreme set of metrics that is going to be applicable to all products and projects. Organizations often track metrics under various heads, such as test effort or management metrics; cost-of-quality metrics; product quality metrics, covering values such as defect metrics (defects by resolution, injection phase, priority, severity, type, cause, or patterns); test efficiency metrics (to understand test execution numbers and pass percentages); quality of testing (to understand and compare defects found before and after release); and test coverage metrics (to trace a test effort to the product requirements).

While there is tremendous value in implementing and tracking some of these metrics, what is important is to devise an overall strategy around such an objectivity implementation so that everyone is clear on the what, when, and how of the effort.

At a very high level, let us understand how a measure and a metric are related. A measure is simply a number measuring a certain item. For example, the number of invalid bugs is a measure. A metric is a comparison or a relative use of two or more measures to derive a meaningful value. For example,comparing the number of invalid defects in relation to the overall number of defects will help you derive a useful value in understanding a trend and a percent of invalid defects.

With this distinction in mind, here are nine practices that can help you devise an objective quality strategy.

1. Understand why you want to bring in objectivity. As a team, it is important to take the time to comprehend why you want objectivity. Justifying the need for objectivity specific to your project will help you get buy-in from the entire team. A team member who is not convinced about this process can hamper your project in several different ways.

2. Focus on what metrics matter to your organization. As mentioned earlier, there is no supreme checklist of metrics that will hold good for all test organizations. Understand the constraints you and your project operate within to create a custom list of what metrics make the most sense for you to track.

3. Do not hesitate to drop long-used metrics that no longer work for you. Times are changing. What may have worked well for you in the past may not necessarily work in the current times. For example, you once may have tracked overall test traceability against defined product requirements. If you are working in an agile world now, not all requirements can be tracked in a specifications document, per se. Verbal collaboration with teammates may be a requirement to understanding the overall scope of product requirements.

4. Incorporate newer values that align with your current needs. Metrics is an evolving and dynamic set. With every release or milestone, revisit your metrics to ensure you are tracking those that help you make the important sign-off decision. Be sure to look back at past releases to incorporate what you’ve learned in building newer metrics.

5. Use metrics that result in actionable items. A metric that does not translate into an actionable item is often of no use. Map every metric you use to see if you are able to derive a resultant action from it. For example, a metric that shows an increased number of invalid defects may require a conversation with the whole team to fix any gaps in product understanding.

6. Do not use measurements as goals. While it is OK to plan for the direction you want your team to head in based on measurements and associated metrics, it is not a good practice to set the measurements as goals. For example, say your code coverage is currently at 60 percent. It is OK to look for ways to increase code coverage through a variety of tests, trying to touch areas that have not been tested, etc. But it is not a good practice to set a goal for the next time to reach, say, 70 percent code coverage. If measurements are made goals, the team becomes number driven and works to achieve the number rather than understanding the true meaning of the measure.

7. Weigh in on what it takes to implement such an objectivity-focused test effort. It is not rocket science to understand that building objectivity in a test organization is a charter of its own that has associated overheads. Efforts include measuring specific line items, deciding what metrics to use, creating the metrics, and understanding resultant action items, then representing them in dashboards and communicating them up and down within the team, to name some core ones. All of this is done in an effort to have a solid backing in signing off on a quality product release.

8. Keep it simple. While it is good to be elaborate in your measurements and metrics, it is also important to keep it simple and not lose sight of the larger goal of what these measures and metrics are for. At a high level, having specific metrics around traceability, defects, and test cases should be sufficient to measure product and test effort quality.

9. A little subjectivity does not hurt. Although all of us strive for objectivity, it helps to have a small element of subjectivity in the final sign-off process. The test manager or test director’s experience in using varied metrics and in making a gut call whether the product is ready to be shipped proves highly valuable rather than solely relying on measures and metrics.

Building an objective quality effort is becoming a need of the day. Keeping in mind the above listed areas will go a long way in building a successful and objectivity-driven quality plan, mitigating challenges around team buy-in, relevance of measures and metrics to the current project, and the overall time taken to build and implement the plan.