In his CM: the Next Generation series, Joe Farah gives us a glimpse into the trends that CM experts will need to tackle and master based upon industry trends and future technology challenges.
Testing is a complex discipline. There are various approaches, methodologies, strategies. Where is the connection with CM? As with development, requirements specifications and other aspects of product development, the connection is on the management side. A software configuration management audit is really about demonstrating that you have test case coverage for your requirements and that the test cases have been successfully run against the target build.
Viva la Difference!
Test case management is similar to software management and to requirements management, to some extent. It's the differences that make test case management interesting. Let's explore a few of these differences:
1) Just as requirements may be applicable to more than one product, so too test cases may be common from one product to another, especially when the same manufacturer is involved, or when an industry standard is being addressed.
2) Test case management also involves tracking and management of test case results so that the FCA can be automated. Unlike the test cases, the test case results are specific to a particular product build.
3) Whereas software tends to evolve, changes to test cases tend more to be additive. That is, new test cases are written to supplement the existing ones, much more frequently than upgrades to existing test cases are made. This depends on the granularity of a test case, however.
4) Test cases are clearly divided along automation lines. Automated test cases can be run frequently, repeatedly and without human intervention. Presumably, the results can be extracted automatically as well. Manual test cases chew up resources. So the goal is always to move more and more of these test cases to the automated side of the fence.
5) Test cases require specific conditions under which they must be run. Management of these conditions, the test environment, are as much a part of the test case as the operational test itself is.
6) Depending upon granularity, the number of test cases may be very large. So it is both important that test case management has some degree of automated identification, as well as hierarchical, and other organizational capabilities. This includes option and variant tagging.
So what is the impact of these attributes on current and future CM systems?
Product Management
The CM process/tools must incorporate product management. The CM repositories must share test cases among products in a product family, and beyond. It's not good enough to be able to export test cases from one product repository to another. That leaves you managing multiple copies of the same test suite, which is really no different than copying a complete set of source code for each development stream.
Test Case Results
It is not sufficient to track the test cases in the CM repository. The data runs must also be tracked. It must be possible to associate a test case run with a specific build and with a specific tester verification session. We need to be able to asked who tested what, when and what were the results. This is the proof we need to conduct the Functional Configuration Audit, so we might as well automate that audit while we're at it, or at least leave the audit to auditing the CM/Verification tools and processes rather than having to wade through the mounds of data.
Tracking test case results in the CM repository also allows a number of other capabilities and queries. For example:
- When did this test case last fail?
- Which requirements have never been tested?
- Which requirements are not yet verified as complete?
- What's the test history for a particular test case?
A number of testing metrics are also easily generated from the combination of test case run data and its traceability back to test cases, builds, and testers.
Change Control of Test Cases
Change to the repository of test cases is most often in the form of adding in new test cases. This is especially the case if the granularity of a test case is small. A good CM system helps to manage aggregates of small test cases so that individual tests can be represented as individual test cases while still allowing the management of less granular clusters.
A fine granularity has the side effect for test cases that changes to the test case repository are generally of the form of adding in new test cases, rather than testing existing ones. This obviously depends on the test specification complexity. But generally, the user interface must be optimized for adding and tagging/organizing test cases. This is not to say that full change and version control will not be necessary. But often, the original test case is as important as the revised and must be maintained as a separate test case. Thus, there is less revision and more management of test cases.
Automated Test Cases
Test case automation is essential to quality. CM tools can help by allowing easy packaging of an automated test suite based on the build and the features and options contained within it or targeted for testing. The CM tools can also help to automate the collection of test results, especially when it can inject the test case identification into test suites. This is highly dependent on both the types of tests and the test engine technology.
The Test Environment
The CM tool can track not only the test case but the conditions for running the test. More advanced systems should allow specific environments to be defined that can be shared by multiple test cases. When test environments and data setups can be tracked separate from the test case itself, there is an economy of specification, and the resulting economy of management effort for changes to the environment. At the same time, the tools should allow a complete test case description to be presented entirely as a single set of instructions. This is especially important when addressing failed test cases individually. This occurs when the failed test case is fed to the development team for diagnosis, starting with reproduction of the test case that caused the failure.
Scalability
When the granularity of the test cases is fine, the number of test cases can easily grow into the tens of thousands. It is important that CM tools be able to scale to these requirements, especially considering that it is desirable to track multiple products in the same repository to facilitate sharing of test cases and requirements across products.
Test Case Management: An Integrated Solution
A next generation CM tool will consider test case management as part of a full ALM solution, and not as an isolating test case management function. Role management, generation of problem reports, coverage of requirements, traceability to builds, and hence to the changes in each build. These are all essential components of test case management. A solution that is designed in the context of an end-to-end lifecycle will yield real advantages.
The user interface is critical, but not just from a verification team perspective. The developers need to be able to navigate test results. Management needs to navigate the data but also correlate it with changes, problem report arrival rates, audit requirements, builds, etc. A combination of object-oriented capabilities, management dashboards and traceability navigation aids should be an important component of the UI. Object-oriented operations should be applicable to hierarchical displays to facilitate reporting, metrics, test case extraction, etc. on hierarchical sub-trees.
