Some CM practitioners object to placing an emphasis on testing when discussing software configuration management, believing that testing is the domain of a quality assurance organization and developers. Such a perspective, while traditional, is flawed, as the goals of the QA, SCM, and development processes are all closely connected. QA engineers are concerned with identifying what they are testing and being able to reproduce results. As SCM engineers, we want to be able to verify that configurations are valid.
Testing is essential for agile configuration environments and, perhaps, for any configuration management environment.
Best Practices, Attitudes and Principles
We will start with some thoughts on last month's theme of best practices, as we consider testing to be one.
Many people think the term “best practice” means one specific practice is better than all the rest for a particular problem. We believe that it was intended to mean that it is one of many best practice solutions, even for a given problem. This also involves the things that repeatedly prove themselves to work better than many initial or naive solutions. This doesn't mean that each one is the absolute best at what it does, but rather that each is proven to be the best among the other options that had been tried at the time.
Agility and Feedback
Agile software development practices are based on feedback and adjustment. Testing is one way of providing feedback, and the agile practice of continuous integration (CI), supported by testing is a key feedback mechanism. Paul Duvall's book, Continuous Integration, explains that your CI build can not only tell you if the software compiles, but can also give you test results, metrics on, among other things, test coverage, and serve as a window into the overall quality of the system. The right kinds of tests enable those using the SCM process to assign some sort of quality metric to a configuration. The trick is figuring out what the right kinds of tests are.
Attitude and Principles
An important point is the attitude and way in which we go about our jobs of developing software or managing configurations.
Continuous improvement (or “kaizen” from in Japanese) is an elegant expression of some powerful ideas (from http://en.wikipedia.org/wiki/Kaizen) :
To be most effective kaizen must operate with three principles in place:
- Consider both the process and the results so that actions to achieve effects are made apparent;
- Systemic thinking of the whole process and not just what is immediately in view (i.e., big picture, not solely the narrow view) in order to avoid creating problems elsewhere in the process; and
- A learning, non-judgmental, non-blaming approach and intent will allow the re-examination of the assumptions that resulted in the current process.
While kaizen (often associated with the Toyota Production System) usually delivers small improvements, the culture of continual aligned small improvements and standardization yields large results in the form of compound productivity improvement. There is a need to be aware of what we are doing and the effects of it, which brings a different quality of approach, and allows us to question current practices and seek to improve them.
This attitude and approach will lead us both to consider and adopt appropriate patterns and also to implement them in such a way that we can measure and improve (or even discard them if they turn out to be inappropriate for our particular requirements).
The Search for Simplicity
Richard Koch, in his book, The 80/20 Principle, refers to research on differences in performance between 40 medium size German companies. The most profitable companies were all simpler than the others and had a stronger understanding of and focus on what was important and kept working at it.
The fundamental 80/20 rule (also known as Pareto's Principle, after the Italian Economist Vilfredo Pareto who first noticed the relationship) is a challenge to us to think about and observe what activities in our lives and work we typically spend 20% of our time on, actually generating 80% of the results. These figures don't have to be precisely this ratio; it might be 70/30 or 95/5. If we can identify the 20% that really makes the difference and we double it, can we achieve 160%? The challenge is there.
Interestingly, Koch gives the advice to set impossible targets for your teams, because this by its very nature will cause them to focus on the 20% of value. This has a kernel of truth in it and, perhaps, achieves the same result as agile practices that focus on business value. This is in spite of seemingly contradicting the extreme programming no overtime practice. Brad has written in his blog (Simple Ain't Easy):
True simplicity is about minimizing and managing overall complexity. Complexity in software design and development comes from the sheer size and complexity of the problem we are being asked to solve, and the richness and vastness of the interconnected maze of interactions within the system and between the system and its environment. The overall complexity of the system is dominated far more by the interactions between the parts that makeup the whole than it is by the parts alone.
For any non-trivial system, simplicity often has less to do with the number and kind of different things involved and more to do with the number and kind of interdependencies between them. So achieving simplicity is less about managing "parts" of "things" or individual point-solutions, and is more about managing rules and relationships between the parts and things and solution-sets.
When dealing with large or complex systems (like most software, and software processes) the number of things (scale) and different types of things (diversity) that need to be managed is overwhelming. If we can come up with a modicum of modest, simple rules & principles to govern our design decisions in ways that help us minimize and manage interdependencies, eliminate constraints, and remove waste, then we are on the path to solving the real problem and meeting stakeholder needs in a way that is both simple and sustainable.
Testing and the Product Development Ecosystem
Software configuration management patterns starts with the idea that how your software configuration management is part of the larger context of your development environment. This includes the product architecture, your organizational structures, and, we can now add, testing strategy. For example, a product development organization that has a highly modular architecture, high communication between product and very good test coverage, might find itself needing to branch less frequently, as variations are addressed by the modularity and stability concerns by test coverage. Likewise, a team with a Big Ball of Mud architecture and poor test coverage may feel the need to maintain multiple codelines.
The SCM Pattern language is one mechanism for illustrating where testing fits in the process. SCM pattern language was written with the idea of building a system where you can be agile, lean, and minimize unnecessary branching. Let's discuss that and then cover how you can apply testing to simplify the SCM approaches for an existing process.
Patterns
A premise of SCM pattern language is doing just enough at each stage of the process that the team can make rapid progress. Rather than having long pre-commit tests, there should be various layers of testing. Before a check-in, a developer might run smoke tests and a number of unit tests, which may take a few minutes in total. Your automated integration build (you are practicing continuous integration?) would run all of the unit tests and perhaps some longer running regression tests, as well. Additionally, nightly or more frequently, you would also run a comprehensive test suite that might take a long time, but that provides more certainty. This approach makes it easier to do development on fewer codelines, as you don't have to sacrifice stability for progress
Another approach might be to have various codelines at various degrees of stability. Isolating work in the name of stability presents another problem: deferred integration. While using isolation for stability is helpful in the short term, it often leads to unpredictability in release scheduling. Integration is where unexpected problems often happen, even in the presence of good component tests. Because of this, integrating early and often, while also testing what you can when you can, is the best way to verify that you are on track and that the product is sound. Product integrity is part of the definition of SCM. And it all depends on tests!
When to Write and Execute Tests
A test is, in effect, is an executable specification of a change, at different levels depending on the type of test. An integration test specifies the end user behavior and a unit test defines the lower level contracts among components. From the configuration management perspective this translates to verifying that the configuration as it exists in the code line works as expected.
Tests provide two benefits:
- The ability to easily verify that the change that you are about to make does what you expect.
- A mechanism to verify that a change did not break existing contracts.
- To realize these benefits it's important to have tests to support any non-trivial change to the code (and code includes configuration data).
- When doing development for new features. This is what people typically think of when thinking of approaches like test driven development.
- When fixing defects, write tests that address the defect.
Some may not see the value in writing tests for defects, as much of the discussion around test driven development focuses on new feature development. By not writing tests for defects, you are missing a chance to test during much of your code writing, as some teams spend more time enhancing existing code than doing new development.
Paul Duvall and Andy Glover explain in their book Continuous Integration why testing during defect fixing is important. Because of the value of testing as a tool for verifying configuration, a development team that includes those involved in the build and release process should execute tests:
- In a developer workspace while developing
- In a developer workspace after updating and before committing changes
- In the integration workspace after an Integration Build
- In a release workspace after building a release candidate build.
- Anytime there is a change to code, meta data, or configuration.
By adding this testing to the CM discipline, you can have confidence that what you are releasing meets the expectations of the user community.
The Value of Automation
While testing adds value to your SCM process, it is not a very useful part of your process unless it's automated and part of an automated continuous integration process. Here's why: SCM is about identification and reproducibility and manual processes are error prone. While manual testing is better than no testing, an automatic, repeatable, testing process is the best way to ensure that your testing process helps to validate your configuration.
Roles (Again)
If testing is part of SCM process, then who's responsible for testing? The reality is that the various roles on a software project need to work together for the project to be successful. Everyone has a role in the testing process. The good news is that, by it's nature, testing will help you to understand how well everyone is working together.
We want to emphasize that testing is effective when it is understood to be a shared responsibility, as everyone benefits from an effective testing strategy. Testing can start with a developer (unit testing), and/ or a QA engineer (integration testing), but the activities need to be supported by the build and release engineers to allow for frequent running of tests as part of the build, and correct association of tests results with builds.
Summary
Testing is a cornerstone of agile practices because it helps provide the feedback that enables agile to work. As we said in our October column, citing Susan Dart, SCM is about ensuring the integrity of a product and making its evolution more manageable. Testing is a mechanism that provides the feedback necessary to ensure product integrity and manage change effectively and is part of the foundation of SCM.
