In short, redistributed testing is a shift in the emphasis and responsibility for testing. Testers are reassigned to work closer to the business with users or business analysts or are embedded in the development team.By being involved in story and scenario writing, the testers help to refine requirements and improve their quality. How could your systems benefit from redistributed testing?
In 2011 I started using the term redistributed testing to describe a change in the emphasis and responsibility for testing that some companies were embarking on. I talked about it in a blog post in which I suggested forces outside testing are causing companies to rethink how they test. Preceding this, the alarmist death-of-testing meme brought people’s attention to the fact that things were changing. It looks like in the UK and Europe, at least, the change to redistributed testing is gaining momentum.
So, what exactly is redistributed testing? I use this label to describe the impact on testing from a number of (mostly agile) approaches that companies are embarking on. The most familiar are behavior-driven development (BDD), test-driven development (TDD), acceptance-(test-)driven development, specification by example, DevOps, and, most recently, shift-left. It’s often that teams implement some form of hybrid mix of two or three of these approaches. I’m not going to define each of these in this article. Rather, I want to summarize the changes as they affect testing and testers.
Redistributed Testing Defined
In short, redistributed testing is a shift in the emphasis and responsibility for testing.
1. Some testers used to working in system test teams are reassigned to work closer to the business with users or business analysts or are embedded in the development team. They operate in two modes, as explained in the next two points.
2. By being involved in story and scenario writing, the testers validate and help to refine requirements and improve their quality. The outputs are trusted requirements and examples that developers find more reliable as sources of knowledge to test and write code.
3. The examples, written in plain text in a structured format, are used with a BDD tool to generate test automation code. Developers (usually) create the test steps or fixtures to drive feature-level tests and the TDD process.
Benefits of Redistributed Testing
This change provides four key benefits.
First, testers ask what-if questions of the customer and create scenarios before the code is created, which leads to a specification that is much less prone to misunderstanding and misinterpretation. That means fewer arguments at the end of the process about what the software “should” do or whether an unintended consequence is expected or a bug that needs to be fixed. This process results in better (we call them “trusted”) requirements and stories. The developers are involved in this process, but the testers are prominent and do most of the challenging.
Second, the structured stories and scenarios provide concrete examples of features in use. They typically describe behaviors and not implementation details. In this way, the developer can use these behaviors to derive automated tests of components, classes, or services. These automated, feature-level checks are helpful to the developers in their design process, and, if retained in a continuous integration environment, they become a natural byproduct of the development process.
Third, system testers know that many of the checks they might have expected to run because developer testing was poor or not visible may already be automated. This frees the testers from much of the repetitive, time-consuming feature-based checking and allows them to concentrate on end-to-end and exploratory tests.
Finally, if developers have a more accurate definition of the requirements and a significant number of feature-level checks are performed automatically, then the quality of software delivered to test should be enhanced. You would expect fewer system testers are required to run fewer tests on better software.
Extreme Redistribution
If you take these ideas to an extreme, you end up with an ideal that looks very much like a DevOps environment that practices continuous delivery. In this case, the testing is highly automated and runs under continuous integration. The time left for manual, acceptance, or user testing between delivery and deployment is minimal. There may only be a limited number of testers who test in this small window between delivery and deployment, and their challenge is to test whatever they can, however they can in a very short time. Flickr famously reported that it deploys to production ten times a day—how do companies like this do enough testing?
Testing before deployment is not the end of testing. Predeployment testing is often supplemented by testing so-called dark launches (canary releases) in production. Other in-production approaches include A/B testing and blue-green deployments tested by selected customers. Needless to say, the operation of a continuous delivery approach requires a high degree of discipline and control—and the trust of customers.
What does this mean for testers? The most important lesson of agile is that smaller, multidisciplined teams benefit from rapid feedback and continuous communication. Clearly, the testers (and developers and business analysts) need to adapt to working in much closer proximity. Testers in particular may find that they need to hone their business and analysis skills and perhaps even learn how to write code to ensure they can work in such close harmony with technical peers.
The redistribution of testing does not mean the core disciplines of testing are abandoned. A questioning, skeptical attitude is a prerequisite for writing good story scenarios essential to making the BDD process work. Testers will find themselves working much closer with developers—maybe even pairing with them—in figuring out how to test and design features directly.
The time available to test after delivery may be much shorter, so the tester needs to let go of working in test stages. Rather, the best approach may be to use a variety of techniques throughout the delivery process while testing all the way, using whatever incremental build or materials are available. I have described an agile test strategy in terms of interventions here.
Agile, shift-left, and continuous-delivery approaches will not be adopted by everyone. For example, it is hard to see high-integrity environments adopting a daily release-to-production regime. Banking, retail, and manufacturing businesses, for example, will probably not adopt such practices for their core systems.
However, online businesses that have grown very rapidly from small start-ups might know no other way. There are enough success stories from these now-substantial businesses to make the banks, retailers, and manufacturers think that their Internet-based systems might benefit from continuous delivery and redistributed testing. So even if you work for an established or corporate business, you might be called upon to change your game.
User Comments
A very timely article. Thanks Paul. We have been hearing a lot about shifting left. Recently at a conference I was in, one of the speakers spoke about the time to "shift right" with a hope to have test teams start interacting more with end users. I agree shift left or right may be just different terminologies that different groups may be using...would love to hear your take on test teams having to "shift right".
There's nothing new under the sun ;) When I first started teaching requirements management nearly twenty years ago, one of the key practices was to involve all stakeholders as early as possible. This includes testers. A tester asking "how can I test this?" or, more directly, "how will you know when this requirement is met?" is a great aid for refining and understanding requirements. Testers are also some of the best people at identifying ambiguities. If it can't be tested, it isn't clear enough.