There is a notion that exploratory testing is devoid of and incompatible with planning. Supporters and skeptics alike tend to have concerns that exploratory testing might not work in their organizations because of this perceived lack of planning. Is exploratory testing necessarily unplanned? Let's take a look at the nature of plans and how exploratory practices fit into the big picture.
My last column ("What Is Exploratory Testing?") provoked some interesting reactions. One came from Yogita Sahoo, who agreed that ET can be useful, then asked, "But can we get others to agree on such an unplanned process?" Ms. Sahoo's question raises some very important issues, such as:
- What is "planning"?
- Is exploratory testing necessarily unplanned?
- Why might we need to do things that aren't rigorously planned?
- How do we coordinate and support each other, if not with a plan?
Here's my short answer: Exploratory testing (ET), as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan. Rigor requires certainty and implies completeness, but I perform exploratory testing precisely because there's so much I don't know about the product and I know my testing can never be complete. Exploratory testers coordinate with each other and gain support in the same way executives run large companies: by talking with each other and building credibility over time. It's not rocket science, except that rocket scientists and all scientists also use exploratory tactics in their work.
Exploratory testing may well be planned…
The American Heritage Dictionary offers us four definitions of
"plan":
- A scheme, program, or method worked out beforehand for the accomplishment
of an objective; e.g., a plan of attack. - A proposed or tentative project or course of action; e.g., had no plans for the evening.
- A systematic arrangement of important parts; an outline or a sketch; e.g., the plan of a story.
- A drawing or diagram made to scale showing the structure or arrangement of something; e.g., the ventilation plans show two intake vents.
Notice that a plan can be a plan without necessarily being a specific, rigorous, complete, or recorded plan. Thus, one answer to Ms. Sahoo's question might be that good exploratory testing is a planned activity, even though it's not scripted in detail. I find it useful to define a test plan as "the set of ideas that guide the test process." This definition is consistent with at least the first two American Heritage definitions. In order to guide testing, the test plan must exist before the testing, but how long before? What if it comes into existence only moments before the testing? No problem. Nothing in the notion of planning prohibits simultaneous planning and execution. In football, when quarterbacks find that their receivers are too well covered, they may quickly make a new plan to run with the ball. An instant later, they execute the new plan. Thinking fast is no crime against plankind.
…but learning comes before the plan…
Still, I may be missing Ms. Sahoo's point. Another meaning of the planning question could be "How can we get others to agree on a process with such an open-ended and unpredictable plan?" To address that issue, it helps to invoke the most rigorous, repeatable activity you can imagine. Perhaps playing a flawless piano concerto. Now ask yourself, is there a plan in that activity? I would have to say yes. Now ask yourself, what happened before that plan existed? I would have to say some kind of rigorous composing and practicing process. Now, was that prior process guided by the same rigorous sort of plan? I suspect not. If we keep peeling back the layers of how things originated, we soon get to processes that obviously were not rigorously planned. There was a time before the choice was made to play that piece, or even learn the piano. Were those choices made according to formula, or were they subject to exploratory processes? For that matter the evolution of music, musical notation, the design of the piano, and methods of teaching piano all have evolved in non-rigorous ways.
Anything explicitly planned must have arisen from a less explicitly planned activity at some point in history. ET is exactly the kind of activity that comes before rigor and makes rigor possible. ET is primarily creative. It has value because it exposes problems, but it also has value because it is a way to create other kinds of tests. It's a great way to learn about a product. And rigorous test plans start with thorough understanding of the product. That gives us an answer to the planning question: an open-ended plan may be all that we have. If you already know for sure what the next test is that would be worth executing, by all means follow that plan. Otherwise, you'll have to figure that out for yourself. And that's an exploratory process.
You get someone to agree to such a process the same way you get someone to agree to allow a house to be built before they try to live in it. If they understand the difference between a house and an empty lot, and they want a house, and they see that there is no house, they may well agree that someone ought to build one.
…and dealing with a complex world requires skill, above all.
Still another way to think about Ms. Sahoo's question is that it's a query about how a relatively lightly planned activity can be organized and managed. This is where the myth of planning really rears its head. The truth is, we organize and manage all kinds of things without detailed plans. In the absence of pre-scripted moves, what do we do? We pay attention. We react to the situation at hand. We anticipate problems. Because we don't always know the right answers in advance, we have developed ways to come up with answers on the fly.
Similarly, we can train testers to excel safely in exploratory testing: start small, provide coaching, learn from failure, and build on successes. I don't send testers on complex exploratory missions until they have gained credibility on the easier missions. Testers must show an ability to design a good test strategy, execute good tests, find important problems, and report them in a compelling way. Furthermore, they must be able to explain their work. Only under those conditions do I come to trust a tester's exploratory work.
Remember it's the skill of the tester, the social system of the project team, and the testing situation at any given moment that enable excellent exploratory testing. Don't dwell too much on planning. If you're challenged to defend exploratory testing on the basis that it's not planned, reply that when the developers let you know where all the bugs are, you will happily write a precise and accurate plan that visits each one. Until then you must spend some of your time exploring in order to find important bugs fast.
