Exploratory tests, unlike scripted tests, are not defined in advance or carried out precisely according to a plan. So where and how do they fit with the other tasks testers must perform? James Bach, a chief proponent of exploratory testing, provides some insight on how best to exercise exploration in your testing effort.
If you, like me, find the exploratory approach to testing valuable (see my recent column "What Is Exploratory Testing?"), then the questions arise: When do you do it? How does it relate to the software lifecycle? StickyMinds Review Team member Yogita Sahoo sent me an insightful list of questions and comments about exploratory testing (ET). I'll be responding to some of them in future columns. For instance, she writes, "In my personal experience, if I start exploring things when I'm supposed to carry on with my allotted tests, the whole thing gets jumbled up. I can neither concentrate fully on ET nor on my regular test cases. That results in less productivity." Let me address that.
A simple way to think of ET is concurrent test design and test execution. To help Yogita better, I want to use a more specific definition: Any testing is exploratory to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.
This definition includes pure exploratory testing, where you explore the product and design a test strategy and specific tests based on your understanding of your mission as a tester, but without any specific guidance. It includes chartered exploratory testing, where you have a specific assignment for what to test and what techniques to use, but no designated procedures. It also includes improvisational testing, where you elaborate on a test procedure or use it to inspire a set of related tests. It includes test procedure creation, too, inasmuch as you perform tests while documenting them.
It's more about how to do ET well, than when to do it.
All testing done by humans is exploratory to some degree, because humans are not robots. That means the question is not when do you do exploratory testing, but rather how exploratory is the testing that you do, and how well do you do it. When I consult about test process, I don't suggest that my clients perform exploratory testing. Instead, I help them become aware of all the ET that they already do, which is probably mixed up with some form of scripted testing (or pseudo-scripted testing, where testers say they follow the procedures, but don't). Once I can get their exploratory testing to "come out of the closet," I can help them improve their skills and overall test strategy so that they do ET, and any other testing, better.
Exploratory testing is all you have at the beginning…
ET fits at the beginning of the test project because test procedures don't yet exist for the new technology being developed. Even if they do exist, you have to learn the product (that requires exploring and questioning it), and the procedures would have to be reviewed and upgraded. The process of writing test procedures is exploratory. Watch anyone, or yourself, writing a test script, and you'll see those thought processes at work.
…and it's how you create diversity in tests later on.
ET fits into the middle of a test project, even when you have lots of scripted tests. Be exploratory in the sense that a tourist on a tour bus is exploratory. Let your allotted tests take you to visit different parts of the product, then improvise on the theme of those tests, briefly. Spend a few minutes working through variations of the tests, then get back on the tour bus and do the next scripted test.
It's also how you assure that there is enough variation and creativity in the test cycle.
Also, throughout the project, I would suggest that you question the value of the tests allotted to you, since no test process provides complete coverage. To improve the breadth and depth of your testing, consider allotting some time, maybe 20 percent or maybe 80 percent (there's no universal "right" amount of time, other than what fulfills the mission of testing) per test cycle to pure exploratory testing. Pick one or more risk areas in the product and design and execute tests for that, seeking to find important problems fast and collecting information that will help the project evaluate the state of the product.
Another way to fit ET into a project is to dedicate a particular tester or test to continuous duty doing pure exploratory testing. I ran a team like that once, and I also interviewed a fellow who ran such a team at Nortel. These teams are well trained, and work like a reconnaissance unit, scouring the product and following up on rumors and risk areas.
Doing exploratory testing well requires skill, no matter when you do it.
Before I made it my goal to master the art of simultaneous test design and test execution, I felt confused and bewildered by the process, too. I eventually developed certain heuristics, notetaking protocols, and skill in modeling, reasoning, communication, and self-management that allow me to be productive under almost any circumstances. The process is creative, but it's a learnable discipline that fits anywhere testers are expected to use their minds.
