As the Agile environment has efficient principles that allow quick responses to changes and the ability to deal with uncertainty, exploratory testing may seem like a perfect match for such projects. However, this is only partially true. In reality, diverse reasons impede its usage in Agile projects, and companies should take them into account before designing QA strategies.
Exploratory testing can be seen as traveling on foot instead of taking the bus, offering the freedom to deviate from a specific route. As the Agile environment has efficient principles that allow quick responses to changes and the ability to deal with uncertainty, exploratory testing may seem like a perfect match for such projects. However, this is only partially true.
In reality, diverse reasons impede its usage in Agile projects, and companies should take them into account before designing QA strategies. In this article, I’ll focus on these factors and share insights into the most suitable cases for applying exploratory testing.
The Ins and Outs of Exploratory Testing
In traditional testing, QA teams rely on a structured, step-by-step approach. This means they, in advance, must compile user journeys that lay a foundation for test cases, regularly trace coverage with appropriate metrics, and can easily reproduce captured defects. Deviations from the scripts aren’t welcome, feedback on software quality isn’t so fast, and alterations to requirements slow down the process even more.
Additionally, ad hoc testing, being random, intuitive, and informal, involves learning a tested system to be aware of its specifics and execute further testing without a test plan. It isn’t a mandatory part of the testing process and mainly focuses on negative scenarios.
Exploratory testing, on the other hand, is an on-the-fly activity of discovering an application to identify and document software issues. Product learning, test design, and execution happen concurrently, emphasizing an unstructured nature of testing. It provides rapid feedback as it helps detect defects quickly and works well when requirements are changing, or the QA teams are pressed for time. That said, it’s important to note that it doesn’t ensure complete test coverage.
Best Exploratory Testing Use Cases in Agile
Agile software development assumes collaboration, high flexibility, quick response to changes, and short sprints (up to 2 weeks). QA teams can apply exploratory testing in 6 cases:
- Learning the system under test. Let’s assume that the QA team joined the project upon the completion of several sprints. Exploratory testing helps to quickly gain sufficient knowledge of product operation, obtain the first documented results, and build the ground for a test model for further script-based verifications.
- Approaching high-pressure deadlines. Supplementing acceptance testing with exploratory checks is a win-win approach, as it increases testing coverage and quickly assesses system quality.
- Lack of fresh perspective. Sometimes there’s a need to verify software soundness without following developed documentation and think outside the box to discover novel, not-so-obvious use cases. For instance, if you have some free time, you may ask project newcomers to perform an assessment with exploratory testing. While using unconventional thinking, they may ask additional questions on software specifics and detect undiscovered issues, thus enhancing test coverage.
- Limited budget. Even in cases of financial constraint, organizations can focus on software quality by hiring an exploratory testing expert who will investigate the application from an end-user perspective rather than spending time on creating test documentation, reporting, and other activities.
- An intricate system with multiple branches in logic and an endless variety of data sets. When building a test model for such solutions, different test design techniques are required and exploratory testing can cope with this task faster and better.
- Minimal requirements. Often in pursuit of a rapid increase in functionality, less time is spent on detailed work with requirements, which leads to high-level or even vague user stories. In this case, analyzing the interaction between the old and new software functionalities falls on the QA team’s shoulders. Exploratory testing performed by seasoned team members helps to quickly delve into these specifics, provide fast feedback for the developers, and write meaningful test cases for further usage by less experienced QA specialists.
Exploratory testing is well-suited for these scenarios, providing QA teams with a versatile approach to ensuring quality.
Flip Side of Exploratory Testing in Agile
Despite its advantages, there are situations where exploratory testing in Agile software development may not be the best choice. Consider the following key points:
Reason 1. Inexperienced QA team
Successful exploratory testing requires creative and experienced QA engineers familiar with similar software solutions, business logic, and technologies. This knowledge helps them identify defect-prone areas and efficiently decide which features to test based on the progress made.
Reason 2. Shortage of exploratory testing expertise
QA engineers should have a solid background in exploratory testing, its techniques, and best practices, as well as an understanding of when to apply freestyle and session-based approaches to maximize QA efficiency. A lack of knowledge will likely lead to poor testing results.
Reason 3. System criticality
When we talk about complicated IT products such as banking solutions or ERP software, the highest priority is to delve deep into all the aspects of their sophisticated business logic and requirements. It’s also vital to understand software architecture to detect issues and ensure continuous, flawless operation of mission-critical functionality.
Therefore, exploratory testing alone isn’t enough to test the nuances of complex software. Instead, one should utilize diverse test design techniques to ensure maximum test coverage.
Reason 4. A poor basis for introducing test automation
If you need to accelerate testing time and expand test coverage by introducing test automation, exploratory testing of any type isn’t fit for purpose. To set up automated workflows and obtain high ROI, you need to write test cases with structured test steps, preconditions, as well as expected and actual results.
Reason 5. A limited ability to learn from mistakes
Picture this: your team has performed exploratory testing, and the product has gone live, suddenly a critical defect occurs in the production environment negatively affecting stable software operation and worse, impeding profit.
After fixing the issue, the first thing you need to do is to analyze the root cause during a retrospective. This analysis allows you to adjust your test strategy so that this incident never happens again. Yet, when sticking to the traditional freestyle approach of exploratory testing, test run results may not be fully documented, complicating the process of learning from your mistakes and resulting in increased business risks.
If you still opt for exploratory testing, ensure you use a session-based approach. This involves writing a test charter with test objectives to guide the process and a test session sheet to document the steps and findings in order to make QA more structured.
Reason 6. Overlapping tests
In freestyle exploratory testing, the lack of formalized test documentation may result in duplicated work when the QA teams check the same scenarios more than once. It adversely impacts time to market and overall project efficiency. Part of the functionality can even be left untested. So, if you still consider exploratory testing a fit-for-purpose solution, choose its session-based variant. It requires identifying the testing scope for each team member in advance to avoid duplication of work.
Reason 7. Compliance-based testing
If there’s a need to ensure that the software meets international standards such as WCAG 2.2 or OWASP, the QA team must strictly follow a set checklist. FDA, HIPAA, and other healthcare regulations require even greater attention as each step should be documented to ensure the IT product provides secure storage and sharing of sensitive ePHI and doesn’t pose a threat to patients’ lives. Exploratory testing doesn’t align with these activities.
The Takeaway
The primary objectives of QA and testing activities in general are to ensure a sound, high-performance software operation that helps businesses attain set goals and for end users to obtain an intuitive, user-friendly IT product.
Exploratory testing, in turn, helps review the quality of a product from a user perspective, think outside the box, and find unexpected ways of uncovering concealed issues. Without a doubt, it’s an effective addition to a testing strategy but not the first necessity—especially for Agile projects. So, plan smartly and pick what’s best for your business.
User Comments
Hi, I have read your article and I disagree with your conclusion that exploratory testing is not ideal for agile projects. I think exploratory testing has many benefits and value for agile teams, such as flexibility, collaboration, efficiency, and learning. I have also found some articles that support my opinion and provide more insights on exploratory testing in agile contexts. I hope this helps. I think exploratory testing is a valuable and effective technique for agile projects. Thank you for sharing your perspective.
Best regards, Hanna
Hi,
It feels like the conlcusion of your article should be "Exploratory testing is not ideal for testing requirments". Nothing in your article really relates to Agile software development. As mentioned in the other comment, exploratory testing is really useful in an Agile world where failing fast is an option.
Exploratory Testing helps testers to get more creative and not get loaded with routine testing activities. Findings from exploratory testing approach can help testing teams to build a strong test management system, which captures learning and expertise of various members in the testing team.
It will be helpful If you woulde have explained it with an example.
@Vitaly Prus
This article is based on fundamental misunderstandings about testing in general, and the exploratory dimension of testing in particular. Those misunderstandings aren't particularly new, alas.
1. "Product learning, test design, and execution happen concurrently, emphasizing an unstructured nature of testing."
This makes me wonder what the author thinks "concurrently", "execution", and "structure" mean.
Structure refers to patterns that persist in a system. I suspect that the author is considering only one kind of structure: a formalized, proceduralized process for performing a test. Learning and testing are both exploratory processes; that is, there is no script for learning things, and there is no set of scripts that will take you directly to each important and hidden bug in the product.
There are, however plenty of things that strucure both learning and testing: our testing mission; our models of the product, and how to cover it with testing; our models of risk; our models of quality criteria (that apply not only to uses, but to all of the other stakeholders of the product); our experiences and experiments with the product...
These things happen concurrently and interactively, but not necessarily simultaneously, and not necessarily at equal intensity. While doing testing work, we may be performing a test, or we may be reflecting on what has happened so far and what we might do next. That is, our level of direct interaction with the product may be very high or quite low from one moment to the next.
Meanwhile, each of the points that is cited here as "not the best choice" can be dramatically false.
Inexperienced QA team
Successful exploratory testing requires creative and experienced QA engineers familiar with similar software solutions, business logic, and technologies. This knowledge helps them identify defect-prone areas and efficiently decide which features to test based on the progress made.
How do we learn about a product? How do we gain experience with it, such that we can test it most effectively? One greaf and fast way to learn about a product is by testing it, or by trying to test it.
Reason 2. Shortage of exploratory testing expertise
QA engineers should have a solid background in exploratory testing, its techniques, and best practices, as well as an understanding of when to apply freestyle and session-based approaches to maximize QA efficiency. A lack of knowledge will likely lead to poor testing results.
One could take the word "exploratory" out of that heading and that paragraph without any change or loss of meaning.
Reason 3. System criticality
When we talk about complicated IT products such as banking solutions or ERP software, the highest priority is to delve deep into all the aspects of their sophisticated business logic and requirements. It’s also vital to understand software architecture to detect issues and ensure continuous, flawless operation of mission-critical functionality.
Therefore, exploratory testing alone isn’t enough to test the nuances of complex software. Instead, one should utilize diverse test design techniques to ensure maximum test coverage.
That miight make sense if exploratory testing were a technique, but it isn't. The exploratory nature of testing — that is, self-direction and agency on the part of the tester — can be applied to any test technique. And again, see the response to point (1) above; exploration is necessary to learn how to test the product, and to identify which techniques might or might not be useful.
Reason 4. A poor basis for introducing test automation
If you need to accelerate testing time and expand test coverage by introducing test automation, exploratory testing of any type isn’t fit for purpose. To set up automated workflows and obtain high ROI, you need to write test cases with structured test steps, preconditions, as well as expected and actual results.
Where do scripts come from? They come from exploratory work, which makes sense, because programming itself comes from exploratory work. There's no script that tells you what risks to focus on. There's no script that tells you what a good check might or might not be. There's no script that tells you how to write a script.
Reason 5. A limited ability to learn from mistakes
...when sticking to the traditional freestyle approach of exploratory testing, test run results may not be fully documented, complicating the process of learning from your mistakes and resulting in increased business risks.
Whose "traditional" approach to exploratory testing is that? Exploratory testing is not by nature undocumented. https://developsense.com/blog/2011/12/what-exploratory-testing-is-not-pa...
I think what the author is saying here is that bad record-keeping makes it harder to troubleshoot problems. When we unpack that, it amounts to saying "bad testing is bad", but that applies to scripted approaches to testing as to anything else. The author's advice to document findings is well-placed irrespective of the approach.
Reason 6. Overlapping tests
In freestyle exploratory testing, the lack of formalized test documentation may result in duplicated work when the QA teams check the same scenarios more than once.
What the author seems to be saying is that poorly managed testing may result in redundant work, but formalized scripted approaches can be poorly managed too.
Coverage outlines, risk lists, and scenarios can help to structure session-based work.
Reason 7. Compliance-based testing
If there’s a need to ensure that the software meets international standards such as WCAG 2.2 or OWASP, the QA team must strictly follow a set checklist. FDA, HIPAA, and other healthcare regulations require even greater attention as each step should be documented to ensure the IT product provides secure storage and sharing of sensitive ePHI and doesn’t pose a threat to patients’ lives. Exploratory testing doesn’t align with these activities.
This is incorrect. The FDA does require evidence, but it does NOT require that "each step should be documented". See the principle of the least burdensome approach here https://www.fda.gov/regulatory-information/search-fda-guidance-documents...
And see the FDA's emphasis on exploratory work here: https://www.satisfice.com/blog/archives/602
Finally, testing is by its nature an exploratory process. Talking about exploratory testing is like talking about "vegetarian cauliflower" or "carbon-based human beings".
http://www.satisfice.com/blog/archives/1509
Thank you @Mr. Bolton for your well-reasoned response. As I read this article, I thought that it was a good way to solicit positive reasons to use exploratory testing on agile projects. Meaning no disrespect to the author, I did have the impression that there was a fundamental misunderstanding of the nature of testing in general, and of exploratory testing in particular.
Thank you @Mr. Prus for writing an engaging article. It is often the discussion afterwards that leads to learning.
thank sir for elaborating on Exploratory Testing and its usage in agile development.
A good article with plenty to disagree on, but well written