I have been doing software quality assurance since 1996, and in that time, Prior to the advent of agile development, good test documentation included a formal test plan and a set of test cases. But with agile, formal test plans have been thrown away.
In my experience working for several different software development companies, I have seen the scope of test cases range from practically nonexistent to excruciatingly detailed, to the point where you can barely make it through the muck and the mire to know how to test the application. The truth of the matter is that the ideal end result lies somewhere in between.
What are some of the roadblocks to producing good test cases, and how can they be addressed? Let’s look at test cases that can be derived from acceptance criteria, uses cases, feature specifications, and design documents.
First, I have found these to be some of the challenges:
- Testing is squeezed in at the end of the sprint, shifting the priority from comprehensive testing to bare minimal testing, and documentation becomes a task relegated to the “I’ll do it when I have time” bucket
- A lack of good supporting documentation results in a lack of complete understanding of the given feature or set of features
- There’s a lack of interest in writing and understanding the details required to build and test an application
- Testers think ad hoc testing is enough to check the testing box
- A corporate culture doesn’t make quality a priority, so there is not even a loose standard to plug into
Despite these challenges, I am reminded of an Arthur Ashe quote: “To achieve greatness, start where you are, use what you have, do what you can.”
We know we face the challenges of not having enough time, lacking supporting documentation, and addressing cultural issues to produce any type of formal documentation. But considering the above quote, there is a minimum that should be produced. It is essential to have some type of document to refer to when you test software to ensure testing is consistent and results are documented. How do we address some of the challenges?
Barriers to Good Test Documentation
Despite best efforts, and even in an agile environment, QA seems to get everything handed to them to test at the last minute or the very end of a sprint. What can be done at the eleventh hour to ensure a quality application?
Well, while waiting for a testable deliverable, hopefully there should have been ample time to learn about the feature, read supporting documentation, and talk to the appropriate members of the product and development teams in order to come to an understanding of exactly what you need to do to test a feature. From this information, QA can develop a set of test cases to ensure whatever you need is in place prior to beginning test execution. Assuming the feature is relatively small, given that it was developed within a short sprint, you should be poised to execute. The time waiting for a feature to be delivered should be used to understand, define, and document your strategy for testing.
What about supporting documentation? Recently, I’ve seen more cases closer to no supporting documentation on the spectrum than to extremely detailed documentation. Most organizations require functional, nonfunctional, and technical specs, but even when those are all present, you cannot be sure that those documents contain everything you need to effectively test the feature. A good tester will go above and beyond what is given to them and seek out more information to fully grasp the feature.
Here are some of the things you can do to derive what you need for developing a solid set of test cases:
- Read whatever documentation is available to you
- Think critically about what you have read and formulate questions about it
- Proactively engage with people to get your questions answered
- Developers for technical questions
- Product managers for feature questions
- Field people to understand how this is going to work with your customers in the real world
- Assemble the information you gathered into understandable and logically flowing test cases
One reason I’ve seen test cases and other types of test documentation created poorly is that testers are not confident in their writing ability and are fearful of producing any type of written document. If there is a need for improvement in that area, it is up to the individual to build up that skill set.
The good news is this type of writing is not the same writing as is used for a great work of literary fiction or even an academic research paper. Your document could be as simple as a bulleted checklist:
- Log in as a systems administrator
- Run this SQL …
- Make this API call …
Keep in mind that something is better than nothing, and the simpler, the better. Well-documented test cases become even more important when someone other than the originator needs to use them, like a test automation engineer who might not know the product features as well as you, or another tester who might need to regression test the feature and has not worked with it much in the past. You should ease the burden for the next person down the line by providing coherent test documentation.
The final challenge I have seen is that people struggle with the amount of detail they provide in their test documentation. Again, I have seen a range from nauseating detail to barely any detail at all. As with most things, the goal here is find a balance and what works within your organization.
A good rule of thumb is to think about your audience and ask yourself some questions. How much can I assume is common knowledge? Where do I need to assume the least amount of knowledge? (For instance, based on new people joining the organization, how much can you assume is covered in onboarding and any initial training that might be offered?) What are some of the pitfalls I ran into while testing, and how likely is that others would run into the same ones? Provide enough documentation so that when others end up in a blind alley, they can find their way out of it.
What You Can Do to Ensure Valuable Documentation
Generally, all these issues can be overcome by being proactive about accomplishing what needs to get done in order to get a quality product out the door with limited time and resources. Do not wait for anything or assume that what is delivered to you in terms of documentation or specifications is going to be everything you need
Also, be involved in what is coming down the pike before anything has come to fruition—in other words, do not wait until feature development is done to start testing. Be involved, ask questions, and help your team understand how to test for quality.
Make it your responsibility to stay informed of new features on the roadmap. Ask questions early on and keep your own notes. This will help you understand the feature prior to it even being put into a sprint or release, and your notes will be the start of your test cases. If you still get squeezed at the end, you are at least ready to “fire the gun” in terms of test execution.
The strategy of staying on top of feature development early on in a feature’s genesis will help remediate the problem with a lack of supporting documentation. If you are inquisitive because of nonexistent or minimal specifications, it will encourage those responsible for documentation to make sure it gets written. At the very least, specifications will be documented via the test cases you have produced.
To address the final point, if your organization does not promote a culture of documentation and sharing information, you should become a champion for a quality-centric culture. Consider it essential to your job to ensure the shipped product is released with the highest quality and does what it is expected to do, without issues that interrupt the normal operation of business or of the people and organizations who are using it.
Pure agile proponents may say you need to go very light on the documentation. I agree that test documentation does not need to be extremely heavy, but it does need to exist, be clear, and cover all that the product is intended to do. It’s a determining factor that results in positive gain for all around: for the quality of the product going out the door, for team members, for the end-user, for your organization, and for you in your career.
