My friend called me to discuss a situation at his workplace. He used to be the sole tester for a project and described how he would write down test cases when he got a test idea. Slowly, the team grew in size and the product added more features, and the number of test cases swelled up to a thousand.
As the test cases increased, the details for each test case were not given much importance. Everyone understood what each test case meant, so it was not a big problem that demanded immediate attention.
Then came the plan to automate the test cases, and multiple problems surfaced from nowhere. The automation engineers did not understand the test cases because they were not detailed enough. The tool used for writing test cases did not support their bulk export. The test case suite had many redundant test cases. The final problem was answering to management, who wanted to know what was holding up automating the test cases. My friend asked me if I knew any tool that could solve all his problems.
I had resisted the urge to interrupt him throughout the call, but at this point I asked him just one question: “What problem or problems are you trying to solve?”
He mentioned the tool again. This time I interrupted and asked if the automation engineers’ ability to use the existing scripts is a problem to solve. He answered in the affirmative and also wanted to discuss adding more details to the existing test cases.
Starting from the first problem, we talked about how the automation engineers’ concern is legitimate. We could start with a sample set of test cases with enough details and give it to the automation engineers. Instead of providing the full set of test cases with meticulous details, we figured we could start with the minimum and see if any new problems crop up as part of this exercise.
The next problem was the complete test suite missing details. I told him about checklists and mind maps, which can replace the test cases. We thought that if people complain about the lack of details in the test cases, my friend could tell them how much time can be saved by keeping the test cases lean, and how that time can instead be spent on interacting with the product.
If anyone on his team worried about some of the test ideas not being tested, I said they would have to spend energy on training the testers rather than making the test cases bulky. Unless your context demands detailed test cases with test data, checklists and mind maps are a healthy alternative to bulky test cases because they are easy to review and update.
At this point we had established two problems: Automation engineers needed detailed test cases, and the thousand test cases needed some maintenance. We decided that while the automation engineers are coming up with the scripts, my friend’s team could spend time converting the test cases to mind maps or checklists. A conversation that started with trying to find a tool to solve all my friend’s issues ended with our realizing a tool wasn’t even necessary.
How many times have you started to solve a particular problem and realized midway that the actual problem is not what you thought it was? While I waited for my friend to try out some of the tactics we discussed, I started thinking about our discussion. Here are some takeaways.
Tackle Problems Individually
I love how in their book Are Your Lights On? Jerry Weinberg and Donald C. Gause define a problem as a difference between things as desired and things as perceived. Many people think multiple problems can be solved all at once, but it is easier to examine problems one by one. In this case, my friend and I attempted to solve the automation test suite problem, followed by the detailed test cases problem, and then there were no more immediate problems to solve.
A Problem Is Relative
When it comes to software testing, a bug isn’t an absolute; it is a relationship between the product and some person. Similarly, a problem is also a relationship between a situation and a person. If you modify the person’s expectations or the situation, the initial problem might just disappear.
Problems and Solutions Are Not Permanent
When my friend’s testing team was small, the test cases were quite dynamic and it was not a problem. The same test cases became a problem once the automation engineers jumped into the context. The solution for one problem might cause another problem. You can decide which problems are manageable for now.
Consider Cost versus Value versus Risk
When thinking about a possible solution to a problem, ask yourself:
- What is the cost of this action?
- What is the value of performing this action?
- What is the risk of not performing this action?
Test your theory with a smaller sample before you spend a lot of time, money, or energy on it.
Not All Tools Solve Problems
As soon as they encounter a problem, many people think of tools as the solution. But sometimes, tools solve some problems in the short run and create many more in the future. Think through the risks before choosing a tool.
If you get stuck while trying to solve a problem, it helps to talk to someone who is not part of the context to get a different perspective. Different might not necessarily mean better, but it may help you think of something you hadn’t thought of before.
What would you have suggested to my friend? Share your thoughts in the comments.
User Comments
Interesting article. Regarding a remedy, I have no easy answer. Regarding the problem, I would say that improper process is at fault. In most cases, projects will end up needing to save time somewhere and that usually ends up requiring the test group to automate.
In my opinion, most test cases should be atomic in nature and thus fairly lean. The process issues arise when stakeholders mistakenly assume that writing complete test cases takes too long and substitute shortcuts like checklists and/or bullets test objective lists. This can work when the group is only a few.
Once the group grows, all the time thought saved gets lost at multiples. I have seen this exact scenario play out too many times to count. Your example is proof.
Usually the test writer is the SME, and really is the one you want detailing the test cases. Of course there are exceptions, but my experience tells me that while you can get away with cutting corners in some places, writing solid test cases is no one of them.
Fight hard and explain to project stakeholders why this is so.
That is an interesting statement, Craig. Maybe, we should ask those who want to save time if they want to save time or reduce testing time? What is their intention?
Do you think we can use checklists irrespective of the team size? I would like the team members to spend more time interacting with the product than on documentation. Your thoughts?
Thanks for the comment.