Learn how one tester brought developers into the test lab and discovered that by exploring software with testers, developers can learn how their software might be used and testers can learn more about the application itself.
As a black box tester working on a variety of development projects, I was often approached by developers who were trying out test-driven development and wanted to learn more about testing. To show them how I worked. I involved them in pair testing, a technique in which two people test an application at the same computer. Interestingly enough, the developers taught me as much as I taught them.
Let me explain how it worked. Before a pair testing session, the developer and I would meet to determine the focus and scope of the test. We would pick an area of the program to test and establish our goal. Sometimes the goal would be to track down an elusive bug, sometimes to ensure that customer acceptance test criteria were met, or sometimes to find bugs in a new piece of functionality. We would write our goals and testing ideas on a whiteboard, keeping a copy of the finished notes for our own use.
Later, the developer would join me at the computer. He would watch how my exploratory testing techniques led to new ideas and plans of attack. Soon, he too would be actively testing. As the sessions progressed, we would revisit our goals and come up with new ideas for testing.
Mutual Learning
The benefits of pair testing extend beyond the developer and tester involved. Developers learn how to test their own code more effectively and gain a new perspective on how their software might be used. Testers gain a more thorough understanding of the application they are testing and learn debugging techniques to find causes of defects. Both developers and testers learn how to uncover more information to write more effective defect reports. Additionally, pair testing can break down communication barriers between developers and testers and facilitate team building.
During pair testing sessions, developers regularly told me that they had never thought of looking at software testing the way I did. The "what would happen if I tried this..." mindset was not as intuitive to them. They tended to think of tests that reflected a typical way of using the software; I thought of tests that might cause failures. As I applied my usual techniques they typically would say something such as, "Stop! Slow down! What are you doing?" Then I would stop and explain my reasoning. The types of bounds conditions or input validation tests that I immediately tried on input fields of new features surprised them. They were also surprised by my attempts to get around the control flow of the program.
The developers weren't the only ones learning something new. I was often struck by how developers' perspectives and testing techniques were different from my own. I tend to focus on the user's perspective, while developers tend to focus on what the program tells them about what is going on behind the scenes. As they shared their knowledge of the underlying code, I learned what areas of the product might contain weaknesses and gained application-specific knowledge that helped me track down problems more effectively and provided more information when reporting a defect.
Once developers applied what they learned about testing to their development projects, they came up with great testing scenarios on their own. Best of all, it became difficult to find defects in the developer's code after pair testing.
Collaborative Problem Solving
Pair testing can also be used as a way to track down hard-to-reproduce defects. Once, after several hours of testing an application, I uncovered a defect that occurred only sporadically. I suspected clients had reported this defect in the form of several seemingly unrelated issues. To track it down, I pair tested with a senior developer. While he had a suspicion of what was causing the defect, we could not repeat it when testing together. At the end of that session, he installed a debugger on my machine and told me what information needed to be captured if it occurred again.
The next time the defect occurred, we tested together until we were confident we had narrowed down the cause of the problem. He knew what code was problematic, tests to narrow down where the problem might occur on the code level. From that list, I derived several more tests. In almost every case, the defect occurred. We were able to track down the problem, fix it, and develop some solid unit tests for the fix based on our sample of test cases.
The whole process took only a couple of days. In my experience as a tester, tracking down difficult defects on my own has taken much longer. Through collaboration, we found the cause more quickly and were able to ship the software with a high level of confidence.
Working Together in New Ways
Sometimes, instead of black box testing together at a machine, we collaborated on automated unit test ideas. We began by working on test cases that involved permutations and combinations. From the large number of possibilities, we determined a sample size and chose a set of test cases for automated unit test development. In addition to these tests, I suggested other testing ideas for developers to implement when writing their automated unit tests. When I tested the same feature, I complemented the developers' unit tests with scenarios they had not covered. Some developers were much more comfortable with this type of collaboration than with pair testing together at a computer in the testing lab.
Risks and Limitations
Pair testing is not a cure-all. As with any technique, there are risks involved, and some pair testing efforts fail. Some failures can be traced to what Agile QA Manager Janet Gregory calls a lack of trust between the developer and the tester. "If one or the other goes in with the idea that it is a one-way learning experience, the experience will fail." Pair testing is only effective in an environment of mutual respect and trust.
In pair programming, both people already understand program design and architecture. In testing, the developer might not understand the testing focus. Whoever is "driving" during pair testing must ensure that the other party is actively participating and understands what is going on. Encourage thinking and talking aloud, keeping the other person informed on the motivation behind your actions.
Teaming a tester who has a knack for black box testing and finding defects with someone who is intimately familiar with the underlying code is a great combination. Try it.
