Why Do I Test?

An email discussion group some time back had a debate on why one should test. Some points made included the following: We test to find errors. We test to show software works. Managers often view testers as a non-value-added task or a waste. What customer uses test products directly? None. And certainly, many managers and company owners would like to see testing activities done away with.

Because, what you and I produce (tests) are not of direct use to the customer, testing is a non-value-added tax, which doesn't produce any sellable product. In the email discussion groups, there were many opinions voiced. Some authors expressed great personal reasons on why they test. But I decided to think about the reasons I test because I needed to understand my own raison d'être. As strange as it may seem, my schizophrenic nature enjoys software testing. I find that testing and quality are parts on a multiheaded beast.

Let's consider the heads of the beast. The complex and diverse heads keep the test life interesting and always a surprise. James Whittaker says testing is not art or craft, but testing is primarily a discipline that can never be fully learned and can always be improved ("Mastering the Discipline of Testing," STQE magazine, Nov/Dec 2002). I agree with this view and it is another of the heads of the beast. Testing certainly has elements of art and of craft. It also has engineering and science, both of which I like. It has management, mathematics, psychology, economics, logic, and much more. Again, I like these also. So these diversities all make my schizophrenic side(s) happy.

Like any multiheaded beast, there is a dark side. I do battle with the poor quality beast-head. I do not fight the developers, for we are a team battling to create a quality product. The teams along with customers are battling the invisible head known as complexity. That complexity often defeats our understanding and shows itself in the incarnation of bugs in software. The testers are but agents of delivery trying to free the functionality of the software from the miscommunication of the beast, which result anytime you have complexity. The miscommunication is between all the players: development, quality, management, users, customers, sales, systems engineering, and others. Miscommunication and complexity result in faulty understanding and thus, errors.

Many ideas, tools, techniques, visualizations, methods, and approach concepts are aimed at these multiple heads. At the heart of software, and software testing, is language. These languages include languages of computers, humans, tools, and notations (not to mention "thought-language" which we are sometimes expected to "read"). Some people believe that many human wars start because of language and the resulting communication-understanding problems. And so it is with software. We use one language to formulate the problems, we use another language to formulate a solution, but because no one player (user, developer, tester, computer) understands all of the languages used, we create yet more languages that try to bridge the problem-space-gap. We end up creating a modern tower of "babble." So, years back and even today, people put forth solutions.

Some of the first solutions were from the formal heavyweight process people. Some of them came along and said, be process based with heavy documentation, and all will be well. But it was not. The industry had semantic and syntactic problems between the players in our systems. We would slay one head of the quality monster, only to have another replace it. Nobody was happy. The whole story of heavyweight, process-focused ideas is much longer than I care to deal with in this article or even in a long book.

Recently, on the scene have come the "Jim Dandy" agile warriors to the rescue. They say the code, the product, and people, are most important and that's that. We, the team, should focus by having everyone involved in the testing and coding as a complete product. Good point. The whole team must be made up of developers, testers, management, and users. But here is one problem with that idea. The industry developed all these other systems of languages, because having the "whole team" understanding and communicating the same way is not always possible.

In the early days of software, there was only binary code (0 or 1), and very few people could deal with them (how many of you have ever written a program in hex? I have). The very early days were probably very agile (one guy knew what to do and was the same person that coded 0 and 1). Back then, problems were simple and people were amazed with what could be done, and kept asking for more. With more functionality, came demands for more programmers. Not many people could deal with programming in binary. Because of this, we added levels of languages, which in turn added complexity and sophistication to programs.

Next we added yet more players: analysts, engineers, customer representatives, lawyers, managers, and the list goes on. Our history tells us there is a tendency in the agile world to evolve to complexity, and some might say towards chaotic systems. There are many environments where agile approaches will work, but the agile teams will constantly have to fight to keep things "simple." And, as we all know, there will be cases where some may fail to keep things simple and follow the agile principles.

But wait, we still produced products. What am I to do, crawl back into the caves and give up?

Life is better in many places because of software. This last statement is subject to many debates of course, but I believe it to be true, or I would not be working in the software industry. The modern world runs on software and the information it processes, even with the multihead quality monster, and the well-documented deficiencies of our industry (e.g., NIST studies). Most people would not give up software and the modern life that it is the language of. So this brings me back to why I test.

I Test to Investigate the Qualities of Software Products
In my view, quality can be both good and bad. A bug is a quality of the software, but not a good one. It is one of my jobs to help the team expose the errors. It is everyone's job to work software quality. I have heard of many teams where, once a bug escapes to the field, management blames software quality and testing. This is only a scapegoat. The reality is the whole team produced and then missed the bug.

I Use Test Techniques, Methods, Tools, and People, to Investigate Quality
I have worked in places where the whole team defines tests, requirements, designs, code, and other products that we place importance on, or we don't create the products. I was surprised years ago when I found projects that did not subscribe to agile ideas, and I was amazed when people felt they had to call these ideals something special like agile. Many Agile concepts are good and deserve consideration.  

• I use formal processes when needed to keep myself and the team out of trouble (thanks to Watts Humphrey and Rex Black), because processes help "store" lessons learned stories and project collective memory.

My discipline and practice as a very low-level test master, helps me know which things to try, and if one fails, I try another. As a test manager, my job is to know all these concepts exist, and to try until I find something that works for my team at a given point in time. I am here to help communication, aid understanding, and not make judgments, except when my job requires a decision with the group. I help people in the team. And because of the evil multiheaded beast of complexity, I don't expect the other team members to be masters of testing. I expect them to be masters of what they do, which will help me, since I am not an omni-capable master. We work on priorities. We work on risks. We focus on problems. We focus on code. We all test. We all code. We all try to understand the problem. No, wait you say. This sounds very agile. Yes, but agile with documentation, plans, processes, and procedures. Oh, so you are really traditional? Yes, but why do I need to be one or the other?

I am depending both on the time and my problem. Can I not use parts of both? People that are hard over on one side of the agile vs. heavyweight-process debate will have problems with me. Good. The beast of quality has no single head, and those people that sell concepts of a single head will find that it only works sometimes. Complexity requires multiple approaches. I can't be static and unchanging.

When managers ask what I do and why testing is necessary, I must help them to understand the complexity and communication problems in software. Managers, as most humans, tend to oversimplify problems. Humans test things all the time. Nobody would buy anything expensive or complex without some kind of evaluation. If managers can't get this "fact of nature," their projects are probably doomed. Marketplace Darwinism will require correct levels of quality, not perfection, but good enough quality where people will continue to buy the software. The team must balance this and the complexity. I bring my mastery of testing to the team. I work with all of the team to improve my mastery. I am never good enough to stop. People that say otherwise have missed the boat, and the complexity beast will get them.

Next time somebody asks why and how you test, consider my list, but wait. This list is not complete and probably not right for you. You should always ask yourself "Why." The answer may change over time. Next time you want to justify going to a conference or training, start with this list, and add your own items. Learn why you test to improve you test skills.