Don't Ignore the Smoke Tests

A few days back, a friend and I were chatting on Skype about test automation. I have a really hard time getting excited about automating a graphical interface, but one aspect of automating the browser I am excited about is the ‘really small smoke test’. These tests can be simple and concise, and because of that, low cost. Spending a few minutes of your time watching the smoke test run can be a fantastic way to defocus and notice some things about your product that may normally go unnoticed. Let's explore that idea a little bit.

Try It and See
A smoke test is a short test or set of tests that will run quickly and tell you something about basic functionality in your software. Generally these tests don’t do a whole lot, and that’s just fine.

Selenium IDE is a FireFox plugin that will allow you to record and rerun actions in the web browser. If you are not a programmer, the IDE can be a great tool to create smoke tests. Take a look at this tutorial for selenium IDE to create a smoke test in your browser, but try to do it for your software, not Amazon.

If you are more familiar with programming, you should be able to write something with the Selenium Webdriver framework relatively quickly.

Go ahead and run the test, click the go-button, and then walk away and get some coffee. When you get back, what do you have? A log file and probably nothing else, that’s what. If everything passed, you know very little about what the customer really saw during the test run. If something went wrong, then you have to go back, look at the logs, maybe read some of the test code, and then go to the software to figure out what happened. Sounds time consuming, right? It is.

Now, let’s try that one more time, but this time don’t leave! Stay in front of the monitor and watch what happens as your test moves along. This time when the script is done running, what do you have? You still have your log files, but you also have your observations and all it cost you was a couple of minutes. If the test completed successfully, you know why this time. If the test didn’t complete successfully, you were there to see what happened. Or maybe the test completed without error but you were able to see something else interesting while it was running that the smoke test wasn’t created to catch.

By staying and observing you have expanded this mimeomorphic (done the same way every time) script into a polymorphic-thought exercise that occurred in your brilliant tester brain. You can observe and learn so much more than a simple script ever could.

Focus and Defocus
These tests are so simple that you could potentially execute them yourself without any extra tools. In the world of software testing, we sometimes use focusing and defocusing heuristics to change our perspective. Jesse Alford spoke about these in a lightning talk at the 2013 Conference for the Association for Software Testers in which he compares focusing and defocusing heuristics to photography. Executing a test like this would be a way to focus in on specific detail of your software.

Creating a script can be a way for you to defocus your smoke test. Rather than concentrating on specific details of your test, you have the opportunity to broaden your perception and observe the software in a general way. In the words of Yogi Berra, you can observe a lot just by watching. While the test is running, you can be scanning the software, looking for interesting things that may normally be hidden under layers of focus.

You Can't Go Home Again
I have previously written in my personal blog about a test heuristic I call "You can't go home again." The essence of this heuristic is that running a single test many times is the same as running many slightly different tests. I think this is the case for two reasons: people have extreme difficulty doing a task exactly the same way every single time and there are innumerable variables for a test and knowing how they will affect the outcome is difficult at best. Using the smoke test technique I've described can encourage usage of this heuristic and may give you a more holistic product view. Two ways you could try this out at work tomorrow are watching the smoke test run many times yourself and purposefully defocusing each time, or, by having a few people watch the test run and sharing notes after about what they observed and learned.

The trick goes like this:

1 - Create the script

2 - Run the script

3 - Watch the script run

4 - Learn

5 - Share what you learned

Figure 1. The Share of Actions: What Humans and Machines Can Do by Harry Collins and Martin Kusch

How We Behave
Humans and computers do not behave or act the in the same way no matter how much some people wish they did. It is in our nature to change our actions to better fit the social and cultural situation. Computers and machines are completely unable to do this. They do exactly what we tell them all day, every day till we tell them to stop. They have no social understanding.

Harry Collins and Martin Kusch wrote a book together called The Shape of Actions in which they talk about the kinds of behaviors performed by humans and machines. The book calls these different actions polymorphic or mimeomorphic.

Polymorphic actions are social in nature; the action that occurs varies depending on the social context. Greetings are a good example of this. The way you greet another person can vary wildly based on social circumstance. Think about how you might say “hi” to a friend at school or a potential employer at an interview. Greetings are a pretty simple example, but could you imagine not changing behavior based on social experience?

How Machines Behave
Mimeomorphic actions are actions that are performed independent of  social context, these actions can be performed in an identical way each time. The way a paint spraying machine paints chairs is a mimeomorphic action. The machine may be able to recognize that there are different chairs, but it still paints the chairs the same way every time.

Figure 2. Automotive assembly line: An example of mimeomorphic action being performed on a large scale.

When you kick off an automated test, the series of steps executed are mimeomorphic actions. You defined a sequence and told the computer to perform those actions using some language and guess what, the computer is going to perform those exact actions the same way—every time. Your test suite will never notice things, it will never scratch it’s chin and say, ““Huh, that’s interesting,” and change its behavior to investigate. You do this though; it comes naturally, and for a good tester the urge is near impossible to resist.

Right Here, Right Now
Try this smoke testing technique out for a little bit and see how it works for you. Reframe how you think about smoke tests and what value they offer. Tell us what you find!