From One Expert to Another: Dale Emery

Payson Hall: Dale, you recently gave a talk about the "Test Automation Zombie Apocalypse." What was the origin of that talk?

Dale Emery: Lee Copeland invited me to fill in for a conference speaker who had dropped out, for a talk about test automation. We created the title first, and it really seemed to resonate with something I had been mulling over in my head: the test automation tradeoffs that people make and the natural human tendency to make tradeoffs that favor short-term results.

Like regular zombies, test automation zombies are relentless and infections, and they eat your brains. Unlike regular zombies, test automation zombies don’t look like people. They look like ideas. Every one of these zombie ideas comes from a good intention, and every one is sometimes the right thing to do.

Payson Hall: What are some examples of test automation "zombies"?

Dale Emery: Some movie zombie stereotypes seem popular and recur often—for example, zombie woman in a business suit at the shopping mall, zombie Jethro from the gas station in overalls, and zombie who waits in the dark for unsuspecting young people. Like cinematic zombies, there are some cliché test automation zombies I have observed in different client contexts. Let me describe a few of these.

The one that scares me most is the record and playback zombie. This zombie is the idea that all you have to do is run through your manual test suite, which you were going to do anyway, and you end up with automated tests. The appeal here is obvious: You get a huge suite of automated tests almost for free, and then you can run them as often as you want.

The first problem appears almost immediately: The tests are brittle. The slightest change in system implementation makes recorded tests fail for the wrong reasons. Then you notice the second problem: Recorded tests are utterly unmaintainable.

And then the worst problem of all appears: People attribute the maintenance nightmare not to the zombie idea of record and playback, but to test automation in general. So, they abandon test automation altogether.

Also common is the test automation group zombie. This zombie is the practice of assigning test automation to a dedicated team of test automators. The appeal is that we can keep developers focused on writing new code instead of writing and maintaining automated tests. The danger is that test automation inevitably lags development, so feedback from testing is delayed in a way that significantly reduces its value. A further danger is that we often assign test automation to less-skilled programmers, which results in brittle tests that are costly to maintain.

There’s also the code coverage zombie, which fools us into believing that high code coverage means that our application is well tested. And the fixed wait zombie, which invites us to sprinkle delays through our tests to accommodate variations in response times but makes our tests take longer and longer. And the chewy GUI zombie, which invites us to automate only through the system’s graphical user interface, making automated tests vulnerable to changes in the most volatile part of the system—even when those changes are utterly irrelevant to the tests.

So far, I’ve identified fifteen test automation zombies in the wild, and the catalog is growing. [1] They’re everywhere.

Payson Hall: It sounds insidious that some of these might be reasonable ideas in some contexts. Is that what gives them life?

Dale Emery: Yes. In every case, the desire that gives birth to the zombie is perfectly reasonable. We want to keep costs low. We want confidence in our tests. We want to minimize our automated tests’ susceptibility to normal variability in system response times.

And, in many cases, the zombie idea itself is perfectly reasonable. Sometimes the best we can do to insulate a given step in a given test from variability in a given response time is to add a fixed wait. Sometimes.

So, that’s where zombies come from. There’s another important element: What sustains them? The trouble is when we forget that we’re making a tradeoff. When we add a fixed wait, we gain a little resilience at the expense of test execution delays. When we delegate test automation to dedicated test automators, we gain a little bit of developer focus at the expense of delays in feedback. Each zombie idea involves a tradeoff. If you know you’re making the tradeoff, and you make the tradeoff mindfully every time, you can avoid infection and protect your brain.

What makes these zombie ideas dangerous is when we apply them by habit or, worse, by institutionalizing them. We stop making the tradeoffs mindfully. Instead, we automatically reach for the short-term benefits and forget to weigh those against the dangers. We automate ourselves. We eat our own brains. We become the zombies.

Payson Hall: If a practice that arguably works in some contexts begins to become a zombie, are there any warning signs?

Dale Emery: For organizations, the biggest warning sign is when a zombie idea becomes institutionalized, so that you can see it either in the org chart or in the organization’s defined practices. If you see a test automator job title or a test automation group in an org chart, you know you’ve institutionalized the idea that test automation is something separate from development. If you create a role to sift through test results, you’ve institutionalized a barrier between test results and developers.

For managers, a big warning sign is that you have attached numerical goals to test automation. This means that you’re focusing on things that are easily countable, such as code coverage or the number of tests, rather than the quality of the tests.

A big warning sign for test automators is habitual copy and paste. And I mean not only in the code, but in the way you solve problems. If every time a test fails due to a long response time you automatically reach for a fixed wait, then you’ve copied and pasted a solution to a problem and you’ve stopped thinking about where the problem is coming from and whether there might be a less costly way to solve it.

Each warning sign is an example of a more general idea: You’ve stopped thinking about the tradeoffs you’re making.

Payson Hall: If an organization has embraced some of these zombies, how might a practitioner gently encourage rethinking the approach without jeopardizing his or her career? Do you have any suggested “zombie killing” methods or magic?

Dale Emery: The most important thing is to make explicit your reasons for automating tests. Then, you can assess how well your test automation policies and practices satisfy those reasons.

For an automated test to be worthwhile at all, it must first be a worthwhile test. It must inform people, must help people make better decisions. Automate a test only if you will value the information it provides, only if the test helps someone make some decision that matters.

And an automated test must satisfy at least one good reason for automating. You might automate a test to make it more repeatable, to make it more available to more people to run at any time, or to run it automatically whenever a developer commits a change to the code so that you can get feedback about the change at the earliest possible moment.

Once you are clear about your reasons for automating tests, you can evaluate each test automation practice against these criteria. Does it speed up feedback or delay it? Does it make test results more available to more people or less available? Does it increase or decrease the quality of the information?

Also important is to notice whenever you are about to institutionalize anything about test automation, whether that’s creating a test automation group, creating a test automation role, standardizing some test automation process, or declaring a numeric goal for test automation. You are institutionalizing a tradeoff. You are piling the weight of the organization onto the tradeoff, making it very difficult to reassess as you learn more and as conditions change.

So, before you institutionalize, ask: What are we trading off here? What do we gain? What consequences are we deliberately accepting?

Above all, continue to assess whether your practices are giving you test results that are relevant, timely, and accurate.

Payson Hall: What do you think of StickyMind's innovation of trying to repurpose its authors as journalists? Do you suppose this might become a "m agazine time-saving zombie"?

Dale Emery: It’s a possibility.

On one hand, I can’t think of anyone better than you to interview me. You know me well. You’re passionate and thoughtful about software development. You ask great questions. Brilliant choice.

On the other hand, if the authors-as-journalists idea becomes institutionalized, if it becomes an unthinking habit, that way zombification lies.