It's frightening how many companies are on their second, third, or even greater attempt to automate their testing—each time junking months or years of effort and work product. Here, test automation advisor Linda Hayes shows the way to avoid having to bury your automation project.
I know Halloween has passed, but I have noticed something spooky lately: lots of dead code. It's amazing how many companies are on their second, third, or even greater attempt to automate their testing—and each time at a cost of hundreds of thousands of dollars and many man-years of work. I know of one company that is on its third major rewrite, having just junked two years of effort by thirty people—for the second time. The horror.
It's not just the cost of a failed automation effort, it's the loss of the productivity you were after in the first place. The greatest benefit of test automation—cumulative coverage—is lost if you keep starting over.
Why does this happen? What can you do about it?
Slippery Slope
Because test automation tools are proprietary languages requiring special skills, many companies either recruit contractors to get them started or hire aspiring developers and train them. The team then develops, to the best of their ability and understanding at the time, an automated test library for the application under test. At the end of the contract, or the next time budgets require across-the-board cuts, or when the new hires want a career change, the original developers leave.
Left behind are tens, if not hundreds, of thousands or millions of lines of code, probably undocumented and certainly unique. An expensive work of art.
As soon as something changes and a different team has to work with this code, they will quickly determine that it is incomprehensible and will be easier to maintain if it is rewritten—maybe in a different language or tool. In other words, it would take longer to understand it than to just start over. This is the dirty secret of developers everywhere: maintaining someone else's code is a form of torture to be avoided at all costs.
And so, we start over and create yet another work of art. And on it goes. How do you break this cycle?
Accepting Reality
Start by accepting this reality: test automation is a development project. It is no different than the automation of any other task. Forget any nonsense about capture/replay and face the truth—we're talking about software here, and all that it implies. Either treat it like a development project or forget it.
Next, inventory what you need and make a build-or-buy decision for each component. Do you really need to develop a defect tracking system, or a report writer, or a test management repository, or can you buy one faster and cheaper? How much of what you need actually needs to be unique? Realize that there are both the cost of acquisition and the cost of ownership to be considered. If you build it, you have to support and maintain it.
Then, for the components you decide you must build, don't let anyone lay down a single line of code until you know what is going to be built and how long it will take. Design on the fly is costly and error prone. Whether you are interviewing contractors or employees, ask to see their proposed architecture and talk to their references. Insist on an early prototype, and short interval delivery schedules. Demand change control, code management, naming conventions—all the things we have learned you need to turn a bunch of code into an asset instead of an expense.Face the Future
The other reality you have to accept is that your automation project will never be finished. The test library should have the same useful life as the application under test, and undergo as great—if not greater—rates of change, which implies enhancement, maintenance, and support.
What this means is that you have to plan for the transfer of knowledge in order to maintain continuity over the coming years. If you use contractors, pair them with employees. Never have a single point of failure. Always back your key resources up with at least one understudy. Insist on adherence to processes and demand documentation. Treat your test library as a valuable investment that is worth protecting and preserving.
What this doesn't mean is that you should not demand and get results along the way. But measure progress by automated test cases executed, not by lines of code. I recently spoke to a test manager who realized that it was taking three days to update the script library for each new version of the application, yet the number of test cases that were automated could be executed manually in only two days. By measuring lines of code it looked like work was getting done, but a look at executed cases showed that work was being wasted.
Keep Your Vision
Always remember that automation is only a means to the real end. The goal is not to write the coolest test automation framework ever, or to experiment with new tools and techniques, or to endlessly tweak and refine scripts. The goal is to increase test coverage through improved productivity, thus reducing the risk and cost of failure.
Keep that vision, and maybe you can exorcise the ghosts of scripts past.
