Behavior-Based Test Automation and the Future of Software Engineering

Current test automation works on the basis of creating a “shadow machine,” which captures a model of what the software should do in the form of object repositories and script logic. While these mirror systems are fast to execute, they are slow to create and slow to change. In fact, having to maintain two systems—the test harness and your tested system—in tandem generally doubles your work.

Management doesn’t want to pay to build it once; customers don’t want to pay to build it twice. This is a problem.

Tooling is touted as the answer to accelerating the speed of change in IT systems, yet shadow machines are often too detailed and heavy to keep up the pace. As a result, they are abandoned when the maintenance falls behind.

The question we should be asking is: Is this design principle the only way, or are there alternatives for leaner, lighter automation?

Robots Inspired by the World around Them

The way we build machines often mimics the human user: Cameras see like the eye. Microphones hear like the ear. And computers are built to play Go like Korean champions. Similarly, the way test automation tools are designed—with the concept that a full logical model is necessary to guide action—mirrors an older view of how the human mind works.

This “symbolic” or “classical” approach has fed the field of computing for decades. Now, however, this concept is being challenged, not just by cognition theorists, but also by the applied sciences it has nurtured for so long: robotics and AI.

As an example, take a look at the beautiful Strandbeest robots walking on Dutch beaches. These wind-propelled, lightweight machines navigate complex environments without any symbolic memory or processing. They simply react to what they sense.

These creations are radical realizations of an approach known as behavior-based robotics (BBR). First developed by Australian roboticist Rodney Brooks, BBR bots are engineered to be mechanically reactive to sensory input and gradually adapt their behavior. As a result, behavior-based robots look and act more like biological organisms than their symbol-computing counterparts. They are also relatively low-cost and low-maintenance.

How can test automation take a cue from BBR robots and self-navigate application landscapes without understanding them first? Let’s look at the design concepts behind BBR.

Designing Behavior-Based Bots

Subsumption architecture attempts to do away with the necessity of a machine fully understanding its environment before it can interact with it. Such robots are designed so that the control loop between sensory input and reaction is as short as possible, and they are tested and honed against the real world.

The solution can be purely mechanical, as in the Strandbeests using pressure differentials in plastic bottles to walk backward when encountering water. But it generally uses stacked behaviors, where higher ones make use of abilities provided by lower ones. A planet rover, for example, would have “maintain balance” as the most basic instruction, before considering the higher “wander around,” “avoid collision,” and “gather objects”. If enough of these behaviors are stacked, then complex behavior—at least to the eye of the beholder—simply emerges.

If we transfer these concepts to software test automation, we sense that we will have to abandon the notion that our “robot,” the automated test harness, will need to understand its environment, the application under test, before it can produce a useful result. Instead, it will have to be autonomous, exploratory, and immediately reactive to what it encounters.

As an example, let’s take the task of testing online forms (as suggested by Gojko Adzic). Analogous to our roving robots, the test agent’s simplest behavior could be the ability to “wander around” on the form. While doing so, it would identify any encountered controls, such as entry fields, selection fields, and button actions needed to continue. Stacked on these behaviors, it would then attempt to input data, randomly and ignorant of context or content. Such data could be sourced from both a “black list” intended to break an app (e.g., unescaped code) and a “white list” intended to pass (e.g., correctly formatted dates). With a few additional behaviors, like the ability to remember working data and deal with immediate reactions such as client-side validation, we could release our bot onto the app and see what happens.

At this point it is easy to imagine that such a test machine could be trained and improved to be able to crawl an entire application and produce a report of both blocked and accomplished paths. We can also imagine that similar bots could explore and test comparable standardized interaction topographies in UIs, APIs, web services, database interfaces, and rule or gaming engines. And all of this could be accomplished with the advantage of being largely impervious to environmental change, while being low-cost and lightweight.

Of course, there is a cost to the lack of understanding. These autonomous test agents would not be able to render judgment on whether their results are desired or not—in other words, they could not determine pass or fail. We would still need a human to separate the signal from the noise.

But what if this behavior, too, could be replaced?

Independent Learning Machines for Test Automation

Imagine now that we take advantage of cloud computing and virtual environments to run a swarm of dumb but connected test agents in parallel. The noise-to-signal ratio of such a rig would probably be rather high. To deal with this, we need to bring the power of the model back, just this time not as a static, maintenance-heavy, symbolic rendition of the app under test, but as a dynamic learning machine.

Modern learning machines are able to examine patterns and interactions and come to predict them via supervised, unsupervised, or reinforcement learning. Such a learning machine could be trained by observing a human judging pass and fail over the bot-produced reports; over time, it would anticipate the correct test verdicts with growing confidence. With every further iteration it would then grow more efficient in separating meaningful results from random testing noise, until it reaches complete autonomy.

Of course, there would be downsides. For one, the design of the BBR agents will require higher-level design skills than test automation currently does (although it already is, in practice, far removed from the “record and play” image promoted by tool vendors). And second, as friend and robotics engineer Guy Immega commented to me, autonomous learning systems are still not understood enough to be mass-produced, making any such technology experimental (for now).

However, the design of test automation tools may already have reached a boundary where it can no longer cost-efficiently keep pace with change in enterprise IT systems, and where alternative solutions can no longer be ignored.

So, the next time you fix your test scripts because the names of GUI objects have changed, imagine schools of simply engineered Strandbeests roaming the wilds of your technology stacks, doing the heavy lifting of software testing with little or no maintenance, and with intelligent learning machines evaluating their outputs.

What is the smallest possible step we could take to move in this direction? Take it, and let us know what you come up with.