The Agile Advantage for Last-Minute Changes

One of the principles behind the Agile Manifesto states, "Welcome changing requirements, even late in development."

This is not business as usual. One of the main reasons I’ve found that companies stay with heavyweight development processes such as waterfall or CMMI is to manage change. This often involves limiting or locking down scope because, as we all know, adding new tasks to a tight project can have huge consequences and be highly disruptive. As a result, traditional methodologies try to prevent scope creep.

“Yeah, we’ll get to that in the next version," is often the response I hear on waterfall projects when new features are requested. But doing this can often have negative consequences for our products and our customers.

Agile takes a different approach. We recognize the value of last-minute changes and embrace change. We do that by making it inexpensive and straightforward to make changes to software, even late in the development cycle, using continuous integration.

From the customer's perspective, the ideas they have toward the end of development can often be their best and most valuable ideas, so if there are ways we can incorporate them into our products, then it's a win for everyone.

The traditional way we conceive of and execute projects requires us to have a plan up front. Agile makes the radical suggestion that we can have a plan and we can also be flexible, be agile, and find ways of adapting the plan to current needs. The agile journey that most companies are taking nowadays is a great discovery process of what's most valuable to the organization and how they can leverage that in the current fluctuating marketplace.

Software is soft. It doesn't have many of the physical limitations that other tangible products do. This means that we can build it differently than we build physical things, and this is the secret behind agile. It consists of having the flexibility to be able to handle last-minute changes.

To accommodate last-minute changes, we have to reconceive of the whole development and delivery pipeline by integrating changes we make continuously. I used to think it was more efficient to design an entire system, build it, and then test it—the traditional waterfall approach. I reasoned that it made sense to batch up similar activities when constructing physical things like buildings and bridges, so the same should apply to software. But constructing buildings and constructing software are very different activities, and taking this approach when constructing software turns out to be phenomenally inefficient.

Rather than batching up similar activities so that we design all the features up front before we start coding, we do almost the exact opposite by focusing on a single feature at a time and taking it from design to code to test so that it is fully implemented in the system before we move on and implement the next feature. The best way I found to do this is through a good continuous integration system that relies on well-written regression tests to tell me that everything is working as expected.

This approach has several advantages over waterfall that may not be obvious at first.

In waterfall, every feature in a version is considered work in progress because no feature is complete until it’s integrated and tested right before the new version is released.

Using continuous integration, we build a system feature by feature so that each feature is fully designed, built, tested, and integrated into the system before we move on and build the next one. When we take one feature from start to finish, it means that we radically cut the amount of work in progress and reduce the overhead of task-switching because we have just one set of things we’re working on at a time.

In waterfall, we don’t see our program run until we integrate all the components late in the development cycle. Before that we may be able to get bits and pieces of it running in the debugger so that we can manually test it, but inevitably we discover that when we bring all those pieces together, they don't fit exactly as we thought. So there's this huge integration phase, and that is often where the nastiest defects show up.

With continuous integration, by building and integrating one feature at a time, we always have an executable version of our system, which means that we can run all of our tools, like compilers, static checkers, and automated tests, as we’re building features. Our build system becomes a source of rich feedback always ready to validate that every change we make to the system doesn’t adversely affect it.

In waterfall, we discovered in the ninth month of a twelve-month project that we were six months behind. Well, actually, we were always six months behind—we just didn’t discover it until the ninth month.

With continuous integration, we see the truth of the system more clearly. Because we integrate continuously as every bit of functionality is being created, we always have a buildable version of the system and every feature in the system is proven to work with every other feature, so there are no surprises right before a release. We get a far more accurate representation of the health and progress of our system throughout development because we can see features integrated and working as we build them.

In waterfall, last-minute changes are often expensive or impossible to make, even though we acknowledged that many of those changes would be highly valuable to our users. Nonetheless, they have to wait until the next release before getting those changes.

With continuous integration and a suite of good automated regression tests, we can make last-minute changes to a system with confidence because our tests can instantly prove that our system runs as expected. This drops the cost and risk of making last-minute changes.

In my experience, this is really the only way to reasonably manage large, complex computer systems. It certainly is less stressful than waterfall.