Fast, Automatic Builds: the Agile Heartbeat

So there's no going back to small builds on a single machine for build managers. In fact, the build manager is expected to cope with ever larger software, on ever more platforms, while at the same time dealing with developers' requests for fast integration builds. Integration builds are invaluable because they enable agile development at the software engineer's desktop; however, they force build managers to produce one or two orders of magnitude more builds per day.

Continuous Integration Redefines Builds
The seminal paper on Continuous Integration (and an excellent, and readable, introduction to the topic) is Martin Fowler's article entitled, simply, Continuous Integration. [i]. It's abstract states:
quot;Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.quot;
To the build manager, the {sidebar id=1} words quot;multiple integrations per dayquot; and quot;each integration is verified by an automated buildquot; are a worrying and radical change from once-nightly builds!
Fowler goes on to list a number of quot;Practices of Continuous Integration.quot;
The first of Fowler's build-related best practices is quot;Automate the Build.quot; He argues that the set of tasks required to go from source code to the final shipping binary is so complex that it should be automated and repeatable. If the build is not automated the build process will be error prone. He also points out that the build includes much more than just compiling the software and building a binary. And he sets the goal of quot;anyone should be able to bring a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.quot;
It's worth stopping and thinking about the implications of that statement for your build organization. It's very likely that getting to such an automated stage seems impossible, when starting from what is often a creaky build system held together by a collection of Makefiles, Perl scripts and other programs whose exact function or operation is often unclear. Clearly, the goal of going from a virgin machine to fully running code is a stretch, but I think it's a good overall goal. Once you reach that stage, you will have put together a build system that is very reliable, and should, if written and documented well, be easy to modify and adapt as software changes.
Such a fully-automated build has advantages outside the direct realm of agile development. How many times has your build team been asked to rebuild an old version of your software and been unable to do so? Important customers can sometimes demand that old versions of code be patched or updated, or a security problem can dictate that all versions of a company's currently supported code need to be fixed and released.
In general, only very well-prepared teams are capable of building an arbitrary version of their code. Even if a good archive of the sources was made at the time of release, other components of software are likely to have decomposed: you may not have the right build scripts, or the right compiler, or the right version of some third-party component any longer. Fowler's goal, which I refer to as a quot;pickled buildquot; (the entire build is pickled in a jar ready for use whenever demanded), means that old versions of software can be rebuilt at will, which helps support developers in their move to agile development.
Fowler's second best practice for builds is: quot;Make Your Build Self-Testing.quot; This implies that the automatic build also includes an automatic test. He asks that all code have a set of automatic tests that can check a large part of the code base for bugs. Many developers are, in fact, already writing such test suites through the Extreme Programming focus on unit testing, or by performing test-driven development (where developers write the tests before the code).
With this test suite in place, Fowler asks that the automated build test itself by running the test suite and reporting the results. Results would typically be reported on screen or by email, if the build and test was run remotely.
Fowler's next two practices have a profound affect on build management: quot;Everyone Commits Every Dayquot; and quot;Every Commit Should Build the Mainline on an Integration Machine.quot; The implication is that every developer will be committing code once per day (at least) and that every commit will cause a build and test on some build resource (the quot;integration machinequot;) managed by the build manager. To put that in perspective, for a team of 20 engineers committing once per day during an eight hour work day, that's a build and test every 24 minutes! For large teams that number increases, and the time between builds is greatly shortened.
The reason agile developers want these per-commit builds is to ensure that integration between developers is faultless and that the software being built works and is testable at all times. The idea that the software should run at all times is central to agile development, and the health of the per-commit build and test becomes an important sign of health for the entire project. If per-commit builds are done quickly and frequently enough, the build will reflect changes made by a single check-in by a single developer, offering an unparalleled opportunity to narrow down the cause of a breakage.
In fact, this build is so important that it is essentially the Agile Heartbeat. Agile teams can install monitoring devices (such as red and green lava lamps) to make the status of the heartbeat build visible to all. Fowler states that no developer should quot;go home until the mainline build has passed with any commitsquot; they've added. This means that all of engineering looks every day to the heartbeat to measure their own progress.
Another important practice mentioned in Fowler's paper is quot;Keep the Build Fast.quot; This seems like an obvious prerequisite to the previous practices, since a team that requires a build every few minutes and every time the code changes will necessarily need very fast builds. In fact, Extreme Programming outlines the goal of ten-minute builds, and I've dubbed them quot;espresso buildsquot; -- builds that only take as long as a coffee break.
Fast builds are also important because developers are looking to the status of the build as a measure of their progress. Every minute that is shaved off the build is a minute saved for all developers who have committed code to that build. With many developers, and many builds, that time saved adds up quickly and fast builds improve the productivity of the entire team.
Conclusion
Agile development puts fast, automatic builds center stage and requires a new approach to software production tasks. Teams can realize some of the benefits of agile development with a strategic approach to continuous integration. Careful planning is required by all parts of the engineering team with a special focus on the build resources. The build team will come under enormous pressure once agile methods are implemented and a step-by-step approach to implementing agile builds is critical.

[i] See http://www.martinfowler.com/articles/continuousIntegration.html for Martin Fowler's paper.