Continuous integration is an automatic re-building of project code base that is triggered by new changes to ensure that new changes integrate well into the existing code base. Continuous integration gains popularity among software organizations for its ability to provide quick feedback on quality of new changes. This feedback is used to deliver timely fixes if the changes don’t integrate well and break the project code base.
While failures of continuous integration builds are generally inevitable, there are a number of causes of repeatable build failures, or patterns. It is important to understand that once recognized, software build breakage patterns can be avoided. Reduced build breakage means decreased risk of the project falling behind schedule and more enjoying software development.
"Five O'clock Check-In" PatternOur analysis shows that vast majority of the build failures falls on the end of the day, with factual build breakage time ranging from 5PM to 8PM, depending on the duration of the build.
We call this pattern "Five O'clock Check-In." As the name implies, builds get broken because of check-ins made after five o'clock. Below you can see a histogram that is provided by one of our customers that shows distribution of build failures by hour (this and other statistics is available to Parabuild users). While numbers vary from company to company, this histogram is very characteristic. You can see a clear spike in build breakage at 6PM.
"Five O'clock Check-In" pattern has even more dramatic effect on software organizations distributed in multiple time zones. The following histogram shows distribution of build failures in a software company that has two R&D facilities, at PST and at GMT+13:
There is a known tendency to check in changes in the end of the day. At that time engineer's ability to critically assert actions is at its lowest level. This, combined with readiness to go home, produces the main excuse not to run clean build and tests before checking in changes (see "It Is a Small Change" pattern below).
This pattern can be (and is) avoided by establishing a simple, easy to follow policy: No check-ins should be made after 5PM (or whatever your de-facto end of the day is). This alone may reduce build breakage, typically by 20% to 50%.
"Spoiled Fruit" PatternModern software projects increasingly use code generation. Often to simplify build scripts code is generated directly into code base directory structures. This approach may save few lines in a build script, but it causes build failures when generated code becomes orphaned and goes out of sync with the current state of the code base. That is why we call this pattern "Spoiled Fruit." After such breakage occurred, the only way to get a clean build is doing a clean check out. With code bases growing in size, a clean check out may take tens of minutes, and even hours, significantly increasing feedback time from a continuous integration server.
"Spoiled Fruit" pattern can be avoided by generating code and storing other build-time artifacts under a single temporarily build directory. The build script should automatically empty this directory at the beginning of a build cycle, usually as a part of "clean" target. Below is an example of a pseudo-code that uses this approach:
A continuous integration server calls "all.clean" target. This allows code base to build with no artifacts from previous build runs affecting build results.
"It Is a Small Change" PatternRepeatedly the size of change serves as an excuse not to run build and tests cleanly before checkin in new changes. "It is just a comment," "It won’t break anything," "I only removed an empty line." Yet, even a small change moves code base into a new, often broken, state. We call this build breakage pattern "It Is a Small Change."
To avoid build failures caused by "It Is a Small Change" pattern, it should be made mandatory that build and tests are run cleanly before checking in any changes into a version control system, to ensure that the changes, to one’s best knowledge, don’t break the build.
Conclusion
It is possible to reduce software build failures by avoiding common build breakage patterns thus decreasing risk of the project failures and making software development more efficient and enjoying.
Slava Imeshev is president of Viewtier Systems, Inc. http://www.viewtier.com, makers of software build management server Parabuild. Slava has been developing software for 16 years. You can reach Slava at [email protected]
