Taking the Complexity out of Release Management

In this article we will focus on aspects of a release management strategy which reduce complexity, minimize branching and ensure accurate identification and descriptions of your releases. We’ll explore the need to:

·         Establish a product road map and use it to minimize branching

·         Manage the superset in your product component tree, but build and deliver subsets

·         Make baselines meaningful milestones and use build increments between baselines

·         Establish clear consistent release, baseline and build identification

·         Select CM tools that make it easy to compare, report on and browse differences between builds and baselines

Establish a Product Road Map

It's too easy to use ad hoc branch labeling or directory copying to define and freeze releases. As support issues arise and as you realize that your verification failed to uncover some significant bugs, the multiplication of branches and re-releases sets in. What you really want is an overall game plan. Your product needs a release strategy that is driven by market and support requirements, not by your tool, team or process capabilities; the latter should fall out of your requirements. Your design architecture will play a significant role in how quickly you can react to these requirements, but you need a product road map up front, one that is simple, but flexible.

Many (most?) shops define a main branch and develop out of the branch, merging back in, and perhaps branching again to define a release or to support a release. It seems like a simple strategy because there's a single main branch. If release dates and criteria could be cast in stone and customer variant requirements minimized, this might be a successful model. In the real world, the release process is a long parallel-stream process. The single main branch model is very complex precisely because it does not map on to the real world requirements where development teams need to be working on a new releases even before the previous release has made it out of QA.

Let's start with a different framework. Look at the product road map and jump 5 years ahead if you need to. Release 3 is in the field. Release 4 is in verification, almost ready for beta, and work on release 5 has started: three parallel release streams, all needing support, perhaps more in the future. The support team may be trying to move users on to the latest release, but it knows that it will have to maintain parallel streams. It doesn't want the support process for one release stream to change just because a new release stream has started.

Rather than using a main-branch centric process, try the following. Look at the product road map. What does it say about how you need to roll out and support releases of your product. The roads can actually run in parallel. They don't have to run end to end (i.e., main branch) with a sudden left turn every time you need to define a support point.
Minimize Branching

Start with your current release branch, say the R3 stream. When it's time to start working on R4, start working on it, but don't branch the world, simply create an R4 branch (if none yet exists) for each component/file when the first non-R3 changes for the component/file go in. Continue to make changes to R3 in the R3 stream. The R3 stream should live forever on its own. It doesn't have to merge back into some artificial main trunk. The process for supporting it doesn't need to change when R4 comes out, as it just slows down as things stabilize.

Changes to R3 required for R4 can be merged into R4. Even better, choose tools that permit the R3 changes to be automatically picked up by R4 until you explicitly branch the associated file(s) into R4, in which case a merge may be required. Most work is going on in R4 and perhaps work on some major areas of R5 are starting. Your tools should be able to tell you when a change to R3 or R4 may need to be applied to other release streams.

So what we have is a 2-dimensional (2D) release branching strategy. Instead of the traditional main branch, there are release streams. There's really no need to hold up R4 while R3 stabilizes. You may wish to focus initial R4 work in more stable areas to reduce the need to merge R3 changes into R4. But the 2D model allows you to have true parallel development, and with the right tools you’ll have minimal branching. If your tool, team or process forces you to branch or re-label all of the components in order to start R4 development, you need to make some changes.

Are you wondering if your project might be too complex for a 2D model? Well I created my first real CM tool for a 25-million line telecom product in the ‘70s and ‘80s, with various product variants and thousands of developers. Not only was the 2D model sufficient, it was necessary!

As a side note, if you really want to minimize branching and merging, make sure your CM tools let you check in your changes before the build manager is ready for them. If developers have to artificially hold back on submitting finished changes, for whatever reason, the requirement for parallel checkouts (and in most tools additional parallel branching and merging) will grow dramatically. A good CM tool will eliminate the need for most parallel checkouts, including branching, labeling and merging, and will permit your components/files to follow your product road map. Investigate the change promotion model carefully to ensure that software is checked in one or two stages before it may promoted into a build.

Manage the Superset: Build and Deliver Subsets
With a 2D release framework which is easily mapped to the product road map, consider variants, customizations, etc. Do not get into the business of managing multiple baselines for these. If you do, you'll multiply your work and decrease quality. Be assured that when you finally release R4, it will not be finally. Design and CM need to work together here. Start with a strategy that says you're going to manage the Superset of your product component tree, but build and deliver subsets. Your tree is going to branch for each release stream, and it should contain all of the files/components for that stream (even though they don’t have to branch yet). If you have an independently developed sub-product, such as EPROMs which encode the latest version of an IEEE standard, then you may want a separate baseline for it (if it's on a different release schedule).

Tag components, subsystems, etc., which are not common to every build/delivery. What you want to achieve is the ability to define your variants as a set of common components plus a set of components containing one or more of the tags required for that variant. For example, your common components might be augmented with English and professional tags for your default build variant. Another variant will have English, German and enterprise tags. Work with your Design team to let them know that this is how you want to do your builds. An even better design would have your variants specified at run-time, minimizing the number of test builds you'll require. It's actually not hard to design to these criteria and it will cut down the number of builds you're doing, along with the associated testing and test resource logistics.

Your source tree browser (in your CM tool) should be able to browse the superset of all components. If it’s a good browser, you'll be able to modify the view just by specifying the variant tags to the browser. So instead of having R4EnglishProf R4EnglishEnterprise R4GermanProf etc. branches and baselines, you'll have a single R4 stream with evolving baselines, and tag sets to identify the variant builds that you need. If your variants are all run-time configurable, you may not even need any tags, but rather a single build per release. This is where working with the design team really pays off: to minimize complexities, but not just for the build team but for the developers who need to do their own builds, as well.

Another side note: having a single build for multiple variants won't necessarily reduce the amount of testing, but it will eliminate the logistics for your test bed (i.e. which build components are loaded, etc.). You simply reconfigure what you've got, hopefully using a variant configuration file which doesn't have to change from release to release.

Make Baselines Meaningful: Define Build Increments in Terms of Changes to the Baseline
The release is delivered from development. You've defined and frozen a baseline. Patches will occur. There may be variant builds. Perhaps customized upgrades are required. There may be a seemingly endless train of builds for a release stream. Rather than defining new baselines every other day, define build increments on top of your baseline. A series of 40 baselines makes it difficult to relate to any of them. If you instead have 5 or 10 baselines in a release stream, with several build increments in between, each baseline becomes a reference point for your whole team.

Build Increments are defined in terms of a baseline plus a set of changes. If you have change-based CM system you'll readily be able to identify the differences between each increment as a set of a few to a few dozen changes. If not, you're likely to see dozens to hundreds of changed files instead, do yourself a favor and move to a change-based solution. Comparing two build increments built on the same baseline is generally an easy task, and a meaningful one, resulting in a list of changes (and hopefully their descriptions).

It’s difficult, at best, to manage releases in terms of changed files. A change-based system will allow your Change Review Board to review and approve requests for a particular product stream and they will be able to clearly identify which of those changes are in each successive build. The promotion model will work on changes, build comparisons will be done in terms of changes, etc.

Establish Clear Consistent Release, Baseline, and Build Identification
Assign regular identifiers (build IDs) to the build increments and use them to track your testing and problem reports. Put the Build Ids in the customer executable (along with the tags used to define the variant subset) so that it is easy to go from a customer site back to the specific build information. This also makes it easy to track your customer sites in the delivery portion of your CM solution.

For release identifiers, I prefer short names, as you're going to want to combine them with other attributes and they'll take up valuable real estate on your reports and browsing tools. For example, R1, R2, R3 is sufficient. Release one may be more readable on its own, but it’s certain to shift other valuable info off the screen or page. Note that the release identifier is actually identifying the release stream. You will likely release more than one build from this stream to your customer base.

Make sure you have a CM tool that can help you with release identification. It should manage release identifiers, baseline identification, build identification (superset and variant), and it should do so in a regular, yet flexible manner.

Use your baseline numbering to identify major feature or quality differences. Tell the customer he has build R4 baseline 6 increment 12 [R4-6.12] and will be moving to R4-8.3. Give a major feature/fix summary of R4-7 and R4-8 to the customer ahead of time and let them know what they're getting. Then fine tune your process so that you move significant changes to the next significant baseline rather than upsetting the user by delivering a major change in the increment from R4-8.3 to R4-8.4.

It is also important to understand the differences between marketing’s product identification and the development and support team’s identification. Likely, the release stream identifiers (e.g., R3, R4 etc.) will relate directly to marketing, especially in an end-to-end tool. Build and baseline identification will likely be invisible, except to the customer who wants to understand what he has.

Select CM Tools That Make it Easy to Compare, Report on and Browse Differences Between Builds and Baselines
When the customer is on the phone, you want to be able to tell him or her when the problem was introduced and when the fix will be delivered. When you deliver the fix, you want to be able to compare the customer's old Build Id with the new one and send the customer a document that details the problems fixed, the features added, perhaps even the data environment files differences.

It doesn't really do a lot of good to say you have build 3421 and the problem fix is in build 3487. The customer will be scared off by the other 65 increments in the build numbering. You need to tell the customer what you’re delivering. This can be time consuming and error prone if your tool doesn't support differencing at the build increment level. We’re not just talking about code differencing. With a good CM tool with integrated customer tracking, you can automatically generate customer-specific reports every time you deliver to the customer, without the need for a full-time customer tracking team.

To Sum Up
So, just to re-cap:

·         Establish a product road map: the 2D parallel release stream map will fit most projects nicely.

·         Minimize your branching requirements so that your file/component branching models the product road map.

·         Manage supersets and build and deliver subsets. Work with the design team to facilitate this.

·         Define key baselines with several build increments (baseline + a set of changes) in between each.

·         Ensure your release identification supports your product road map. Use your tool to identify builds and baselines.

·         Make sure your CM tool will translate two builds into a difference set of features, fixes, changes, etc.

If you do these things, you’ll come through on top.  Otherwise you may find yourself buried in complexities!