Quite often software configuration management (SCM) is backed against the wall because following the process is going to cause someone a delay, or worse yet, inconvenience. Using a bit of bit humor, we’ll look at some common complaints and the underlying reasons why we put the rules in effect in the first place.
Here they are in no particular order.
- Why do I have to write a separate issue record for each item? Can't I just lump them all in one?
- This can't be tested, just put it in production.
- It was only one line of code, what's the big deal?
- No, no, we use a serious CM tool -- VSS!
- I'll just modify the file in production. We can copy it to the repository later.
- We'll test this one and just recompile before we release to production.
- They fixed that in production but we haven't retrofitted it to development yet. We'll just promote ours to production and re-fix it later.
- Why don't we release it now and put the request in the CCB for approval tomorrow?
- What you are seeing can't be happening. I'm certain.
- Don't use the words "bug" or "defect." That would automatically imply there was something wrong with the code. It's an "issue.” We don’t want to upset the developers.
Each of those items would suggest that following process in just burdensome and gets in the way of “real work.” Inherently, project teams seem to achieve an almost mythic capacity for optimism when debating on how much process is necessary. Conceived and executed properly, process will save time and money and, possibly, a career or two. But this cost savings comes at an organizational level, rather than individually.
Consider this process:
- Run as fast as possible.
- Hold baton firmly in right hand
- Locate teammate ahead of you.
- Get in step with teammate ahead, matching pace.
- Move baton to specific angle in hand
- Hold baton at specific distance from body
- Maneuver into teammate’s hand
- Confirm grip of teammate
- Release grip on baton
- Reduce speed and move off track
Ten steps for just one part of a relay race. Can we do the relay without the process? Sure we can. Can we hope to be the race winners without it? Doubtful. It may be more comfortable for a runner to hold the baton in the left hand. Perhaps the angle of the baton at transfer is awkward. But this is the type of process we want to achieve. Each individual in this example is motivated to deliver in a way that specifically contributes to the project’s bottom line (winning the race) and reduces their own potential for failure. The project team quotes early in the article reflect a team that isn’t working together or isn’t aware of the potential conflict they are causing themselves.
Let’s look at the first example.
Why do I have to write a separate issue record for each item? Can't I just lump them all in one?
Lumping issues together into one record causes several issues. First, the record cannot be passed until every issue listed is satisfied. This typically causes the most difficulty at the end of a project phase when the team is debating on which issues to defer to the next phase. It causes frustration from a traceability standpoint. I now have code labeled or packaged for release that is directly linked to a failed functionality. Breaking the issues down means you can more clearly identify code that is signed off and ready for production instead of being questionable. Most directly, it’s frustrating for a developer who now has to parse out which part of the code base is viable and which is not, while tracking dependencies. It will almost certainly take more time to split out code than it would have to start with individual issues. Most issue tracking systems are user friendly and do not take any significant amount of time to enter issues.
This can't be tested, just put it in production.
This gem is more often a failure of imagination than it is a resource issue. Many project teams will say it can’t be tested because of various systemic issues. And that may be the case. But commonly, a testing team pressed for time, because it’s their time getting cut first, will just agree with the assessment and skip the work. Putting untested code into a production release is truly a last resort effort. Depending on the type of software you are working with, code failures may be far more expensive than the delay being avoided.
It was only one line of code, what's the big deal?
It’s the same as untested code. Changing a line of code, even a comment, has been known to cost companies significantly. I know of one case directly where a comment was added post testing but the developer inadvertently failed to close the comment causing more than $3 million in product correction. One line of comment, not even code, sparked a completely avoidable, and expensive, event. Developers often view it as an insult to their ability or trustworthiness. It’s simply good business sense. As the Gipper put it, “Trust, but verify.”
No, no, we use a serious CM tool -- VSS!
This is no dig on that goliath software company; it’s simply a recognition of what constitutes a CM tool. A good CM tool will allow the user to both version control the code and then relate a version to various releases or issues and the maturity of each version in the lifecycle. Being able aggregate the code at that build/release/issue tracking level provides a significant amount of change control that is at the heart of configuration management. It should allow the administrator to define at least some level of process within the tool for the benefit of its users. I pick on Visual SourceSafe not because it’s made by a particular company or fails to do what it’s not designed to do. I use it here as an example of how project staff misunderstand the function and practice of good configuration management. A good CM tool should allow you to channel users in good, defined process to make it easier on them and accelerate their work.
I'll just modify the file in production. We can copy it to the repository later.
Uggh. Production support teams, development teams, anyone with access to the production servers are under tremendous pressure to get things running again. There is always money on the line. But this particular comment is geared toward individuals who are not production support. Many organizations have processes in place to work prod fixes back into the code base. It’s not a preferred arrangement but then doing business often forces certain compromises on us. This is for the character that’s hacking on the fly and being rather cavalier about the code base. Ultimately it’s the development staff that loses. All development should start from a known good state. When code is not worked back into the repository, we are left with a mismatch. We are almost guaranteed to put the offending issue back into production that was fixed by our coder. Hopefully we have audit functions that can catch the mismatches but we rarely have the capacity to do 100% production audits for numerous reasons. Be assiduous in tracking production changes outside normal development channels. No matter who’s at fault, some unsavory material will stick to you when bugs are reintroduced.
We'll test this one and just recompile before we release to production.
The issue at play is that testing is putting their efforts into a specific release and when code is recompiled we effectively start back at zero. The argument put forward most frequently is that if we use 99% of the original code base we remain 99% accurate. Many, many cases of new bugs are the result of the unanticipated effects of changes. Similarly, adding 1% arsenic to 99% healthy ingredients still leaves you with poisoned food. Under controlled conditions this is not necessarily a problem. When the CM takes the validated code and recompiles with a known good environment variable file, it’s completely appropriate. Adding in additional, untested code is at heart of the beast indicated in the statement above. Code may work just fine in unit testing and fail utterly in system testing.
They fixed that in production but we haven't retrofitted it to development yet. We'll just promote ours to production and re-fix it later.
When I first heard this statement, I thought perhaps it was a function of not hearing the statement accurately. Then I thought it was a translation/accent problem. Ultimately of course, the answer was “no, you are not doing that.” The first rule of CM is don’t break production. It’s a very rare event that a set of functionality has to be released before recent production fixes can be merged back into development. Releasing old bugs is a good way of causing problems the business side wasn’t expecting. If you truly have to go down that road, the business side of the house should be fully on board and ready with their work-around.
Why don't we release it now and put the request in the CCB for approval tomorrow?
The point of the CCB is to provide a balanced, organizational view to what should be released into a given environment or pursued within a project scope. It allows various functional areas to have a say into how a given change will affect them. Ambushing them with a release before their approval can cause frustration, rework, and resentment needlessly. If you need a more flexible approach to your CCB, revamp your CCB. Devise more effective ways of communicating and approving. Don’t short-circuit your process.
The comments we receive that drive us batty, like those in this article, are often good indicators that either our process is not understood or is not serving the organization as effectively as we might hope. Sometimes it’s just a function of a project team that hasn’t quite achieved the level of maturity to which we might aspire. Regardless, that leaves it up to us to fill the gaps and provide the leg-up for the rest of the team to move to the next level. It’s about communication. The very best process is self-motivating and it channels a user in the right direction without inhibiting them or the organization. CM is a practice that is both guardian and old salt.We seek to better everyone’s lot in the lifecycle. Software creation is a discipline, with all that word entails.
Randy Wagner is a Contributing Columnist for CM Crossroads and Software Configuration Manager with ABN AMRO. His experience in SCM ranges from global financial institutions to multimedia multinationals to the Federal government. Working in small to large project efforts has given him a unique perspective on balancing the discipline of SCM with the resources and willpower each organization brings to the table.
