Where Did Configuration Management Go?

Amid all the excitement of DevOps, continuous delivery, and the magic of single-push-button deploys, it seems that some folks have forgotten the prerequisites. The fact is that you cannot implement continuous anything without having effective configuration management.

This article will help you reassess where you are and ensure that you have the basic building blocks in place to ensure your success.

Effective configuration management starts with source code management. This is an absolute requirement for success, but sadly, I have seen many teams struggle with unreliable tools and even losing code. Version-control systems (VCS) are a must-have, and you need to know how to use them effectively. This includes being able to label code baselines and create branches to manage variants in the code, such as bug fixes. Regardless of which tool you are using, the most important consideration is ensuring that you provide training to your team that explains how to use your VCS, as well as essential procedures such as tagging and branching.

Build engineering must be fully automated and is also an absolute prerequisite to continuous integration, delivery, and deployment. You should always be building code from an identified baseline and embedding an immutable version ID into each configuration item. Having a repeatable process to build your code enables your team to support bug fixes without running the risk of code regressing due to the wrong version of some compile or runtime dependency.

This step is where you verify that the code baseline can be built using your existing procedures. Many companies fail to implement effective build engineering. I find that the key to success is ensuring that someone else besides the developer can successfully build the code, usually a build engineer.

Environment management includes understanding and monitoring all environment runtime dependencies. Many companies struggle with environment management, and it is a very common cause of system outages. Too often developers do not communicate runtime dependencies, and this information is lost as they move on to their next project. You need to ascertain and manage environment dependencies from the very beginning of the lifecycle, or else you will be discovering them one by one when your system goes down.

Release engineering helps provide a framework for packaging releases. You should have a manifest in your release package listing all its contents. This is often called a bill of materials (BOM). Your release container should also be signed using asymmetric cryptography with a public/private key pair, which means you can be certain that the package has come from a trusted source.

I always include a script to recalculate the BOM directly from the runtime configuration items. I call this my release map, and it has saved me many times by giving me a way to compare what I shipped to what is actually found in the production environment. This is how I discover changes made by systems administrators who may mean well but whose actions could cause the system to crash.

Deployment engineering focuses on automating the promotion of the release package to the target environment. Procedures to fully verify that the deployment was successful are known as a physical configuration audit. The most common problems I see when assessing CM best practices is that most companies cannot prove they have the right code in production, and they also cannot detect unauthorized changes, which creates a tremendous amount of risk. Your procedures must be completely reliable, and you must have both a way to verify that the correct code was deployed and an automated procedure to detect unauthorized changes due to either malicious intent or human error. You also must have procedures to back out a deployment if necessary.

All these practices should be driven by an effective change management process that helps identify technical risk. Risk is not inherently bad, but you must be able to understand and mitigate any potential challenges that might occur. Don’t forget to manage changes to your process, as well—this is usually handled by the software engineering process group (SEPG).

Getting started with CM best practices should always begin with an assessment of your current practices and a pragmatic plan for process improvement. I tell folks to start with a couple of easy improvements to help the team realize that change is very doable, effectively improving their culture and sense of effectiveness. The SEPG takes responsibility for communicating all process improvement initiatives, including establishing IT controls to meet audit and regulatory requirements.

DevOps can certainly help establish good CM by improving effective communication and collaboration. Continuous delivery and deployment cannot happen without configuration management. When we are asked to help teams establish continuous delivery, our assessment often finds major gaps in configuration management best practices that must be addressed before the magic bells and whistles get implemented. DevOps and continuous delivery may be the flavor of the day, but you still must have the ingredients in place to bake the pie.