Product and Project Software Configuration Management (SCM)

Products and Projects 

Before we venture into any SCM discussions, we first must classify them. The lean development mailing list had a pertinent discussion on this. Mary Poppendieck commented:  Products are development efforts that are (usually) funded incrementally. They start with a concept and end with a product launch. Along the way, they are funded incrementally as they progress through stages such as feasibility, test market, commercialization, etc. Products are also expected to have a long and useful life over which timeframe they are generally expected to undergo constant improvement.

Projects, on the other hand, due to their contract origins, tend to be fully funded at the beginning of the project. This front-end loaded funding tends to drive the people providing the funds to ask for scope and schedule plans, which are then treated as commitments.  After all, the funds are committed, so they want to know what they will get for the money. Additionally, on-going change after the end of a project is generally regarded as bad. Alan Shalloway contributed: Lean extends some othe Agile methods (e.g., Scrum, XP) that are project centric. Products are about something the company offers to its customers (internal for IT products) while projects are about building the product. Thus good products will grow, mature and continue to add value to the customer in different ways and may spawn multiple projects.

Many companies have more waste in their project selection process than in their project building process. They will spend 1 - 2 years deciding whether to approve a project. Once they have selected a project to build, scrum and XP can be put into place. However, what they really could use to help them is the understanding of waste that Lean provides. Waste includes delay, task-switching, hand- offs and other things which don't show up on an accountants balance sheet. Value stream maps are incredibly useful tools in this area.

Internal vs. External
We also have the difference between internal and external products.

External products (or systems or applications) are things that we sell in order to generate revenue. The more customers we have to support (even on different versions/platforms), the more revenue we generate in return.

Internal products (or systems or applications) are pure operating cost and do not directly generate revenue. The more customers we have to support (especially on different versions/platforms) the more it costs us to spend time, money and effort on activities that are not revenue generating. The people and time we spend on them means less money spent on revenue-generating activities. We only do it because we think we are getting productivity and/or cost-savings (e.g., over not having the tool/applications) in return and we still want to minimize it. Internal apps generally have a harder time justifying their existence/value to the business and their impact on the bottom-line, which may as well equate to justifying CM.

Also, there is a huge difference between delivering to customers who don't live in your own backyard and aren't part of the same organizational culture and politics. There are typically extra layers of customer care/service between development engineers and direct customers/users, and the internal customers also have a lot more visibility and transparency and closeness into what is going on.

Even more interesting is the notion of consultants who are hired strictly for the duration of a project (regardless of if it is internal or external) and how much that motivates and rewards them to think and act in the best interests of the product and not just of the one or two projects that they will be around for. It is easier to come in and put something together fast and get a group up and running. The poor folks who are stuck with the internal support and maintenance, if the outcome is ill-structured, un-factored, poor quality code, are the ones that the company may end up spending a lot more money for in the long-run.

Just because we are picking on consultants here, doesn't mean that organizations doing it all themselves don't suffer from similar problems. We have seen short-term project thinking within a large program of internal development.  The teams came together, delivered the projects and then were disbanded (due to internal charging models) without paying much attention to the long-term maintenance requirements.

Deployment Models
There are many possible models of deployment that can apply, particularly to products:

• Local deployment & installation, with a private/independent copy at each installed site
• Centralized, shared, monolithic, such as a travel service web site, or even something like Google Docs
• Decentralized and distributed

There are others, too. Each one of these may or may not lend itself well to a particular maintenance and support model:

• Supporting multiple historical versions
• Supporting only current version while working on the next one
• Supporting a product-family of variants of the same set of components

Which model you use depends primarily on upon total number and type of distinct customer-bases and the market (i.e., how likely it is you can expect them to be willing and able to upgrade to new major versions and how much they are willing to pay you to support older ones).

Architecture/SCM/Organization
With the above definition, we can review some of the SCM considerations as to the difference between products and projects. We have discussed how many release engineering issues can be addressed through a balance of architecture, organizational approach, and SCM strategy. Also, using the above definitions, we can consider project SCM needs to be a subset of product SCM.

Product development tends to have release lines - potentially multiple independent projects under development at the same time. As Laura Wingerd discusses in Chapter 7 of her O'Reilly Book (Channeling the Flow of Change):

• First, at any point in time there is likely to be more than one supported product version available to customers. Ace must be prepared to field customer calls, diagnose problems, and fix critical bugs in all of their currently supported versions.
• Second, not all development tasks have the same urgency. Some are expected to yield results immediately while others are targeted for distantly future releases.
• Third, software development is not entirely predictable; some projects go according to plan, others get mired in unforeseen difficulties.

Laura presents release and project branching patterns, and also discusses the Mainline branching model.

The ITIL Service View
Many organizations undertake a variety of internal projects that deliver new or enhanced applications or systems into live usage within the organization. A typical new system or product, delivered as part of a project, might follow a lifecycle and branching model shown below.

From an ITIL viewpoint, the service transition phase (e.g. into live usage) is very important and one of the key measurements is the ongoing documentation and support process that allow the system to be cost effectively maintained in the future. This recognizes the reality that the initial development of most systems is 30% or less of the overall lifetime of a syste

As mentioned above, project teams who are measured only with regard to successful delivery of their project and then leave, for the next project, can leave considerable maintenance problems behind them. We have seen this plenty of times as a significant problem for organizations. And yet this model is still very common.

Enhancement Projects
Many internal projects are actually enhancing already existing products, so the branching patterns should really be represented as starting from existing state and delivering back a changed (enhanced, we hope) version.

Multiple Parallel Projects as Part of a Product or Release

We then get to the case where there are potentially multiple projects or programs, each of which is delivering particular enhancements. These multiple projects may overlap in some way and both be changing a particular System, which will then need to be integrated 

Lifecycle of multiple project development shows multiple projects or programs in development with their code being integrated for a particular release (i.e., 6.10) that occurs during system test. Prior to this point the projects are relatively independent of each other. The individual solutions may be concurrently updating the same application (Y in the example above).  Integration and merging of the changes is done at the point of systems integration and test. The diagram omits the detail of the stage 1 (dev) branches. Also omitted are changes that may need to flow back to the projects as a result of problems during integration.

A very common model here is to have regular release cycles (i.e., several per year), where multiple projects are integrated and delivered into live usage as part of a global release.

In the IT Process Institute Change Configuration and Release Performance Study the most important two findings are:

• Release trumps change: Rigorous release build, testing and rollback practices have broad impact on individual performance measures and overall performance. Change tracking and change oversight practices are necessary, but not sufficient, to achieve performance improvement on their own.
• Process discipline matters:  There are no change, configuration and release silver bullets. Building a process-focused culture and monitoring and responding to process exceptions predict top-levels of performance more than many of the industry recognized best practices in these areas.

Indeed, the report overview (available for free if you register) says the most important practice is:

• Release scheduling and rollback-In this set of practices, IT organizations develop and maintain a fine-tuned cycle of build and test, and then release only during maintenance windows with tested rollback plans. Data about the root causes of release exceptions is then fed back to systematically improve the process.

Agile Impact
To this point, we have been discussing general SCM implications more than Agile in particular, so now let us consider some agile impacts.

Iterative Development - Releasing Frequently
Iterative development is not restricted to agile methods, and yet is a core part of agile development. If you release frequently, you become more skilled by looking for automation and not ignoring the many little details that can otherwise trip you up.

TDD and Test Frameworks
This is becoming increasingly common practice and one of its main benefits is that a good set of automated tests provides a safety net for future changes. They also act as documentation for the , thus making it more likely to be maintained along with the code.

There are still challenges in terms of finding the right level of automated test frameworks. Initial work focused on unit test suites during development. More progress is being made in automating tests later on in the cycle - integration and acceptance tests, as well as unit tests. This can, of course, be particularly challenging for systems that have many external interfaces.

Conclusion
Projects and products are different and yet how important is the difference? As Mary Poppendieck reminds us, “Besides this direct value of thinking products instead of projects, the concept also reminds us that our customers are not really interested in software anyway. The software we deliver is a piece of a larger product that delivers value to the customer. We must always remain value-centric to the customer.”

From an SCM viewpoint, the most important relevant distinctions that drive the selection of appropriate SCM patterns are:

• Internal versus external
• Deployment model
• Degree of parallel working involved
• Number of supported releases

There are plenty of good patterns to use and agile methods that emphasize continuous integration, regular builds and regular delivery of working systems that deliver value and can be very successful in either camp.