History of Agile SCM
Prior to the February 2001 gathering in Snowbird, UT that resulted in the Agile Manifesto and the name "Agile Development Methods," Agile methods were called "lightweight" or "minimalist" methods. From 1995-2000, hardly anyone knew of any of them except for Extreme Programming (XP), though there were several others during that time (including Scrum, ASD, Crystal, DSDM, and FDD). Discussions of the application of SCM to such methods were fairly limited:
· XP had declared a set of 4 values, along with some principles & practices, and was quickly gaining popularity (and notoriety). On occasion, the moniker eXtreme CM (or XCM) might have been bandied about a few times on a handful of mailing-lists.
· Any mention of SCM for these development methods typically referred only to a small handful of CM-related practices (e.g., continuous integration) and how to create a CM environment (with a strong emphasis on tools & tooling) that would enable those methods and empower the team.
· Rapid Application Development (RAD) was around before Agile & XP came on the block, as were collaborative development environments. And again, almost all mentions of SCM in these contexts were in the form of enabling these kinds of project environments with tools and procedures (mostly tools).
To our knowledge, the term "Agile SCM" was first coined in 2000-2002 by Brad Appleton and Steve Berczuk (before "Agile" was trendy & popular) while collaborating together on their SCM Patterns book [7], and in a subsequent appendix they wrote for Ann Hass' book on SCM Principles and Practices [8]. Being part of the software patterns community in the late 1990's made both Brad & Steve privy to numerous discussions with, and review-drafts of papers from Agile pioneers such as Kent Beck, Ward Cunningham, Martin Fowler, Alistair Cockburn, Robert Martin, Michael Beedle, and others who would later found the Agile movement, create the Agile manifesto, and author the first books on eXtreme Programming, Refactoring, and Scrum. Some of those early discussions (prior to 2000) even took place on the original wiki-web (first created by Ward Cunningham), and many of those discussions are still chronicled there (albeit rather dated by now).
Interest in, and articles about, Agile SCM seemed to grow slowly at best from 2000-2004. About the only other folks writing specifically about Agile/XP and SCM were Darcy Weber (CM Strategies for RAD), Christensen (Tracking Change in Rapid and eXtreme Development), and Bendix & Ekman (see the ASCM project). The CMCrossroads community & portal sprouted at that time, as did the CMWiki-web. There were some initial exploratory (and heated) discussions there on some of the forums (and also the version-control mail list, when it still existed). In April 2003 the CM Journal had its first issue devoted to the topic of Agility and the following month gave birth to the regular "Agile SCM" column.
Somewhere near the end of 2004, there was a small spurt in articles on the subject (by authors other than articles and presentations by Steve, Brad, and Rob, and the annual CM Journal issue devoted to that particular subject). Several SCM tool vendors strarted to enter the SCM+Agile arena at that time and offered whitepapers and marketing geared towards an Agile audience. Since 2005, interest in "Agile" SCM has been growing a bit more quickly, no doubt due to the popularity of the "Agile" buzzword becoming more main stream and making its way into more organizations and SCM tools marketing.
Also in 2005, the greater project management community began to build momentum toward embracing Agile techniques and values, and their manifesto-like Declaration of Interdependence (DOI) was born. Since software CM has close ties with both project management and development, the DOI helped strengthen connections between CM with agile and project management, and Scrum is quickly becoming the most widely adopted Agile method. Increased economic trends towards globalization have also increased the focus on collaboration & feedback across geographically dispersed sites. The subject of scaling agile methods became the most popular in the Agile community during 2005 and 2006. So it is clear that much more "Agile SCM" history no doubt remains to be written.
What is Agile SCM?
Our definition of Agile SCM has not changed dramatically over the past 5 years. It is more than just CM for an Agile project, and encompasses attaining agility within the CM process itself. In addition to enabling an Agile project environment, true Agile CM requires taking on an Agile mind-set embodied by the Agile manifesto, applying that with Lean principles and techniques (including Theory of Constraints, and even parts of Six Sigma) and combining it with the principles and discipline of configuration management. The challenge of Agile CM comes when attempting to reconcile apparent conflict between Agile/Lean values and principles with CM principles and discipline. A synergistic balance must be achieved that attains the goals of both without compromising the values of either one.
"Agile" names a family of methods (rather than one particular method), that includes Feature-Driven Development (FDD), Scrum, Extreme Programming (XP), Crystal Clear, DSDM, and others. Some of the various agile methods have very significant differences, but they all share the common values and principles of the Agile manifesto. Very often people use "Agile" as a synonym for XP and/or think that all agile methods are more-or-less like XP (because XP is what they typically hear the most about, and often has the most "attitude" to go with it). FDD and XP for example are markedly different, yet both share the values and principles espoused by the Agile Manifesto.
In addition to holding common "Agile" values and principles, much of the theory and application behind Agile methods has its roots in the principles and techniques of Lean Production and the Theory of Constraints or TOC (including Critical Chain). These days, Six Sigma has fashioned a "merger" with Lean called (surprise) "Lean Six Sigma" (or just "Lean Sigma") that has paved the way for certain aspects of Six Sigma to "leak into" what it means for "CM" to be Agile.
Agile SCM is not CM-Lite!
The basic CM principles are the same. Some new principles/tenets are added to create new requirements, and a resulting new style to go with it in order to match projects having those additional requirements. There can even be a (more or less) common core set of practices, but the way they are instantiated and the additional practices and techniques and mind-set still create a different result. One size does not fit all kinds of projects. Just because a core set of principles may be the same doesn't mean everything else about it is.
To reiterate: Agile SCM takes nothing away from traditional SCM but seeks to achieve the same results in perhaps a slightly different manner while remaining true to the basic principles of SCM. So it builds upon traditional SCM and still contains the traditional SCM non-development activities of configuration planning, identification, control, status-accounting, audit & review, build & release management, and the corresponding process for managing those activities.
Agile SCM does however inject the agile mind-set (and some would say paradigm-shift) into the discipline of CM. With that, comes an emphasis on flow and throughput of the project value-stream, and operating in service to that and to those who create value. Sometimes this has led to the slogan, “Add nothing but value, remove nothing but waste!" when it comes to any form of process change or improvement in a Lean/Agile environment.
One way in which the mindset can manifest itself in SCM is in emphasizing flow more than discrete events (e.g., rigid baselines, and authorization/access controls), and treating the integrated "whole" as greater than the sum of its separately assembled parts. This can be difficult for many to adopt, particularly when it comes to practices like continuous integration and test-driven development using fine-grained tasks and many commits and integrations during the lifetime of the complete implementation of a requested feature, fix or enhancement.
Agile Manifesto
The agilemanifesto.org states the following: We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more. We propose some equivalents or alternatives that add value to SCM.
Individuals and Interactions Suitably Supported by Processes and Tools
SCM processes and tools should support the way that you work, not the other way around. Often, the reason behind frustrating SCM processes is trying to impose an overly-rigid process on a team, or attempting to use a tool that imposes a process into an organization that is not what the organization needs. SCM must be done in service to the flow of the value-stream and those who create value. It must achieve goals like repeatability, reproducibility and even traceability without compromising flow as a result.
One view of agile methods is that it is a license to hack and requires less discipline than traditional methods. In fact the reverse is true: agile methods typically require greater discipline and skills than traditional methods. A key tenet of Lean methods is that in order to move fast you can't afford errors and problems and thus need to be rigorous.
Applying this to SCM suggests that better understanding of SCM principles and tool usage by individuals and encouraging them, or making it easier, to do the right thing, is better than enforcing the process and absolutely forbidding the wrong thing.
Working Software and CM Processes over Comprehensive Documentation
CM supports the production of working software, but the challenge is to automate and ensure that appropriate processes are used that do not require huge process documentation.
Applying this also to the system being produced, we seek to automate as much as possible the production of supporting documentation (such as traceability reports). Information should not be repeated, and there should be a single master version.
Customer Collaboration over Contract Negotiation
SCM can facilitate communication and interaction among stakeholders and help manage expectations. The appropriate tools and processes can provide customers with visibility into the status of a project and allow them to contribute to the project where appropriate.
We value transparency and traceability that is maintained by the tools without overburdening the practitioners
Responding to Change over Following a Plan
SCM is about facilitating change, not preventing it. Use SCM policies and structures that allow your development team to progress at an appropriately rapid pace, but without losing control. Don't attempt too rigid a process or procedure. In conjunction with the first section on the importance of individuals, don't legislate for every last scenario in your process (all possible exceptions), but instead allow suitable experienced and trained individuals to make decisions in accordance with sound SCM principles.
Agile Principles
The full list is at: agilemanifesto.org/principles.html
We have selected and adapted those that we consider the most relevant:
· Our highest priority is to satisfy the customer by maintaining the integrity of the software throughout its lifecycle, and making early and continuous delivery easy and simple to achieve.
· We welcome changing requirements because we can manage and control them in a light-weight, but consistent, way to harness change for the customer's competitive advantage.
· Change and configuration management are the responsibility of everyone involved in the project from business people to developers. SCM people must work with the rest of the team on a daily basis.
· Give individuals appropriate tools and environments to perform effective configuration management throughout the lifecycle.
· While agile processes encourage conveying information face-to-face, SCM requirements lead to some level of easily recording, tracking, and managing change to that information, preferring automation to manual processes.
· Working software is the primary measure of progress (not SCM metrics!).
· Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
· Continuous attention to (SCM) processes and systems enhances agility.
· Simplicity - the art of maximizing the amount of work not done - is essential.
· The best architectures, requirements, designs and SCM processes emerge from self-organizing teams with SCM responsibilities devolved.
· At regular intervals, the team reflects on how to become more effective in their SCM processes and procedures, then tunes and adjusts its behavior accordingly. This includes taking incorporating ideas from Lean thinking, Theory of Constraints and other such initiatives.
Who are the Customers of SCM?
SCM is a foundation stone of development processes and underpins many aspects of software development. SCM is also an enabler of other processes and procedures. Thus the customers or stakeholders of SCM range from developers to QA, project management, release engineers, potentially auditors, and indirectly the end customers/business users.
Orderly Flow of Change and Evolution
SCM seeks to control change and we just need to make sure that this is adapted to consider the flow of change and ultimately the evolution of the system.
Systems that do not consider their evolution tend to become brittle and fragile, resistant to change. Activities such as refactoring can seem like unnecessary overheads, and yet their long term worth has been proven to many. Software must evolve gracefully, which requires supporting practices such as unit and acceptance test frameworks. Do your SCM processes have equivalent frameworks so that process can evolve in a well-supported manner?
Repeatability, Reproducibility, Traceability
Fundamental watchwords of SCM, and yet how are they achieved? If the effort is too high, then they become merely theoretical: "Yes I could reproduce the systems as released, but it is too much hassle and is it worth it?" This re-iterates the importance of appropriate automation (with unit tests or the equivalent), to ensure that systems can be advanced forwards or rolled backwards with faith and confidence.
Traceability should be something that can be satisfied at the touch of a button with information collected unobtrusively. It should not be at the cost of manual reporting and cross referencing, or of significant extra effort on a as part of their daily work by developers.
The 3 Pillars of (Vetruvian) Architecture
These are:
- Utilitas (utility): useful and usable (in terms of form, fit & function)
- Firmitas (firmness/durability) : product integrity (correct, consistent and complete)
- Venustas (beauty/aesthetics): simplicity and elegance of design
Note that for CM, all three of the pillars would apply to form, fit & function
These classic principles have influenced architects through ages, including Louis Sullivan, Le Corbusier and Mies van der Rohe with equivalent statements of "form follows function". As suggested by Jon Zingmark ([14]), the notion of venustas is often neglected. One of the requirements is short lead-time in delivery of software which calls for more focus on what you are supposed to deliver than on how you deliver it. We should aim to abstract the principles as much as possible and to automate the low levels in order to take care of venustas as close to the customer needs as possible.
Aspiring to be Configuration Directors Rather than Configuration Managers
One idea to take on board is expressed by noted business author Charles Handy [13]:
Go to the theatre and look at the programme. Everyone connected with the performance is listed, no matter how small their contribution. People like to be recognised as individuals. The word "manager" is reserved for those in charge of things, not people, the stage manager and the lighting manager. The people who are in direct communication with the customer, the actors, are directed, not managed, by someone who actually leaves the scene once the project is under way. He or she trusts the cast to go it alone, and as often as not, they improve on the production once the director departs. Trust inspires. One more thing - at the end of each performance they receive an expression of appreciation from their audience, direct feedback from the people who matter. No waiting for the annual performance appraisal.
Thus we can manage configurations of CIs, but we work with other people in the development team to direct the overall configuration and delivery according to CM requirements.
What can we do as SCM professionals to direct our teams to achieve SCM principles in an agile manner, without seeking to control every last action?
What training, education, inspiration, vision and support are necessary to achieve this? How can we empower the team members and let appropriate organization emerge from within the team and yet satisfy the traditional requirements of CM. The challenge is to get really involved with the team and work as part of the whole effort. This is something that experienced CM professionals have long been doing, but we all know how easy it is to slip back into a somewhat "ivory tower" mode, where we can almost be viewed as adversaries of the development team, people who "get in the way" of them doing their job.
Conclusions About Agile SCM, and Next Time: Part Deux!
Agile SCM is about having an Effective SCM process that helps get work done by establishing and ensuring the orderly, efficient flow of software development change & collaboration. Our aim with this article is to advance the cause and definition of "Agile CM". Anything claiming to be "Agile" needs to legitimately embrace the ideals of collaboration and the human element that Agile emphasizes, as well as Lean principles and techniques, including some heaping helpings of Theory of Constraints (including Critical Chain) and [Lean] Six-Sigma as they apply to CM and the "throughput" of change-flow, and change management.
The term "Agile" and the ideas behind it are no better or different from other labels (often called "fads") like Object-Orientation or [software/design] Patterns. They had lots of hype and buzz and they had some legitimate improvements to contribute to the state of the practice (even if the ideas themselves weren't new - that's not necessary in order to have great impact/influence). But once everything was all "hyped out" they faded into the woodwork, as a natural, acknowledged, part of the fundamental knowledge we should use and apply everyday.
Many tools/vendors are already headed in that direction, and have recognized the trend foretold by Tom Friedman (The World is Flat), Peter Fingar (Extreme Competition) and others like Dan Pink, John Seely Brown, and more. With CollabNet, MS VSTS and Rational Jazz, the emphasis is on connecting and collaborating. Ways of doing software development and CM that fail to acknowledge that, and at least the first five of what Steve McConnell calls the ten most important ideas in software engineering are going to have a tougher go of things than they have in the past.
As to whether or not "Agile CM" legitimately exists in its own right, that is determined by the people who end up actually doing it, and we count ourselves as well as other contributors to CMCrossroads as amongst the vanguard (although some would call it good practice as opposed to "Agile"!). Just as there are many different styles of architecture, and each has its appropriate context and use, so too are there different styles of CM, each with its appropriate context and use. There is more than one way (style) of doing good CM (and many more ways of doing it badly), and not all good ways will have the same style, nor should they.
The proof of the pudding, as ever, remains in the eating, and we thus welcome contributions/comments as to our success or otherwise of our efforts so far!
Next time we will be looking at the application and relevance of ideas from Lean Development, TOC, Six Sigma, and will also consider applying agile ideas to Change Control and CCB aspects of CM, as well as Value based software engineering (VBSE)
References
[1] Agile Configuration Management Environments; by Brad Appleton; CM Journal, April 2003 (Vol 2. No. 4). Also a March 2004 presentation to C-SPIN (the Chicago Software Process Improvement Network).
[2] The Agile Difference for SCM; by Brad Appleton, Robert Cowham and Steve Berczuk; CM Journal, October 2004 (Vol 3. No. 10).
[3] "Balancing CM and Agility" CMCrossroads.com [General CM] forum, July 2005
[4] "CM in an Agile Development Environment", CMCrossroads.com [General CM] forum, May 2005
[5] "Spiral Development", CMCrossroads.com [General CM] forum, May 2006
[6] "What Needs to Change in CM for 2007?", CMCrossroads.com [General CM] forum, January 2007
[7] Software Configuration Management Patterns: Practical Teamwork, Effective Integration; by Stephen P. Berczuk and Brad Appleton; Addison-Wesley, November 2002.
[8] Agile SCM by Steve Berczuk and Brad Appleton; "Appendix C" in Configuration Management Principles and Practice by Ann Hass; Addison-Wesley, December 2002
[9] Software Configuration Management in Agile Methods; by Juha Kosklea; ESPOO 2003, VTT Publications 514, VTT Technical Research Centre of Finland (ISBN 951-38-6260-7)
[10] Lean Software Development: An Agile Toolkit; by Mary and Tom Poppendieck; Addison-Wesley, May 2003.
[11] Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results; by David J. Anderson; Addison-Wesley, September 2003.
[12] Lean-based Metrics for Agile CM Environments; by Brad Appleton, Robert Cowham and Steve Berczuk; CM Journal, March 2007 (Vol. 6 No. 3).
[13] Myself and Other More Important Matters, by Charles Handy, Arrow Books, 2007 (ISBN 9780099481881).
[14] Private email by Jon Zingmark.
