David Thach and Rick Rene share what they have learned are the most effective and readily adoptable agile processes, as well as a few techniques to integrate hybrid waterfall approaches. Companies adopt an agile software development framework to become more effective and more efficient, not to become a model of purist agile utopia—which, if attempted, ironically can be immensely costly and detrimental to progress, if not disastrous.
Through our combined forty years of experience consulting and running IT organizations in Fortune 1000 companies, we outline in this white paper what we have learned are the most effective and readily adoptable agile processes, as well as a few techniques to integrate hybrid waterfall approaches.
Companies adopt an agile software development framework to become more effective and more efficient, not to become a model of purist agile utopia—which, if attempted, ironically can be immensely costly and detrimental to progress, if not disastrous. A purist agile approach is not realistic for highly complicated Fortune 1000 companies that inherit unideal agile conditions. Teams with technical specialization, complicated reporting structures and governance committees, long-term roadmap planning and budgeting, conglomerate and subsidiaries in geographically dispersed locations, multiple shared product owners, mature legacy third-party installed applications, inherited roles absent in purist agile teams, a variety of IT vendors and partners with near-shore and off-shore blended workforces, and entrenched waterfall frameworks throughout hundreds of integrated applications are just a few Fortune 1000 company conditions that make it difficult to adopt agile.
As an example, one of our clients attempted to reconcile the Fortune 1000 conditions mentioned above with a full-on purist agile approach. This resulted in a fifty-six-page slide deck of prescribed instructions on implementing agile in their inherited Fortune 1000 environment that would also pass muster with all other stakeholders and their ancillary Fortune 1000 processes. It was complete with eight process flows, half-a-dozen swim lanes, and dozens of decision trees. They created this agile document after a year-and-a-half was spent gathering requirements in a waterfall-based approach. Their management, frustrated with the lack of progress, wanted to adopt what they heard was a new framework to push work forward more rapidly. However, the team ended up spending even more unproductive time creating the agile slide deck through countless discussions and approval rounds. And this was all before iteration 0 had even started.
In the end, they abandoned the fifty-six-page agile implementation slide deck, disbanded their attempts to socialize it, and instead called on us. In twelve weeks we not only implemented our simplified framework version of agile—what we are calling the “least common denominators” of agile—but we also developed the foundational components of their next-generation operational software.
As you can infer, there are aspects of agile principles that are more easily implemented in large, complex IT organizations that generate almost immediate benefits with very little relative disruption. These “least common denominators” of agile offer the biggest bang for the buck while being unintrusive enough to phase in at Fortune 1000 corporations. There are also hybrid waterfall and agile techniques that can be combined in order to gain the benefits of agile while both maintaining the benefits of the waterfall approach, which has been entrenched in the majority of Fortune 1000 companies, and enabling the framework to be more readily adoptable within these highly complex IT organizations.
Least Common Denominators of Agile
Agile is meant to be simple, but like all processes that start simply, it can quickly spiral out of control as layers are added and canonized in an attempt to standardize behavior. As a result, there have been tens of thousands of pages written about agile. But in this article, we are going to try to do the exact opposite and distill agile to its least common denominators. Agile can be broken down into three main process themes: artifacts, ceremonies, and engineering techniques.
Artifacts
The three key artifacts of agile are the product backlog, the iteration backlog, and the burn-down chart. Any individual team within a complex Fortune 1000 IT organization can adopt these artifacts by simply making a list of software requirements and to-dos (product backlog), breaking them into logical groups of work with estimates and assignments (iteration backlog), and tracking the work remaining (burn-down chart).
Ceremonies
The four key agile ceremonies are the release planning session, the iteration planning session, the demo, and the retrospective. Any individual team can adopt these ceremonies, even within a complex, nonagile, waterfall-heavy Fortune 1000 IT environment. Scheduling a meeting to discuss comprehensive software needs and high-level estimates? That’s a release planning session. Scheduling a meeting to break requirements into tasks, make estimations, and give assignments over a two-week period? That’s iteration planning. Scheduling a meeting to showcase your work over the iteration to your stakeholders? That’s a demo. Scheduling a post-mortem meeting to discuss what went well and what went wrong? That’s a retrospective. By holding regular demos and retrospectives with stakeholders and product owners every two to three weeks, the agile and iterative feedback will ensure the project stays on track.
Engineering
Engineering techniques associated with agile, including test-driven development, continuous build integration, automated regression testing, and scripted deployment, are more difficult to achieve but still fundamental. Start with a pilot team and ask developers to create test harnesses that fail until the functional code they write passes the unit tests. That’s test-driven development. Ask all developers to run a build of the code along with the test harnesses every time they check in files. That’s continuous build. Adopt a quality assurance tool that can automate behavior testing. You’ve got automated regression testing. Spend an iteration having developers work with the infrastructure team to create a deployment script. Scripted deployment complete.
The reason the least common denominators of agile are so easy to adopt is that it can be done in small pilot teams where it makes sense, without large organizational behavior change, major schedule impacts, new documentation requirements, additional resources, software adoption, costs, or substantial approval requirements. But by implementing these basic common denominators of agile, the benefits gained include an increased rate of software success, decreased software development costs, and more alignment with stakeholder vision due to the framework’s inherent flexibility and iterative process.
Hybrid Agile Techniques
At Daugherty, we have lead some of our clients to engage in thorough requirements elicitation prior to the start of software construction, very much like the start of a waterfall process. However, afterward we iteratively construct software using the least common denominators of agile described above. What makes this method ideal for Fortune 1000 companies that must engage in detailed planning for steering committees is that it provides the benefit of being able to have clear requirements up front, as well as associated benefits such as being able to accurately estimate a budget and resources and establish a high-fidelity timeline. All the benefits of the waterfall methodology are preserved, but you also get the benefits of agile.
At Daugherty we have honed the requirements process into what we call the RPM, or the rapid process map. Largely developed by one of our principal consultants, Phil Delanty, internally we also call it the 4-4-2. Those numbers represent the prescribed method of spending four hours eliciting requirements, four hours reconciling the requirements, and two hours reviewing them, then iteratively repeating the process until the scope of requirements is documented. As part of the four-hour requirements capture and four-hour requirements reconciliation, technical architects, user interface designers, database administrators, and other contributors capture and document use cases and wire frames and create data models for the two-hour review. It is rapid and it is aggressive. It requires talented individuals who can think and work on their toes. Once captured, the information is documented in a proprietary orthogonal view that allows for accurate work estimation and timeline planning to carry out in either a waterfall or an agile fashion.
The Old with the New
When the Agile Manifesto was published in 2001, it ushered in a time period of rapid agile software methodology adoption by IT organizations. Research began to show organizations that adopted agile practices created software with higher quality, better return on investment, and greater stakeholder satisfaction while delivering solutions faster and less expensively than traditional waterfall approaches. However, contrary to those statistics, on large enterprises within Fortune 1000 organizations the authors started to see high-profile project failures associated with agile strategies. Without a more disciplined approach to agile at the enterprise level, the agile movement will risk losing the hard-earned momentum agile pioneers have worked to generate. Sometimes it is better to adopt the least common denominators of a process and create a hybrid of new and old than to throw away the old and adopt a purist framework.
User Comments
"...being able to accurately estimate a budget and resources and establish a high-fidelity timeline."
I was surprised to read that accurate estimates and high-fidelity timelines are considered a benefit of Waterfall. We found estimation to be Waterfall's Achilles heel.
Our inability to make sufficiently accurate estimates (25%), quickly enough and at low enough cost to satisfy our customers, was the main reason we went agile 10 years ago.
I would love to hear more.
Hi Robert. Thanks for your interest. At Daugherty we have been implementing our unique, proprietary RPM requirements process at some of the largest Fortune 500 companies in the US. It's not just our process, but our talented people who repeatedly implement RPM at our clients sites that generates accurate estimates and timelines.
Feel free to reach out to me directly if we can help your company with our agile transformation process.
I find several inaccuracies in your interpretation of Agile/Scrum that need to be addressed. First, Agile/Scrum is not a process, it's a framework under which to build a quality product, including software. Let's start with artifacts.
These should be accurately be called user stories, which are organized into a product backlog and team backlog. There are no other artifacts required, although Agile does not exclude other valuable documentation like use cases and test cases. Yes, the language of a story is prescribed because it is a time tested way to share requirements and elicit conversation.
The team or project burndown chart is a dynamic information source, not an artifact and should be reflected in story points rather than hours. A burnup chart is even more effective.
Simply making a list of requirements circumvents one of the more valuable parts of Agile. As Robert points out, estimating is one of the downfalls of waterfall. By continuing this practice, you lose the concept of accuracy vs. precision. As far as I know, no software organization has been able to estimate with precision weeks (or even days) in advance of the start of development. By recognizing this, Agile uses the concept of Story Points which loose precision as the task gets larger and more complex.
Concerning ceremonies, you miss a critical one; the daily scrum or stand-up. The whole point of Agile is to make course corrections immediately upon encountering a deviation from plan. It is the reason that Agile works and waterfall doesn't except in environments that are guaranteed to be static. If you can write requirements or a plan that will never change, by all means use waterfall.
Finally, regarding the scaling of Agile, you are trying to fit (excuse my overused metaphor) a square peg into a round hole. With any change to the way an organization operates, change to the organizational structure is almost always required. This applies to business productivity tools as well as business process changes. To think otherwise is unrealistic. I am in the middle of the "Agilization" of a software development organization of over 500 people in a Fortune 1000 company. We recognized that the structure of the business and development organization must change as well as the relationship between them. We are doing quite well.
If you need some objective evidence for the above assessment, please refer to "The Principals of Product Development Flow", by Donald Reinertsen.
Thank you Ed for taking the time to expressing your knowledge, experience, and opinions with agile. We hoped that our article would hit a chord with practiioners out there to generate more discussions of this exacct natue.
In fact if you get technical, even the user stories you mention are not required as the agile manifestio never mentions it. Neither does it mention burn up or burn down charts. And, obviously our experience tells us that story points can be equally misleading. I can also appreciate your distinction between acuracy vs precision as well as framework vs. process. But, I think all of these "inaccuracies" reflect what we mentioned in our article as intolerant purist thinking and an attempt to over prescribe agile, which ironically is not the spirit of agile.
Working wtih 500 people in a scaled environment, you can understand that not everyone in your organization will have the same opinion as you. And, a part of being a good agile coach and leader is to be tolerant of the ideas of those you coach in order to be effective in helping them become a better organization, not just insist your opinions or what you know is right.
In the end our article is not about what is right or what is wrong. Unfortunately we see too much energy in that unfruitful debate. Instead our article is about what, in our 40 years of combined experience running and leading IT organizations, have made our clients successfull.
Robert/Ed,
Thanks for your comments. I can tell you it was shocking to me as a huge Agile fan to see how many Fortune 500s were struggling with Agile adoption.
When David started coaching teams and not doing it the way I had successfully done it with middle market companies, I was initially concerned. However, he ended up being wildly successful where other coaches adhering to pure Agile methods were failing. He listened and made small gradual changes without preaching and without trying to force my purist thinking on them. Now he has converted 4 separate teams in being highly efficient Agile teams, but by migrating them slowly to the least common denominators.
His methods worked and proved to me to be far more effective in complex environment lide the Fortue 500. In addition, I see Daugherty actually doing a great job at rapidly defining requirements that create a wonderful prioritized backlog that create the best of both worlds. An actual estimate that our current funding methods normally require but not taking massive time up front to build them. However, taking enough time to actually understand at least some gaurd rales on scope. Just enough scope to get an relatively acruate estimate, but not rigid enough to eliminate the scope flexibility needed with the Agile process.
I now subscribe to a continues learning process on what works best for the more difficult fortune 500 Agile adoption. My purist thinking is now being inspected frequently and adjusted to what is actually working at clients! :-)