There are many accepted processes and approaches to managing enterprise requirements. These approaches vary from model-focused to document-focused and are well known and practiced by requirements-gathering specialists throughout the business world. The real intent of any enterprise requirements management organization is to accurately capture and manage needs of the business, and organizations are built to create repeatable processes for doing so. However, once the focus shifts toward processes for eliciting and managing requirements, and away from the business goals, problems can arise.
Defect reduction through early and detailed requirements specification is a common outcome of process improvement efforts in product development. Enterprise requirements management becomes a specialization that requires expert business systems analysts to gather and document specifications that are detailed, accurate, and complete.
If we apply Lean principles to the requirements gathering effort, we see a backed up queue. Working to "Eliminate Waste" is a fundamental premise of Lean thinking in Software Development. Mary and Tom Poppendieck consider this so fundamental that they make it principal number 1 of 7 in their Principles of Lean Software Development. [i] They dispel the myth that "early specification reduces waste."
In fact, for software development, early specification is waste. This article extends this myth-busting by exposing what can happen when process improvement techniques are blindly applied to requirements gathering. The suggested solution is to replace early detailed specification with solution roadmaps that can be detailed by collaborative teams at just the right time. Agile methods provide the structure and mechanics to allow business vision to lead product development with cross-functional teams that unfold detailed requirements when needed.
There are many accepted processes and approaches to managing enterprise requirements. [ii] These approaches vary from model-focused to document-focused and are well known and practiced by requirements-gathering specialists throughout the business world. These approaches all have common objectives that can be summarized to be:
- Eliciting requirements
- Managing scope
- Managing risk
- Managing change
- Validating for accuracy
- Validating against business and enterprise architecture
The real intent of any enterprise requirements management organization is to accurately capture and manage needs of the business, and organizations are built to create repeatable processes for doing so. However, once the focus shifts toward processes for eliciting and managing requirements, and away from the business goals, problems can arise.
Author Alan M. Davis emphasizes this in his requirements primer, where he states "Requirements management is a means to an end; it is not a goal." [iii] Organizations that focus on processes to manage these objectives represent an anti-pattern that clearly violates Agile and Lean Principles. At the simplest level, this creates an immediate distraction from the reason the requirements are gathered in the first place--to create business value. [iv] This is called out as an impediment of the enterprise in Dean Leffingwell's nice treatise on scaling Agile practices. [v]
The notion that process-focus can create dysfunction is not new. The originator of Lean thinking, W. Edwards Deming, warned of poorly applied productivity measures back in the 1950's. In Out of the Crisis, Deming states: "Measures of productivity do not lead to improvement in productivity... On the other hand, an orderly study of productivity, to enquire whether any given activity is consistent with the aim of the organization, and what it is costing, can be very helpful to the management." [vi]
Focusing on process allows work to occur that is not governed by business value. It's reasonable to see how the good intentions of creating a controlled requirements management organization can create intense focus on processes to gather and manage requirements. These good intentions can result in a very efficient requirements gathering "machine."
Unfortunately, this machine creates a backed-up queue of work, and creates a chasm between the people implementing the requirements and the visionaries that are driving the direction of the business organization.
By insisting on efficiently and accurately documenting requirements as understood by the analyst, the overall organization is left with volumes of "locked-down" requirements specifications that make the organization brittle, since these documents must undergo rigorous change control that slows down the organization's ability to change direction. The well-intended requirements silo represents clear violation of three values of the Agile Manifesto, by valuing "process" over "individuals and interactions," "comprehensive documentation" over "working software," and "contract negotiations" over "customer collaboration."
Waste(ful) Management
Early specification of detailed requirements forces an organization to manage costly artifacts using equally detailed project plans. These detailed plans serve to create the illusion of control, but really track the transformation of requirements documents into different phases of models and code through analysis, design, and coding phases. These transformations add little value to aligning the project team with the overall goals of the project, and should be viewed as waste.
In fact, organizations that focus on processes to manage and transform documents and models lose line-of-sight with the end result. An unintended consequence of these distractions is that teams lose their identity, since their whole worth is now tracked against how effective each individual is in completing his/her assigned task from the organization's process map. Add to this the need for detailed project plan tracking, and the end result can be a highly dysfunctional organization that is poorly aligned with creating the highest value business products. This should come as no surprise, since this violates the Agile Manifesto by valuing "following a plan" over "responding to change."
In large enterprises, early specification is not only wasteful, but becomes redundant when compared with acceptance test cases. These same organizations require enterprise system tests that must be painfully created to validate the enterprise requirements. The system tests on their own represent the clients' desired view of the system, and can stand alone as enterprise documentation of the system. Lean and Agile practitioners often suggest that the most efficient approach is to let the acceptance tests themselves represent the legacy documentation of the enterprise.
"But That's Not What the Requirements Said..."
Perhaps the most obvious (but most ignored) sign of a dysfunctional IT organization is when a well-meaning project manager rationalizes bad execution by claiming "...we implemented according to the requirements." When this excuse is heard, it is time to stop the assembly line and tear down silos. Unfortunately, the typical organizational approach is to bring in "process experts," thus kicking off the painful search for the "root cause" of requirement gathering issues.
Requirements gathering is seen as a process that can only be undertaken by specialists, and so the Business Systems Analyst organization appears. This group of specialists means well, but its activities can end up creating an inefficient buffer between business and IT. This is analogous to commissioning a portrait, but having to communicate with the artist through a third party--through detailed documentation created by a well-meaning art specification specialist. Imagine an artist, painting your portrait according to a written description, with a third-party "expert" providing feedback and clarity to the artist throughout the creation of the portrait. This is a scary analogy, particularly since software development is often referred to as "art."
There is no doubt that some requirement specifications are better than others. There are clear and vague ways to state the same thing. But there will always be opportunities to misinterpret words based on varied backgrounds and experiences of the reader (especially if the reader is isolated from the writer and can only request clarity by capturing a list of issues and emailing them to the author).
Most organizations realize this and require rigorous inspections, complete with formal sign-off in an attempt to create quality by inspection. Unfortunately, these inspections are not held often enough, and inspectors are expected to review hundreds (or worse, thousands) of pages of documents in a short amount of time. A clear sign of dysfunction is when well-meaning inspectors come to these formal reviews and call out spelling and punctuation errors in the document, and miss the real intent, which is to verify accuracy and relevance to the business solutions required by the effort.
Creating detailed early specifications represents a Lean anti-pattern that violates the Lean principle "minimize work in progress." This is like writing thousands of lines of code before compiling and integrating.
In the end, silos of optimized processes create organizations that cannot tolerate change. This should come at no surprise, since the techniques for process improvement come from Six Sigma, whose goal is raise quality by making processes repeatable and non-varying. This trait is manifest in a dysfunctional organization when project teams categorize new requests as "requirement changes" that must go through "change control." Change control is a costly symptom of a dysfunctional, enterprise organization.
What About Use Cases?
Legions of proponents espouse the virtues of the mighty use case. Its celebrated purpose is to bridge the chasm between IT and business by providing user-system interactions described from the user's perspective, in language understood by both business and system developer. Use cases gained acceptance as organizations formed around the Rational Unified Process (RUP). Well-written use cases have served software delivery teams well, in that they help break flows into discrete, analyzable steps.
These steps map well into the analysis and design models that are fundamental to many enterprise delivery organizations. The intended value of use cases is best realized as a tight coupling of business goals to the system design. The most powerful example of this value is when use case steps are listed beside logical and physical sequence diagrams, as described by Ambler. [vii]
The business value of well-written use cases has been so widely recognized that organizations have been created focused on the use case as its deliverable. The unintended result of creating an organization to generate use cases is that the artifact becomes the group's focus and reason for existence, instead of the direct creation of business value (product). The pattern below is not unusual in large IT organizations:
Charged with optimizing the creation of use cases, process experts rush in and focus on how to most efficiently capture and manage use cases. The logical place to focus is on re-use. "Why should we create the same use cases over and over?" is a valid question. Generic use cases begin to appear with generic actors. These are managed with "exception steps" that allow a generic user-action/system-response pairs to be used over and over, as long as the appropriate statements are added. Soon, well-written and easily understood use cases transform into documents that are understood by neither business nor developer. In the end, the well-intended goal of re-use creates unusable documents.
This is a noble sub-optimization effort. But it is a lean anti-pattern, clearly breaking the "optimize the whole" principle. Over time, the management of enterprise use cases creates integration and validation work for the business and development organizations. In Lean terms, we have a backed up queue of efficiently generated requirements that are difficult to read, integrate, and validate.
In a moment of courage, I once challenged a silo of use case experts to prove that a set of enterprise use cases were "accurate." My challenge to the inspection team was simply to trace through the documents for a specific actor, performing the most basic function, and validate each step along the way. This happy path tracing ended up taking a room full of experts two full days to complete. During this inspection, infinite loops, jumps to nowhere, and other issues were discovered and documented.
These integration "bugs" would not have been uncovered until analysis, which is typically when developers start looking at the requirements. Imagine the flow obstruction (Lean anti-pattern) that would result from tracking these "issues" to closure. It is no wonder software is often over-engineered and full of waste.
The Business Value Roadmap: How Index Cards Bring Structure to Business Vision
We've reviewed how two bridging efforts (requirements gathering specialists and use cases) can cause unintended outcomes and actually extend the chasm between Business and IT. So what is the answer? How can structure be brought into enterprise requirements management for product development or IT project efforts in a way that reduces time-to-market, keeps quality high, and increases organizational learning? The answer requires courage--courage to empower the enterprise organization with the ability to work in self-sufficient, highly collaborative units that are allowed to pull prioritized capabilities and implement based on the team's capacity.
This activity is led by a Business Value Roadmap, written in business terms and goals, which is detailed at the right time based on each team's capability to begin implementation. It presupposes that an Agile organization exists, and that top-down support is in place for removing dependencies from each Agile team. This prerequisite is not easy to achieve, but once the organization has scaled Agile methods to the point where multiple cross-functional teams are in place, the Business Value Roadmap can guide the enterprise and keep tight alignment with changing business needs.
The roadmap itself is easy to generate and manage. Capabilities called out in Project Charters and Statements-of-work can be prioritized and changed as rapidly as business goals dictate. These capabilities can be tracked as epics, which are defined to be high-level business goals and are typically broken into smaller features defined as user stories or more simply stories. In the Agile world, an accepted practice is to capture epics on white index cards, and stories on yellow index cards.
Mike Cohn defines a user story as information that "...describes functionality that will be valuable to either a user or purchaser of a system or software." [viii] It can be thought of as a small piece of functionality captured in business language that can be written on an index card with a permanent marker. The index card metaphor is valuable in that it limits the amount of detail that can be captured up front.
When detail is required, an cross-functional Agile team breaks the story down into small tasks, which are typically captured on blue index cards, along with an estimate. The key take-away is that the detail is not determined until it is required. The roadmap itself should keep visible Capabilities (captured by epics) that are tightly coupled with business goals.
An expected, reaction to capturing enterprise requirements on index cards is laughter. So much energy has been put into creating enterprise templates to capture Project Charters, Capability Matrices, Use Cases, Requirement Specifications (functional amp; non-functional), etc., that these artifacts will be defended by the organizations that creates and manages them. So how does one introduce Agile requirements in such an organization?
My recommended approach is to spend extra energy preserving the mapping between high-level capabilities and epics, and finding ways to map stories back to the enterprise documentation. This is a nice transition role for Business Systems Analysts looking to contribute on an Agile pilot. In the end, capturing ideas on index cards and placing them on a prominent wall gives visibility and line-of-sight to all work in progress.[ix]
Lean principles dictate that work-in-progress should be minimized in order to maximize productivity. Early and detailed specification of requirements represents too much work-in-progress that is wasteful and distracting. Pushing large requirements specifications through project factories spreads focus thin, as low priority requirements suck as much time and energy from the organization as the highest priority requirements.
Challenged with such volumes of poorly understood requirements, over-engineering flourishes (e.g., "I'm not sure what that means, so we better design for the worst case"). A better approach is to free up an organization's energy by empowering teams to do what they know is right--focus on testing and quality so that better designs and architectures can emerge as needed. This just-in-time mentality requires courage, since control is relinquished from management and given to collaborative teams that undertake team learning to solve difficult problems of technical product development.
Ken Schwaber sees the solution space to be totally within the rules of Scrum. Ideally, Scrum scales perfectly, and the "enterprise's work can be organized, top to bottom, into a single Product Backlog." [x] This represents well the conceptual scaling of Scrum. One should also be aware that scaling Scrum teams adds degrees of complexity that are best framed by Lean principles, since teams of teams have different dynamics and communication needs than teams of individuals.
The enterprise product backlog prioritization should be guided by such principles as optimizing the whole, eliminating waste, and delivering fast, and a properly scaled enterprise can then pull from the prioritized backlog based on each team's capacity and the needs of the enterprise.[xi]
Enterprise Silos
As mentioned above, an unintended outcome of Enterprise Requirements Management is the building of silos. It's not uncommon for a mature technology organization to have a silo devoted to managing a portfolio of technical tools that are approved for use within the enterprise. The justification for this organization itself is several degrees away from any business vision (unless the business happens to be technology).
Summary
Examples have been presented where Business Analysts and Use Cases can be misapplied to the Enterprise requirements effort, and can result in Lean anti-patterns that fail to bridge the chasm between business and IT. Agile requirement techniques, such as the Business Value Roadmap, focus Agile teams on prioritized chunks of work, and allow quality to emerge without waste of legacy enterprise methods. Warning signs of dysfunctional organizations that result in Lean anti-patterns include:
- Organizations focusing on process instead of business value.
- Specialized teams focused on creating and managing enterprise documents.
- Backlogs of use cases managed by use case experts.
- Formal inspections of enterprise documents that result in grammar and punctuation corrections.
- Change-resistance enforced by rigid change control procedures.
- Generic use cases.
Work to eliminate these and set your enterprise up for successful product delivery.
[i] Implementing Lean Software Development: From Concept to Cash, by Mary and Tom Poppendieck
[ii] Link Borken
[iii] Just Enough Requirements Management, by Alan M. Davis, Dorset House, p. 172.
[iv] See Alan Shalloway, Lean Anti-Patterns and What to Do About Them
[v] Scaling Software Agility: Best Practices for Large Enterprises, by Dean Leffingwell, Addison-Wesley, p.93.
[vi] Out of the Crisis, by W. Edwards Deming, MIT CAES, p. 15-16.
[vii] Ambler (http://www.agilemodeling.com/artifacts/sequenceDiagram.htm)
[viii] User Stories Applied, by Mike Cohn (p.137-141)
[ix] See Beaver ???
[x] The Enterprise and Scrum, by Ken Schwaber (p. 70)
[xi] Private discussion with Alan Shalloway of NetObjectives, who promises a separate blog on Lean and teams of teams...
