In a previous article addressing the challenges of Enterprise Requirements Management, it was suggested that the legacy enterprise organization requires restructuring so that productivity gains promised by Agile methods can take root and grow. At the same time, there is a growing chorus of IT skeptics who are singing about the ineffectiveness of the CIO. Legacy enterprise organizations struggle to stay coupled with business drivers, with the most collaborative relationship occurring at the CIO-to-peer level.
In this article, the structure of the misaligned IT organization is revealed as process-centric silos which have created an ever-widening chasm with business clients that the enterprise organization is supposed to serve.This chasm is best bridged using Lean and Agile methods. Enterprise agility begins with building trust by rapidly creating business value using cross-functional teams that can be scaled upward. Agile barriers and enablers are presented to allow CIOs to assess their organization's potential for restructuring and adapting to Agile.
Introduction
The legacy IT enterprise organization is a structure consisting of silos of technologists that rarely collaborate with business units (see Figure 1). This organization pattern of silos has been called the "Technology Organization" by Kirk Knoernschild in an excellent article on the Agile Matrix. These silos come about from good intentions as well-meaning process experts attempt to decompose software development into repeatable sub-processes. Once specialized process (such as requirements management or object model analysis) becomes the focus of an organization, time-to-market begins to slow, and survival becomes management's highest priority within the silo. Project teams become matrix units where specialists from each area participate in virtual project teams and, in the truly dysfunctional case, are assigned to multiple projects at the same time. The management role becomes a specialization in maximizing the project load for each resource in the silo. Organizations with this structure can manage waterfall projects successfully, but speed and change-tolerance suffers. Symptoms of such organizations are manifest as well-formed change control structures emerge to protect the already over-burdened resources from unplanned and unexpected work. Managers of these resources focus on protecting their resources from "the business," and are seen as good managers if they can "manage their business clients."
The Legacy Enterprise Organization: Yearly Planning and the Origination of the Death March
Sometime in the middle of the summer, the enterprise project cycle typically begins. Struggling project managers are asked to review and "SWAG" project briefs for next year's project list.[3] This up-front planning effort is clearly a Lean anti-pattern, and violates the principle of eliminating waste, since up-front specification is wasteful in software development. Even worse, this activity is a clear distraction from current project work at best, and at worst, it commits an entire organization's resources to next year's activities with no information other than short summaries of capabilities and business cases. The yearly planning cycle fixes scope, resources, and delivery date. Experienced Agile practitioners will attest to the variation in estimates for just 10-day cycles. With this basic truth of technical product development established, it borders on absurd to commit millions of dollars of resource capacity to long range software development projects with fixed scope and no opportunity to allow discoveries to feedback into the original plan. Instead "discoveries" are treated as planning errors.
The enterprise project planning effort is led by the silo of requirements experts. These highly trained business systems analysts represent an organizational attempt to enhance communication between heavily multi-tasked development teams, and the business clients that they serve. This particular silo has a weakness that is difficult to recognize, particularly in long project delivery cycles that result in infrequent feedback to the overall enterprise. That weakness is, bluntly, that the requirements experts ultimately have no skin in the game. By the time the long-lived project effort is completed, the originators of the requirements maze have long moved on to other projects, leaving the implementers to deal with backlogs efficiently gathered requirement specifications.
The Art of "Thrashing"
As the project planning cycle progresses, attempts must be made to maximize the number of projects each resource can simultaneously handle. This is a fundamental flaw of prioritizing projects, as opposed to individual business needs, and leads to the brittle state of the organization, which is poorly equipped to make changes based on business value and changing markets. The yearly budget cycle forces unproven efforts to predict precisely what market forces will occur over the next 18 months, and even more challenging are the technology changes that are anticipated over the same long period. The brittleness and change-intolerant state of the legacy enterprise is demonstrated by the rigorous change control boards that are in place to protect the over-committed project teams from more thrashing.
As project portfolio planning unfolds, the legacy waterfall life-cycle begins. Teams struggle to understand requirements during an analysis phase, and a backed-up queue appears with issues to be tracked to closure as analysis models are instantiated to vaguely represent the team's understanding of the requirements specifications. In order to track progress, well-meaning project managers report status on analysis models while the team attempts to interpret the system to be built. These analysis models are typically reviewed by the client in a formal inspection, but real progress in architecture and design cannot truly be made until implementation, when the real limitations and constraints are uncovered. The enterprise reacts to this dilemma with a silo of architects who attempt to enforce compliance based on last year's architecture mistakes. These groups typically are conservative and result in legacy code and applications existing well past there useful state, creating untestable code and infrastructures that further complicate the team's ability to deliver quality technical solutions that are unconstrained by infrastructure limitations. Legacy enterprise architecture silos tend to become ivory towers of idealists, who over time, lose collaboration with the project teams struggling to meet business needs while being forced to meet architecture requirements.
The Promise of Agile
The promise of Agile is that collaborative teams empowered to deliver business value are free to replace wasteful, over-engineered designs with working applications, backed up with automated testing, and fueled by rapid feedback from business clients and rapid technical validation. Well-formed Agile teams empowered to refactor at the appropriate time, can rapidly build trust from their clients without large up-front design efforts. Productivity gains expected from Agile are well established and representative expected metric gains are shown in Figure 2.[4] The next thrust for Agile evolution is now in transforming organizations to a scaled Agile structure.
Transition Roadmap: Building Trust
The transition to Agile from a legacy enterprise organization begins with building trust and bridging the chasm that the organization has opened with its business clients. This bridge is best built starting with a successful pilot that regularly delivers business value. In order to bring the correct sense of urgency to the pilot, a highly visible, high business value effort should be selected. This requires courage, since new approaches are typically piloted on low-risk, low-visibility efforts to minimize the impact of failure. To minimize the real risk, the pilot should pull no punches, and create a fully-functional Agile team with all key skilled resources collocated in a collaborative environment. Pilot teams should be trained by seasoned Agile coaches, and set up with a stable pipeline of prioritized work, staged with the help of Agile consultants experienced in organizational transformation and turning backlogs of user stories into working product. Scrum can provide a solid framework that gives process-focused organizations a starting foundation of implementing a lightweight method for agility. Organizational learning can be amplified by piloting multiple teams simultaneously, pulling work from a common backlog and sharing retrospectives after each iteration.
Once the team has been selected, trained, and collocated, expectations must be set regarding the group becoming a real team. Every effort should be made to create a highly collaborative environment, and this can be aided by focusing the team on as few tasks as possible. This is all established in the daily stand-up meeting, where the team self-organizes to close tasks and stories prioritized by the business and made visible by the leader (ScrumMaster). It is not unusual for a seasoned project manager, new to Agile, to try and assign as many tasks as possible in an effort to divide and conquer, but this sets the team up to work as individuals instead of together as a team. Using Agile status tracking artifacts such as the burn-down and burn-up charts helps expose this behavior.[5] Make productivity and quality visible by updating frequent counts of completed stories, and any other key metrics required by the organization. By keeping these metrics visible the team gets a constant reminder, and this is another key focus topic for the daily stand-up.
The real success driver of the pilot will be the pleased business client who gets incremental delivery at regular intervals. The higher up this key client is in the business organization, the better exposure for the Agile team. Ideally, the business client has frequent interaction with the team, and helps validate stories as they close. Once this pattern is established, the focus becomes the next story in the prioritized backlog, and change tolerance becomes normal as product manager has the freedom to change priorities of work outside the sprint. Keeping the effort tightly coupled with changing business needs is the tangible difference that will draw the right attention to the pilot, and ensure its success is contagious throughout the organization. New Agile pilots typically expose legacy enterprise silos as barriers to fast delivery. The CIO should expect this tension and be ready to assist in changing behaviors and expectations of the existing organization.
With success, the CIO should begin to focus on scaling up the Agile pilot successes. An established approach is to evenly split the successful pilot team, and bring in new resources to work with the experienced Agile resources on each of the two new teams. The teams themselves should ideally hold "try-outs" for the new members to ensure that they are good fits for the established Agile team. This also builds trust with the teams that are already empowered to deliver business value using their own capacity. This is the ideal time to begin pulling resources from the silos, and dispersing them into Agile teams, creating an organic reorganization. Specialist skill sets can now be maintained by creating a new matrix organization that regularly pulls specialists together in order to synchronize how they implement their skills within the Agile teams. As collaborative Agile teams mature, cross-training naturally occurs, and an organization emerges that can adapt to changing staffing needs. This is the same "Agile Matrix" described in an earlier referenced article is shown in Figure 3.
Summary
This article has presented characteristics and behaviors of the legacy enterprise organization, summarized here.
Warning signs and behaviors of legacy enterprise organizations:
- Projects getting larger and slower.
- Silos of specialists who are multi-tasked with multiple projects, who receive and create hand-offs.
- Projects demonstrating high cost at commitment, brought about by detailed requirements gathering.
- Organizations focused on resource loading instead of product development.
- Separate production support groups that have little or no coupling with development teams.
- Project teams that demonstrate simplest first features due to lack of prioritization.
- Project managers requesting status tracking tools to ease burden of status reporting.
An Agile pilot approach to build trust and lay the foundation for a transformation of the organization has been suggested, along with successful strategies for setting an Agile pilot up for success. This approach should give the CIO a roadmap for creating an IT organization that is tightly coupled with business vision, and not distracted and slowed by over-engineering which results from silos of technical and process specialists, operating in a process-focused, non-collaborative environment.
[1] John Soat, "The Evolution of the CIO", Information Week, http://www.informationweek.com/story/showArticle.jhtml?articleID=203101647
[2] Bob Evans, "The Inexorable Rise Of The New CIO", Information Week, http://www.informationweek.com/story/showArticle.jhtml?articleID=203101599
[3] Scientific Wild A Guess (the reader is left with homework to discover what "A" represents)
[4] Jeff Sutherland, "Hyperproductivity in the First Scrum"
[5] See "The Agile-V Scorecard"
