Implementing Agile Approaches in the Public Sector

Agile methodologies are easy to implement in newly formed organizations where processes and procedures are changed every month. But in mature organizations that have processes and methodologies with long histories of success, there likely will be reluctance to adopt a change unless they think they’ll be sure to see big benefits.

Before getting into agile models, I am going to share a small story of success regarding implementation of agile methodologies in the public sector. In 2001, the FBI launched an initiative to adopt a software case management system, with the goal of replacing paper processes with digital records and automated workflows during investigations. By 2005, after spending $170 million, the project had died; mismanaged requirement analysis, poor design, ineffective development, and flawed project management were declared the major causes of failure.

The next year, the FBI announced a plan to develop a a new electronic case management system known as Sentinel. The project was contracted out to Lockheed Martin and budgeted at $305 million for the first two phases. In 2010, the project was only half done and $100 million over budget. The FBI tapped its CIO (and veteran Wall Street tech executive) Chad Fulgham to oversee Sentinel, and he was given full control of project resources and development. He switched Sentinel to an iterative Scrum approach that leveraged two-week sprints.

Fulgham used plan-driven processes where they were mandatory by law. For example, Congress was expecting a quarterly briefing on the project’s progress. Before briefing, Fulgham’s team was required to submit a brief document that followed the guidelines of the Office of Management and Budget (OMB). The team got the project under control, came in under the revamped $451 million budget, and, though it missed its initial September 2011 deadline, credits agile for getting the project done.

Federal IT evaluation guidance, such as IT investment management instruction and OMB IT reporting requirements, specify evaluations at key milestones and annually, which more closely aligns with traditional development methods. For example, for major projects, the OMB requires a monthly comparison of actual and planned cost and schedules, risk status, and performance measures.

If I were given an opportunity to implement agile while keeping existing plan-driven methodologies, I would look into one of three waterfall-Scrum models: waterfall on demand, waterfall on top, or waterfall on the edge.

Waterfall on Demand

In this model, the waterfall process is implemented at the end, typically around release time. You use agile throughout the software development process to build the software and get it completely done, then hand off to the team who processes the release items to use waterfall for release to production for governance inspection or any other internal process. If you need to keep a plan-driven methodology for regulatory compliance, this model is a good choice.

Adopting this new methodology requires an operational support model. This model facilitates releases from the user acceptance testing environment to prestaging, staging, and, finally, production, and is used to push items between QA, staging, and production.

A plan-driven model also requires a lot more documentation; you have to supply release notes, communication channels, etc. To facilitate it, you can include these items in your product backlog along the way and deliver it at any time. It is also not uncommon for spring teams to schedule a sprint toward the end of a project or process with the goal of releasing the software, getting it all the way out to production.

Another popular alternative to waterfall on demand is to, when scheduling a release sprint, add other teams to your meetings and make attendance mandatory. The operations team is made contributing members of the Scrum team for the last few iterations before the release cycle and joins the Scrum team for daily stand-ups.

Waterfall on Top

If you are required to produce a lot of paperwork before you can start delivering value, you’re in a waterfall on top situation—which is a painful process. To gain approval of the software, you have to produce tons of documentation up front.

A better method is putting the deliverables for the specification and documentation into your product backlog. This way you can use Scrum to produce documents needed for project approval and funding. Even if you have a long, linear document, you can still estimate the whole thing all at once for acceptance.

Recognize that this is not a rolling product backlog planning process, which means you will be investing a lot of upfront planning into work that may never actually get executed and will become waste. To avoid any surprises, you have to communicate everything to all stakeholders and, if possible, get acknowledgement from them.

When working in a plan-driven environment, you are usually given a large specification document. To make it more manageable, utilize agile practices by creating epics and themes to help arrange the product backlog. For example, in one project my team's velocity was relatively predictable when estimating stories of sizes 1, 2, or 3 points, but it didn’t have a great track record with stories sized at 5 points and above. To tackle this issue, any story estimated at 5 points or above was broken down further before it was added to a sprint.

Breaking up a large story has the added advantage that multiple smaller stories will travel the sprint board faster than a single large story. Though the same amount of work may be occurring in either case, you avoid the significant negative effect on a team's morale from a single dominating story sitting in the "In Progress" column for the majority of the sprint. An active board not only helps the team maintain a feeling of forward momentum, but also encourages stakeholders to take an interest in checking the board each morning to see the progress the team has made.

Waterfall on the Edge

When two or more teams are trying to deliver the same product at the same time but some are working with an agile model and others are using a plan-driven model, it is the most painful hybrid situation.

The challenge here is the culture; the simple truth is that these two types of teams need different things to keep them on track. A plan-driven team usually has hours-long meetings, while Scrum meetings occur more often but are short. A plan-driven team also does not implement thin slices of software functionality, and if any change is required, it reverts to the document that spawned at the project. That change is later tackled using appropriate change-control methods. On the other hand, an agile team looks at the software as the single source of truth about what’s going on in the project.

To implement waterfall on the edge effectively, the agile team should invite the plan-driven teams to be part of its agile process. The key is being as transparent as possible by putting information radiators up on the wall. The challenge is being careful to not be pushy about adopting agile, because it can create an arrogant posture toward the team that would ultimately decrease communication and lead to the downfall of agile within the organization.

The Key to Success

If your team exists in any of the above models, the key to success is making the commitment to adopt agile methodologies as much as possible. There are engineering practices, such as test-driven development and continuous integration, that you can use regardless of your methodology. We know that daily stand-ups and frequent opportunities to inspect and adapt make a more effective software development team. You can do those things even when you’re working in a plan-driven project, and you can demonstrate your progress by showing your software to stakeholders frequently, even if what you’re showing may not have the definition of done you’d like to see.

In my experience, information radiators can become central to an agile software development process for collocated teams. Most activities revolve around the task board, where the project team can see items moving across it on a daily basis. Keeping information radiators visible and current, even if by using a Gantt chart, is far more effective than not sharing information.

It’s also helpful to find small batches of work. Look for ways to deliver a small feature without implementing an entire layer within an application. If you are able to produce an initial specification document as a product backlog, then you can deliver thin slices of functionality over time once development starts.

Last but not least, even though we may need to provide metrics and status reports the way plan-driven teams often do, it won’t hurt to also deliver metrics that are expressed in more agile forms along with them. For example, you also could share your team’s velocity, and potentially even releases and burndowns. This will help other teams learn a bit more about how you do business and the value that agile can bring.

Converting Your Team to Agile

In the public sector, a change in standard processes and procedures requires significant effort and, often, approval from external vendors and elected officials as well as internal stakeholders. To get buy-in, you have to utilize all Scrum tools at your disposal to show the value of the proposed agile process. For example, adding product backlog items that represent deliverables—such as documents, checklists that operations need to take software into production, and any analysis activities, like architectural spikes—can help gain management support to adopt Scrum methodologies.

If you are not given enough power to introduce change on a broader level, you can find a way to make elements of your waterfall-structured plan more agile. For instance, instead of doing testing and development as two separate activities, you could propose that those teams come together and form Scrum teams that can deliver incrementally together.

Connecting existing software development practices defined by the prevailing public entity with agile methodologies can be a great way to get other teams to see agile’s value. Demonstrating how everyday processes can be improved is ultimately the best way to spread agile’s message throughout your organization.