Infrastructure Refactoring

At first glance, this may not appear to be a major issue, but very soon the team learns that the infrastructure is not well suited for Agile and changes must be made to it. For example, the test infrastructure may be hierarchical in nature due to its rigid entry criteria and, therefore, slowing down the process of migrating code from development to test. Another example is the build process is both slow and occurs only on a weekly basis producing numerous broken builds. The project team realizes they need either continuous builds or at least nightly builds to minimize the merging effort and reduce the rate of broken builds. Ultimately, the infrastructure should be reflection of the methodology being used.

As there is a focus to change areas to better align the infrastructure and associated processes to Agile, the project team must continue their development work to deliver functionality and business value per Agile principles. How can this team solve the problem whereby they must continue to deliver value, but must also make changes to their infrastructure to reduce problems and gain velocity? One suggestion is to utilize a newly termed approach called “infrastructure refactoring”.

As many folks know, within Agile there is the practice called refactoring. As Martin Fowler describes it, "Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a refactoring) does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring."

Similarly, when confronted with a stogy infrastructure for an existing product line which has not been aligned for Agile, infrastructure refactoring can provide similar advantages. Infrastructure refactoring initiates a "restructuring of the existing infrastructure". It really should occur in a series of small changes to preserve the overall integrity of the infrastructure and the existing release processes for the product line. When an infrastructure of an existing application is changed, the change must ensure that the product can continue its development without any significant downtime in any of the infrastructure pieces in use, therefore each refactoring should be small. The infrastructure must be kept working after each small refactoring, reducing the chances that an infrastructure can get broken during the restructuring and ergo injuring the ability to support the existing streams of development for the product.

The goal of infrastructure refactoring is to take the existing infrastructure that may be more complex and supports a phased approach and instead streamline and simplify it so that in supports a more incremental and agile approach.

Infrastructure Challenge

First, let's define what infrastructure means in the context of this article. Infrastructure refers to the technical structures such as environments, tools, and the processes that are used to support the product. Examples of infrastructure related components include the servers, network, and build tools used within the product development context. Infrastructure for a product typically gets defined and established in the first release of the new product. In the past, most project teams have used more traditional methodology approaches that follow a phased approach.

What happens if that product line wants to move away from a phased methodology to a move iterative methodology like Agile? Can the exact same infrastructure be used? To a great extend yes, but what becomes apparent is that many of the processes built into the infrastructure are built to support a more phased or waterfall approach. A refactoring change approach to infrastructure implies small continuous changes in order to reduce the chances of something going wrong, versus attempting to make all infrastructure changes at once with the potential of numerous problems occurring as a result.

What areas of change in the infrastructure must be considered when moving from a phased approach to an Agile approach? In following a project lifecycle, the infrastructure may require changes to the requirements, configuration management (CM), testing, development, architecture, database, release management, and other engineering spaces. In tackling this challenge, an approach that focuses on small continuous changes will be advantageous to ensure that the existing product maintenance and support continues while adjustments are made to the new development model. Infrastructure refactoring applies the Agile approach of continuous change and integration to improve an existing infrastructure while maintaining the integrity of the current support and maintenance line that must occur.

Strategize for Infrastructure Refactoring

Many Agile efforts get underway as new product development. This implies that there is no infrastructure in place so it is essentially an effort of establishing the infrastructure. However, what happens when a product already exists and therefore so does the infrastructure and then the methodology changes from a phase approach to an Agile approach?

This is where a re-engineering of the infrastructure may need to occur. Possible infrastructure changes to suit agile may include changing the use of an existing tool, introducing a new tool, changing how environments are used, changing existing process that impacts tools and environments, and even changing how infrastructure service provides engage with the project team. With this in mind, strategies should we considered to adapt our infrastructure for the agile methodology. The product owner should be the primary driver of this work but may assign a more technical representative from the project team to focus on the strategies. Once the strategies are in place, the execution of the strategy will be owned by the infrastructure team with the project team as the customer. More details on this to follow. Strategies to consider include:

Initiate an Iteration 0

The first strategy is to consider using an iteration 0 cycle to examine the changes needed and set a strategy on how to move forward. If the potential number of infrastructure changes that are involved (e.g., requirements, CM, testing, development, architecture, database, release management, and other engineering spaces) are significant, it is worth initiating an Iteration 0. Within this 1 or 2 week cycle, the goal is to better understand the number of changes and place them into a backlog, assess the impact of the changes, identify the dependencies of the changes to other areas, and then determine the work that needs to be done to get the changes to occur. The dependency identification will help in the prioritization process of what to tackle first, second, and so on. This is also the time to consider breaking the work into small sizes. This helps to establish an expectation of how much can be accomplished in each refactoring task.

Think in Iterations

Since refactoring implies a small series of changes, it’s beneficial to think of changes in relation to iterations. Another strategy is to use an Agile approach of short iterations (e.g., 1 week). Identify and assess a small enough chunk of work that can be managed within this timeframe. Near the end of the iteration, the project team user acceptance tests (UATs) the change per a standard Agile iteration. At the end of the iteration, the project team and product owner decides if the infrastructure change is acceptable and ready to go live into the working infrastructure and start being used by the project team. If not the change can be bundled with the deliverables of next iteration(s) and then released together.

Work the IPAIR Loop

The next strategy in approaching the changes is to consider the steps that will help you through an iteration focused on establishing or changing infrastructure. Because it is less likely that there will be a fully dedicated infrastructure team assigned due to the multi-tasking nature of infrastructure work, we want to keep the steps fairly simple and consistent. Also infrastructure professionals may not be into Agile as much as the project team, so a simpler variant of the traditional iteration is suggested. In this case, consider an IPAIR approach or something similar:

·         Identify changes and place in infrastructure backlog

·         Prioritize changes

·         Assess impact of changes

·         Iteratively make the changes

·         Re-assess impact of change seeking approve and release of changes

·         Begin the loop again (e.g., rinse, then repeat)

The steps within the iteration should be brief and simply enough to follow. The changes that are identified should be small enough that they can be made within the 1 week iteration. If you encounter a situation where you need more than one week, be flexible enough to expand the iteration timeframe. Also, you may find that if it is a larger change, that there may be a way to break up the change into smaller chunks of work to fit a short iteration.

Parallel Streams

The last strategy focuses on a model that includes two different groups who need to work together to adapt the infrastructure to the Agile method being used. The first group is the project team who is using Agile to deliver working software for the end-user (the ultimate customer who perceives the business value). The second group is the infrastructure team who improves the infrastructure. Interestingly, the project team becomes the customer for the infrastructure team, just as the end-users are the customers for the project team. Just like the standard practice where the customer should be part of the project team for continuous feedback, the same holds true that a member of the project team should be on the infrastructure team. This is important to understand because the project team is the group that must assess the value of the changes so that the infrastructure team knows which changes are perceived to be of most value and which to work on. 

The team agrees that changes must be made. They stop progress for an iteration and instead introduce an iteration 0 which simply focuses on infrastructure change strategy and identifying the changes. They identify several infrastructure personnel to form an infrastructure team. The project and infrastructure teams decide that iteration 0 will be 1 week and focus only on identifying infrastructure improvements.

Note that the task of forming an infrastructure team can be very challenging if the organization is not used to Agile and the iterative approach it follows. The services provided by infrastructure support functions typically align with a more phased or task driven approach. Another challenge is getting members of the infrastructure team to be fully dedicated since infrastructure work tends to be multi-tasking in nature.

During iteration 0, the project team discusses the various areas of infrastructure that are constraining productivity. They review the infrastructure (e.g., environments, tools, and working processes such as requirements, CM, testing, development, architecture, and database) and identify areas of improvement. Then they prioritize the improvement areas to best determine the value to project team (a.k.a., the customers of the infrastructure team). One area that has slowed them down is build management. Working with the infrastructure team, the project team takes the higher priority improvement areas and divides them up into small changes that represent no more than a one week iteration. In this case, the build management improvements are broken into: implementing a nightly build process (from the weekly build); setting up private work streams for the developers so changes can be built and checked into a private branch; evaluating a continuous build tool; and implementing the tool and supporting continuous build processes.

The duration for each task will vary depending on the number of people available in that week. They then assess the impact of the higher priority changes which includes determining the size of the change and the dependencies to other areas.Note that the relationship your company has between the project team and the infrastructure team will have an impact on how well they work together thru this process and their ability to adapt to the best practice for their working environment. Also per Agile principles, the more dedicated the infrastructure resources, the better the ability to utilize the iterative approach.

Once iteration 0 completes and there is a clear path for infrastructure improvement work, the project team can re-initiate their iterations of functionality development (aka, the project stream) while the infrastructure team begins work on the infrastructure improvements (aka, the infrastructure stream) also in iterations. Effectively both streams are happening in parallel.

As the infrastructure team makes changes (including testing by a member of the project team), they re-assess the impact of deploying the changes, revisit the project team (aka, the customer for the infrastructure functionality changes) seeking acceptance of the changes. If so, the infrastructure changes are deployed to the working environment at the end of the current iteration. If not, they work the IPAIR loop until the end of the next iteration and re-assess if the bundle of changes are ready. This will progress until such time that the infrastructure is now well suited for Agile.

Summary

When you find yourself in a position of inheriting an existing infrastructure and find that it is not well suited for Agile processes, consider the infrastructure refactoring approach. Introducing strategies such as iteration 0, thinking in iterations, a simple improvement loop, and running the infrastructure work in parallel with the project work may help you get to an infrastructure that is much more closely aligned with the project team's needs.

The infrastructure refactoring approach helps you focus on a series of small changes in order to preserve the overall integrity of the infrastructure and existing release processes. This reduces the risk of breaking the infrastructure therefore allowing those working in the current development stream, the maintenance stream, and the bug fix streams to continue to be productive, deploy changes to production, and ultimately deliver continuous value.

References

1. Refactoring Homepage " http://www.refactoring.com/ ", maintained by Martin Fowler, hosted by ThoughtWorks