If Agile is going to make a difference to an organization, it must accomplish two things. First, it must assist us in being driven by business needs—not the development organization. Second, it must help us with the entire value stream—not merely part of it. Lean-agile practices presents us with an opportunity to reunite the business and software development organizations so our Application Lifecycle Management (ALM) can focus on value, not merely delivered software.
"We can't solve problems by using the same kind of thinking we used when we created them." – Albert Einstein
If Agile is going to make a difference to an organization, it must accomplish two things. First, it must assist us in being driven by business needs—not the development organization. Second, it must help us with the entire value stream—not merely part of it. All too many organizations start and achieve seeming success with team centric pilot projects. Yet, many of these do not have a positive impact to the bottom line of the organization. Lean can assist Agile methods by providing a broader perspective in what needs to be looked at—both in the value stream and in how to achieve the transition itself. Thus we must turn scaling agility on its head and think about agility at scale right from the beginning with Lean-Agile.
It is often forgotten that software, by itself, is useless. Software is only of value in how it provides value to those using it. Unfortunately, we see violations of this simple principle over and over again. It can be argued that the mere separation of business from IT is an example of this. Why do we have two organizations when there is only one business? In product companies, the situation is somewhat better, but the problem is still mostly there. While Agile methods have greatly enhanced the capabilities of teams, it is time now for Lean thinking to provide insights of how to attend to the entire value stream in order to maximize the value of software development delivered. This presents us with an opportunity to reunite the business and software development organizations so our Application Lifecycle Management (ALM) can focus on value, not merely software, delivered.
For each success you hear in the Agile world, it seems there are a few others that don’t reach as high. We must learn why some Agile projects succeed while others fail. Does failure occur merely because the organization does not have the wherewithal to become Agile? Or is there something wrong with Agile methods that do not allow them to be successful under certain circumstances? I think these are important questions to delve into. My personal story speaks to the dangers of thinking the success you had is the success others need. I formed Net Objectives in 1999 as an organization devoted to assisting developers in better design and coding methods. This was where I had the greatest experience—and success—almost 30 years as a developer, software architect and C-level of small software product and IT organizations. Being a consultant, however, gave me a different perspective. Instead of seeing only 1-2 projects a year, I now saw dozens. This enabled me to see the challenges of a large variety of organizations.
This broader perspective quickly taught us that although the drama was often with the developers and testers when the deadline loomed the real problems for many organization lay outside of this area. Over the years I’ve progressed from focusing on the development team to looking at how requirements were created to looking at the testing organization, to project management, to product portfolio management, to deployment and support. Each year seemed to extend my view of the value stream up towards the business side and down towards deployment and support—finding new challenges at each step of the way. This broad view taught me one irrefutable truth—the major blockage in your development organization can occur anywhere—and most methods, e.g., design patterns, XP, Scrum, even Kanban, only deal with part of your value stream. It also emphasized that when different parts of the value stream focus on different objectives, the value delivered to the customer is less than it could have been.
Insights from Lean
I observed this phenomenon because we kept finding teams that our previously successful methods could not help. As time went on, however, I saw that a few principles of Lean, as I learned them in a deeper way, seemed to always apply and to always provide insights into how to overcome these disparate challenges. The primary insights were:
- Look to see how to shorten the cycle time (time from idea concept to customer consumption)
- Be wary of improvements achieved by focusing on one area—they could adversely affect other areas—that is, one must look at optimizing the whole, not one section
- Be aware that the source of a problem may occur well before where the problem is observed
- Attend to time, not productivity (see my blog Learning to Manage What Matters – Not Always Intuitive)
- Eliminating delays between steps improves productivity and quality
- Conversely, delays between steps actually causes more work to be done
- Higher quality can be achieved by doing the right thing at the right time with the right people without adding to the amount of work to be done
- Close knit teams do not work in the same way as individuals from different teams trying to work together
- Managing how much work you have at any one step can achieve great results in quality and cost
- The amount of work given to a development group can affect their efficiency more than anything else
- The system that teams work in (e.g., are they co-located, do they have easy access to others they need) has a significant impact on them—often stifling their abilities and skills
This article is about using Lean principles, attitudes and mindsets to discover where your challenges are, and what you need to attend to solve them. It is about integrating the different steps in your methods so that all of those involved can keep their eye on the goal—value achieved by the customer. The times of easy improvements achieved by teams that embraced Agile and focused on delivering value iteratively are over. As Agile has crossed the chasm, these easy situations have most likely been encountered and solved. As we’ve jumped across the chasm, we have found situations where the standard team-based agile methods do not work nearly as well or at all. The Agile community’s focus on team-based Agile methods has been met with frustration by many and is creating a backlash against Agile in general. It is important that we not ascribe flaws to these later adopters and casually explain away their challenges. We must realize that methods that have worked in certain situations just do not work in others.
Using Lean to Guide Agile Adoption
What we need is a way to be agile that attends to the entire organization. How do we do this? I believe the Lean insights I provided earlier, which I often call “Lean-Science” provides significant guidance. Lean, fortunately, provides us with much more than this. Lean goes beyond merely suggesting we attend to certain ‘laws’ of development. It also suggests ways people best learn. When transitioning people to new methods, most of us have observed pushing people too far can have an adverse effect. It’s not just that people stop improving, they actually get worse in many ways. Lean suggests a model of continuous improvement—of taking small learning steps to both learn better and keep the emotional aspect of change as a positive force.Lean thus broadens our horizons in another way—that of suggesting we control the transition to new methods. Lean suggests we start where we are and respect the way people are currently working. We want to establish a method of continuous improvement—not just dramatically change our process and hope the team moves forward. This is a great difference between Kanban and Scrum at the team level. Scrum suggests throwing people into a new structure and organization that we know works. We assign new roles (ScrumMaster and Product Owner), we provide new work methods and timeframes. This dramatic jump often works. But in larger organizations, it seems to throw many people into chaos and fear—conditions not conducive to learning.
Kanban suggests a better learning approach is one of continuous process improvement. To do this requires two things. First, the policies of the team must be explicitly stated. Explicit policies merely means how the team works is explicitly discussed by the team and expressed to management. It does not mean writing them down or making them hard to change. The point of explicit policies is so that we can see 1) that we are doing what we say we think is the best way, and 2) if what we are doing is working.
These beliefs manifest themselves in Kanban development by having a Kanban board that reflects the process the team has been explicitly discussing amongst themselves while also making management (leadership) aware of how the team is working. People don’t follow the board, rather the board reflects the team’s explicitly stated best way of doing their work—in a sense, the board follows the team. This enables the team to try different things and see if these new ideas work.
A team starts Kanban by observing how they are currently working and then decides how they believe their work might be improved. While many people think Kanban is mostly about limiting work in progress (WIP) or doing Lean-Flow it is really about providing this transition management system. That is, it provides companies a method of managing the change in their organization to give an alternative to just abandoning their current methods and hoping for the best. It also enables us to start where we are. We don’t need to work on just one team and possibly adversely affect others. I believe this is a useful transition method in most places. However, it is absolutely essential where the creation of teams due to highly specialized people prevents effective team building across the development organization.
Lean therefore provides a more holistic approach in two dimensions. First, Lean suggests optimizing the whole—that is, looking at the entire process from idea initialization through consumption by the customer. The second one provides us with a method of managing how we introduce improvements and attending to the rate of change that is optimal for the organization making the transition.
Transitioning to Agile With Lean to Guide Us
Thus, Lean arms us with:
- A foundation of respecting people
- A scope of looking at the entire system
- An awareness that we must attend to the emotional and learning aspects of a transition to Agile methods
- A focus on time
We can combine these principles and insights of Lean to give us a roadmap for our Lean-Agile transition. Like any roadmap, we need to know:
- Where we are
- Where we want to go
- What our means are to get there
- What our suggested path is
Let’s see how Lean provides us with insights on each of these.
Where we are: In Net Objectives’ experience in transforming medium (300-500) and large (1000+) development groups to Lean-Agile, we have seen the following major challenges facing an organization:
- Business stakeholders not having clarity on what will add the greatest value to the customers of the development organization understanding
- Teams not knowing how to coordinate with each other
- Too many projects being dumped on the teams
- Teams not knowing how to deliver value incrementally
- Not being able to create well-formed teams across the organization
The prevalent assumption that the teams’ development methods are mostly always the primary problem is not founded in our experience. Lean’s analysis of the value stream helps you learn where to start, because it’ll tell you where you are.
Where We Want to Go: There are several challenges in transitioning an organization. First, everyone needs to be on the same page as to both where and why they are trying to improve. If there are disparate views, different parts of the organization will actually work against each other—often without even being aware of this. Lean provides a common vision—our goal is speeding time to market. While this may appear obvious, I would suggest that most organizations do not actually live like it is true. The following are common attitudes we find that are evidence that people are not looking at the big picture of shortening cycle time:
- Product portfolio management is merely looking at what’s needed and not considering the loads (and therefore possible inefficiencies) they are putting on their teams
- Developers are just trying to get the code done
- Testers are just trying to get the code tested
- No one is paying attention to what the costs of integration, deployment and support are except for those doing those jobs
What Our Means Are to Get There: So how does Lean help us achieve our goals? Let’s look at the common problems we mentioned earlier.
Business Stakeholders not Understanding What They Need Created: This results in too many and too large projects which overwhelm the teams. Lean adoption will suggest we need to do product portfolio management to alleviate this challenge. More importantly, Lean suggests that what is developed must add value to the customers, not merely be software with great features. Adding value to customers means to focus on their operational and commercial value streams. This gives us a way to connect our software development efforts to our customers (whether they be internal as in IT organizations, or external as in product companies).
Teams Not Knowing How to Work Together: For years I’ve worked with many teams and lamented the ineffectiveness of Scrum-of-Scrums. Coordinating the work of agile teams that are dependent upon each other or who are working on different parts of the same feature is particularly difficult when the organization does not have a really good method of coordination already in place. While Lean focuses on cycle-time, it can also be used to think about the time until you get useful feedback. When teams need to coordinate together, one level o useful feedback is when they integrate their work. If integration takes place after all of the teams have completed their work on the feature in question, this work may take a considerable amount of effort—merging branches as well as larger integration tests. Lean suggests giving smaller pieces to the team to get them done more quickly.
Inspired by Lean’s focus on time, in particular, getting feedback quickly, I have seen that selecting stories and then giving the appropriate fragments of these stories to the appropriate teams, speeds up integration and greatly reduces the need for coordination. The details of doing this is beyond the scope of this article but will be presented in December’s edition of Agile Journal.
Too Many Projects Being Dumped on the Teams: This can be addressed by limiting how many projects are in play at once. Even if Scrum is used, limiting work in progress at this project level is still a very powerful method.
Teams Not Knowing How to Deliver Value in Small Increments: At the team level, either Scrum or Kanban may be effective. We should chose which one we want based on the ability to form teams as well as the ability for team members to adjust to change. If Scrum is chosen, it is critical that many of the practices that Kanban suggests (e.g., visibility, explicit policies, limiting work in progress) be incorporated to enable all parts of the value stream to work better together. The interested reader should refer to our Practices All Scrum Teams Should Follow.
Not Being Able to Create Well-Formed Teams across the Organization: Until Kanban, all Agile methods were organized around teams being formed. While Kanban does not require teams, it can be practiced when teams do not exist. This is very useful for those organizations that have people who have specialized skills, expert domain knowledge or are familiar with certain parts of the company’s legacy code and need to be shared across projects. In these situations, forming teams may just not be possible. The mandate of cross-functional teams, while ideal, may just not be possible. It is not possible to easily cross-train someone to learn what a person knows who was around 10-15 years ago when a legacy system was built. Nor is it easy to replicate a person who is a specialist in stress testing airplane wings that requires a Ph.D. and eight years experience. People like these typically need to be available to multiple teams, making the Scrum model inapplicable.
What Our Suggested Path Is: It is very tempting to begin your transition to agile methods with a pilot project. Most every company has at least one or two projects that could benefit from doing so. However, I have seen many organizations start their road on agile by merely starting a pilot project without regards to what impact it has on the rest of the organization. This often appears to work, but is fraught with danger. See my blog How successful pilots can hurt the organization. Our path becomes one of looking at the entire organization. Focusing on adding value and looking to see where we need to start and while keeping an eye on where we want to go. While we may start a pilot project under Lean’s auspices, we’ll do it while considering the entire value stream. Any changes we make will attend to how they impact the broader picture.
Summary
The agile community should be proud of its accomplishments over the last decade. Agile is clearly accepted as the model for teams building software. The challenges in scaling agility, however, have shown us that a bigger picture view, afforded by Lean, is necessary. Lean provides insights that can align all components of the development organization—from the business drivers through support. It also gives us insights into how to achieve continuous process improvement and to introduce agile methods in organizations that might not be able to easily accommodate the structural requirements of currently popular methods. I fully expect that this combination of Lean and Agile will become much more widespread over the next few years.
