Product portfolio management has become an essential discipline for all development organizations that want to achieve enterprise agility. The repeated process of selecting, sizing, and prioritizing the work to be done ensures that their development teams are delivering the most valuable products and enhancements for the business’ clients. This is required for both external clients in the case of product companies and for internal clients in the case of IT organizations. However, the subject of this paper is another, possibly even more important, reason: avoiding the overloading of the organization’s development teams which greatly lowers their efficiency.
This article will discuss: 1) how overloading teams significantly degrades their performance, 2) a principles that can guide the sizing of the features coming out of your product portfolio management process, and 3) why product portfolio management is a critical component of a transition to agile methods.
Effectiveness and Efficiency
Agile organizations need to be both effective and efficient. Effectiveness is doing the right thing and efficiency is doing it without wasted effort. Being inefficient, even if working on the right thing, is like trying to pile boxes up to climb over a wall when a ladder is available. Efficiency without effectiveness is like quickly climbing the ladder that is placed on the wrong wall. Agile organizations have to look both at selecting the right work for teams to do and then using appropriate methods to get teams to do their development faster. We can think of this as a two step processes—selecting and developing, as depicted in Figure 1. Done properly, we call this “fast-flexible-flow” because we can readily select what is needed (effectiveness) and development it quickly (efficiently).
Figure 1. The flow of selected products to the development teams
Also, illustrated in Figure 1, isthe relationship between these two parts of the development process. Consider what happens when too many things come out of the pipeline on the left, forcing their way into the pipeline on the right. Or what happens if these selected items are very large. In both cases, the development pipeline jams up. To achieve flow across the entire pipeline, the organization has to not only prioritize the proper features, but size them appropriately and only allow a certain number to reach the teams at any one time.Inducing Work
Thanks to the spread of Lean-Thinking, software development organizations are talking more and more about the need to “eliminate waste.” It is good to get rid of waste. It may be even better not to create it in the first place! Unfortunately, software development organizations create new work for themselves without even realizing it. How does this happen? Consider Figure 2,showing two ditch-diggers, one making work for the other.
Figure 2. Creating work due to a bad process
One digger is throwing his dirt into the other man’s hole. Now the second man has to shovel that out in addition to his own dirt. Their system of coordination (or lack of it) has created new work—double the work—for the poor guy! And once created, the work must be done. I call this “induced” work, because the way they are working together is causing it to occur.
Is it absurd? Maybe. You would think the second guy would quickly see what was happening, turn around and change how they are working. This is because in the physical world it is readily apparent what causes induced work. In software development, the act of inducing work is not so apparent and therefore, is more insidious and more difficult to stop. Furthermore, what we often look at to guide our process often increases our induced work under the guise of making us more productive.
To quit shoveling work into each other’s ditches, we have to look at new kinds of measures. Rather than defining efficiency by “productivity,” we have to look at “time.” Consider the following software related example. In an attempt to make themselves more efficient, a testing group decided to batch up the code being tested. The assumption was that this focus would increase their efficiency and therefore, their productivity. The result, however, is that developers subsequently had to wait days or weeks longer to be told that they made an error. These delays made the developers less efficient and lowered the quality of the code they were delivering to be tested. Consider the following conversation I’ve had numerous times with developers.
Me: How many of you would say you spend most of their time fixing bugs?
Developers: (Almost all raise their hands).
Me: Consider what happens if you write a bug and are immediately told there is an error. How long does that take you to fix?
Developer: Not much at all.
Me: Now, consider the same error. But this time, you are not told about it for a week. And imagine nothing else has changed. How long would it take now to fix it?
Developer: Obviously, much more time. And, if something has changed, if someone else has been working on the code, it will take a lot more time! There would be so many changes to have to wade through.
Me: The fix was the same in both cases, but now there is extra work. The delay, from error until detection, caused this extra work. I call this induced work because it was created by the delay.
If you agree with the developers I have talked with, that they spend significant amounts of their time fixing bugs, then surely the amount of induced work we face is significant. That is a lot of dirt!
The Common Problem: Delay
In software development, there are many different types of induced work; however, they all have a common element: delay. The most common delays are the time between when information is needed and when it is used and between when an error is made and when it is discovered. And they are made worse by delays in getting information needed to act. The primary cause of delay is doing more than one thing at a time. The math is simple. Doing two things at once doesn’t take twice as much long; it takes twice as much time plus the time to do the newly created (induced) work. Accounting for waiting, detecting, remembering, stopping and starting, the resulting time can be significantly longer.
But, you say, “We need all of this work done. We can’t afford to do them one at a time!” I have heard this justification time and again. All of the work items are critical. The development teams need to work on all of them. But what we need and what is reality are often different. Imagine being a pilot of a two-engine airplane. The co-pilot says, “Captain, number 1 and number 2 engines are out.” You respond, “That can’t be! I need those engines on!” What you need and what reality is are two different things. The same thing is true with our work; what we need done and what is possible are two different things. To create the greatest chance of getting what we need done, it is critical to avoid creating new work that does not add value.
Focusing on Productivity Actually Works Against Us
Can we just make developers more productive? If they just worked smarter not harder, wouldn’t they be able to get it all done? We have all tried this. It does not work. On the contrary, going for increased productivity directly usually results in decreased productivity. Why? Because keeping people productive means keeping them busy, even if that means starting new tasks before completing old ones. This creates multiple things being working on simultaneously and that means induced work (think back to Figure 1).
Let’s see how this happens. Imagine a development team is working on a project and needs some information. They want to ask questions of an analyst but find they have to wait to get them answered. This happens because the analyst, wanting to stay productive, has started working with another team. The developers just sigh, go back and decide, well, in order for them to be productive (busy), they’ll have to work on something else too. Since they can’t continue with their original project they start working on a new feature. Before long, everybody is doing several things at once, each waiting for the other. It gets so bad that people stop trying to talk to each other and instead start using emails to communicate. And so it goes, worse and worse.
It is a horrible situation and all too common. We have all been there. We are incredibly busy but do not achieve real value compared with our efforts. We fall into this trap because the goal of productivity does not provide us insights into achieving it. Poor decisions are made because we take actions that increase local productivity—keeping each person working—but ignore the big picture—getting value delivered to our customers.
The Software Development Value Stream—A Necessary Component of Increasing True Productivity
This value stream is the flow of work from its conception to its consumption. Figure 3 illustrates this, showing the roles of the customer, business, management and development team.
Figure 3: A conceptual view of the software Development Value stream
Also illustrated in Figure 3 is we must go to our customers (internal or external) to see what they need. Business sponsors need to select, size, and prioritize those features that are going to be created. By creating the smallest features that are still useful from a customer’s perspective and viable for the business to deploy, product managers can provide work assignments that teams can quickly build and deploy[1]. Teams, for their part, must learn how to develop in smaller steps than they may have been used to. Management’s job is to facilitate how teams’ interpret the business/customer need, provide visibility to the business of what is being accomplished and to improve the organization so that impediments to the value stream are removed. This combination of business, management and teams is required for agility at the enterprise level.
Cycle Time: The Key Metric
The value stream provides us with a better metric to guide our decisions than productivity levels which may actually work against us. One way to lower induced work is to remove the delays that cause it. Reducing delays will reduce the time it takes for our work to flow from conception to consumption. We call this the value stream’s cycle time. Shortening our cycle time will necessarily shorten the delays between the work being done. This makes cycle time the key metric to focus on to see how efficient our process is. The value stream highlights where induced work is occurring by making the delays between our work visible. These delays show up as bottlenecks and delays. Looking at cycle time helps you understand when features are too big or when there are too many in the pipeline. Properly managing the size and number in your portfolio will have a significant impact on your cycle time.
Starting an Agile Transition Without Considering Product Portfolio Management
The common mantra in the Agile world today is to start a pilot project—focused on creating team agility. The intention is to scale this process throughout the entire organization after one or two teams have learned it. In many organizations, however, the real problem isn’t so much the teams’ process as much as it is too much work being handed to the teams which keeps them working in a haphazard and inefficient fashion. Oddly enough, this team-pilot approach often has great initial success but is not able to scale to the organization. This occurs because the team selected for the pilot projects is told to work on one project. They achieve much greater productivity mostly because they have created the team to be focused on one-project, are often co-located, and have all of the skills they need to do this on one team. We’ve seen these factors have just as much impact on a team’s efficiency than agile methods.
Of course, they may not truly understand this and likely will attribute it to their new process. As they add more agile teams, the rest of the organization actually gets worse and worse. Eventually, the local improvements of the agile teams is more than balanced out by the increasing burden placed on the other teams. Some point to this as evidence that agility works since the agile teams are doing well—even if the organization is not improving. Ironically, the more the teams as a whole are being overloaded (that is, the more the need for proper portfolio management) the better the pilots will appear compared to the other teams. This is one of the key reasons agile pilots work but then agility at the enterprise level fails.
Ken Schwaber, co-creator of Scrum, has said “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.” While this number has been generally agreed to among software consultants, the reason why is not. I was initially going to say “hotly debated” but in fact, most agilists haven’t investigated the cause for this other than the immediate assumption that the failure occurs due to lack of motivation or discipline within the organization.
Our experience is that team-focused agile methods don’t provide the insights required to remove the organizational impediments that they face. A more holistic, value-stream oriented view geared toward shortening cycle-time provides these insights. Scaling often fails because the root cause of the impediments in the value stream are not observed, let alone dealt with. Agility can be achieved across an organization when the entire value stream is considered—not merely for a lucky team or two selected for the pilot project.
[1] Note: Features of this type are often called Minimal Marketable or Minimal Viable Features.
