“The CTO’s measure of success is that all teams see at least a 50 percent increase in velocity by sprint six.”
This was once told to a team I was coaching. It took perhaps ten seconds before someone was grinning from ear to ear. Even the team manager saw the flaw in the directive he’d just passed on. “Okay, so in sprint five we just need to make all our five-point stories thirteen-point stories, right?” joked one of the team members.
Welcome to Goodhart’s law in action—essentially, when a measure becomes a target, it ceases to be a good measure.
This is true of people metrics, too. Setting performance goals based on team metrics can dampen morale and even set teammate against teammate. Instead of promoting collaboration, it leads to cross-team competition and gaming of metrics to look better.
With the transparency of agile and the granularity of team-based metrics, it is extremely important to be responsible in how you use your measurements. There are five principles I adhere to when dealing with metrics: start collecting early and often, be consistent, stay focused, measure the project and the teams separately, and—most importantly—measure responsibly.
1. Start Collecting Early and Often
Start collecting metrics from the very first sprint. If possible, you want to capture how your teams were performing even before the agile transformation. When you’re already six sprints into an agile transformation before you start tracking your metrics, you’ve lost the opportunity to see where you came from.
When I signed on to coach a twelve-team organization, they were already three months into a rolling agile adoption. I was lucky that they had been entering data into their agile lifecycle tool. I continued to collect data as I observed the teams, so by the time I was three months in, I had a really good picture of each team. This gave me a place to work from as I dove into engaging with the teams. Already having trends and areas to focus on meant I was able to get some early and fast wins.
2. Be Consistent
This isn't so much of an issue with the agile projects themselves as it is with the tools we use to track measurements. While physical task boards can become cluttered, inconsistency is most often exacerbated by online tracking tools due to their customization abilities, which often results in different features being used in different ways from team to team.
When your teams cross over multiple managers or organizational structures, it can quickly become unwieldy to track the various metrics each team or department wants to use. This is especially true when using any of the larger, enterprise-scale agile lifecycle tools that allow easy customization.
You need to have some common, agreed-upon measures for all teams and organizations. My recommendations are cycle time, escaped defect rate, planned-to-done ratio, and a team happiness metric. Even across a broad and diverse section, these are some consistently understood and measured criteria. A good team-level assessment tool is also helpful here.
For example, I was working with teams using both Scrum and kanban. In order to have a consistent view of metrics (for trending, not for measuring teams against teams), we created artificial timeboxes for tracking the teams’ performance. The kanban teams reported metrics in monthly increments, the Scrum teams reported in two-week increments, and the metrics charts calculated everything by week. So if a kanban team had four escaped defects per iteration and a Scrum team had two per iteration, they both were trending with one defect per week.
3. Stay Focused
Our brains really only work well with about three to five benchmarks. If you're tracking ten different metrics, odds are a good five of them are not getting the right attention or even being tracked correctly.
One of the pillars of good agile and lean development is focusing. We need to do the same thing with our measurements. Measure what matters the most, get it working well, and, only if it’s needed, then shift focus to measure something else.
4. Measure the Project and the Teams Separately
This is one of those times when focus sometimes can look like a lack of focus. Just because a project is going really well does not mean the teams working on the project are in good shape, or that your agile adoption is sustainable.
In my old command-and-control project management days, I used to say that any project manager could ship a product on time, on budget, and in scope—once. Your project can be in awesome shape, only to be in trouble when half the team quits after the launch. Keep your project and people measurements separate from each other. Failure to do so runs the risk of corrupting all your data and wasting all your measurement efforts.
I recommend setting up three separate focuses: team-based metrics, project-tracking metrics (to monitor progress), and agile health metrics (to track the adoption itself).
5. Measure Responsibly
With all this data, it’s essential that the information be used responsibly. Here are some key points to remember:
- It is a bad idea to measure teams against each other. Don't try.
- Use supporting metrics so that if one metric is gamed, another will reflect it.
- When measuring project success, stick to customer-focused metrics, such as Net Promoter.
- Don't use team estimates to forecast project timelines. Use statistical forecasting.
- Metrics are a lagging indicator. Use them to find better ways to do things in the future, not to punish the past.
Simply having metrics in place is not enough. You need a plan for how you will use them, and you need to constantly check to make sure you haven't mixed up or polluted how you collect, monitor, or use that data.
