Kent McDonald writes on how to manage business analysts without measuring them. You can do so if you view management as helping business analysts improve their skill sets while helping them be productive members of their team. If, however, you view business analysts as “resources,” you will more than likely find individual measurements quite useful.
There are many people who believe you can't manage what you don't measure. That thought carries over to the people working on delivery teams, especially if the organization tends to refer to people working on projects as "resources." As a result, I get asked quite frequently how business analysts can be measured. There is not a clear cut answer to this quandary, but I wanted to take a shot at answering it with a combination of my thoughts and the observations of others.
Why do You Want to Measure Business Analysts?
The first question I ask when I hear that someone wants to measure business analysts is "What are you trying to accomplish?" Some common reasons for measuring business analysts include: to assess how well they are doing; to aid in performance appraisals; and to identify areas in which they can improve. Considering that all other employees are measured, it seems to make sense to measure business analysts as well. The answer to this question often exposes some of the core assumptions that an organization’s leadership has in relation to its employees.
Organizations that view business analysts as another set of "resources" often will be looking for ways to measure how efficient and effective individual business analysts are and ways they can be improved. Measurement systems in these types of organizations typically focus on measures of output, such as the number of changes made to the requirements after the original business approval and the number of defects found in testing, production, or the completion of requirements by the target date. These types of measures also point to an assumption that the purpose of a business analyst is to write requirements. In other words, requirements documents are a business analyst’s end-product.
I am not a fan of this perspective on measuring business analysts for several reasons. First, the purpose of an analyst is not to produce requirements. The real purpose of her role is value management—making sure her delivery team is delivering the right thing. Requirements are merely a means of conveying that information and should not be viewed as an end-in-and-of-themselves. Second, measuring things such as changes to requirements, and defects found drives sub optimal behavior. Analysts focus on perfect requirements and resist making changes. This does not reflect the real world where the team, including the analyst, does not know everything at the point where requirements are written so it is unrealistic to think that changes will not need to be made. Third, measuring individual analysts on these types of metrics discourages collaboration. If I am an analyst and I knew I was being measured on how quickly I get requirements done and how many errors were found after the fact, I am going to be less inclined to help the rest of my team when there is a backup in some other part of the process, such as testing. Metrics such as the timeliness of requirements lead to an increase in the use of the phrase "That's not my job." In addition, when gaps or errors are discovered (and they will be no matter how careful an analyst is) I'm more likely to spend a great deal of effort trying to classify these errors in a way such that "it's not my fault."
Measure Outcomes Rather than Output
An alternative approach I'd suggest, which is similar to one suggested by Laura Brandenburg in her post “ How Do You Measure the Success of a Business Analyst,” is to measure the effectiveness of business analysis by looking at whether an organization's projects are meeting their objectives. The first difference between this approach and the one mentioned above is that it measures the effectiveness of analysis work as opposed to the analysts who are doing it. This implies the belief that while a person with the title of analyst may be the primary driver of analysis activities, several members in the team have a part to play.
The other main difference is that the measurement focuses on outcomes rather than outputs. In other words, you are more concerned about whether business objectives are being met than whether intermediate outputs (requirements documents) are being produced. This approach assumes that you are more interested in using measurements to help your delivery team members improve their processes and the resulting outcome.
Some measures that can be used to determine the effectiveness of business analysts with this mindset include: the percentage of efforts where there is genuine stakeholder consensus about the objectives of the project; the percentage of those projects where the objectives were met; and the percentage of users who adopted the solution. You can find more ideas for possible measures of this sort in Laura Brandenburg's post mentioned above as well as Joy Beatty's post “ Measuring the Success of a Requirements (or Business Analysis) Center of Excellence .”
How to Make Meaningful Metrics
Last April I had the opportunity to collaborate with Todd Brasel, a QA engineer and project manager at Pitney Bowes, to share his approach to working with metrics in a business setting. During that collaboration, Todd introduced me to six key steps to creating a metric for process improvement, which I think is helpful when trying to establish measures of business analysis.
The first step is to identify the problem you are trying to solve or a question you are trying to answer with the metric. For example, we may be interested in knowing whether our business analysis efforts are being successful in driving agreement among a project’s stakeholders on the objectives of that project.
The second step is to identify quantitative ways to represent progress toward the goal. Regarding reaching an agreement on business objectives, we want to know the percentage of projects where the stakeholders agree on the objectives of the project.
The third step is to determine criteria for measurement and how to get the values. We want to see if the stakeholders for a given project all have the same objective(s) in mind for that project.
The fourth step is to identify data to collect and how to collect it. You don’t want data collection to be onerous, so think about what data do you already have that can tell you that you need to know. If nothing exists right off the bat, think about what is the simplest way to gather that data. In the case of our example, we would individually poll the stakeholders of a project to see if they give the same answer for the business objectives.
The fifth step is to identify the baseline, which gives you a comparison point against which you can identify improvements that happen after you put your plan into action.
The final step is to put your plan into action and review the results. This step assumes that you have a specific reason for measuring the performance of business analysis and that your main purpose is to check progress.
How Do We Tie This Back to Individuals?
The approach I suggest above works great if you are truly interested in improving the processes that your team uses to perform business analysis. But how do you relate that back to individual business analysts? I personally prefer approaches to providing feedback to people on the team that have been suggested by members of the agile community. Kane Mar provides a nice summary of those ideas in his post “ How to do Agile Performance Reviews.“ If you are currently working in an environment that you would not describe as being agile and really want to have some individual performance measurement, Adriana Beal provides a balanced discussion of the topic in her ebook Measuring the Performance of Business Analysts.
Considering all these perspectives, I think you can manage business analysts without measuring them if you view management as helping business analysts improve their skills sets while helping them be productive members of their team. If, however, you view business analysts as “resources” you will more than likely find individual measurements quite useful.
