Calculating the Economics of Software Inspections

As we near the thirtieth anniversary of the development of the inspection process by Michael Fagan at IBM, I continue to be amazed at the number of software development organizations that do not use this powerful method to improve quality and productivity.

I recently had a discussion with someone about Return on Investment (ROI) for the process. He said they did not do ROI calculations, as they knew they were getting a good benefit from requirements and design inspections. He then said, "We do not do code inspection as they take too much time." In the absence of any ROI calculations, how would he know?

Let's take a look at this situation and explore some relatively easy ways to evaluate either potential or actual benefits from inspections. It really does not take a large amount of data, and the cost to do the analysis is minimal. For the potential benefit, we will make some assumptions about the cost of inspections and try to estimate the savings from reduced test cost. I'll also provide a spreadsheet for doing "what-if" analysis of various different savings based on inspection effectiveness, and how much defect removal in test might cost.

The inspection process itself will not be explained in any detail, as it has been explained in many books and publications (see a short list at the end of the article).

The savings can be calculated as follows:

Savings = (Original Test Cost) - (New Test Cost) - (Cost of Inspection)

How each of these is derived is shown below.

Inspection Cost
If you are not already performing inspections and therefore do not have your own data, the following rules of thumb can be used:

One thousand lines of source code will take about 50 hours of effort for a four-person team at the recommended preparation and inspection rates of 150 lines per hour. (The number is actually 53, but rounding to 50 makes the analysis easier.)

A 35-page design or requirements document will take about 50 hours of effort for a five-person team at the recommended rate of seven pages per hour for preparation and inspection.

We will use these numbers as baselines for ROI calculations. In addition, there is the defect rework cost for the defects found in the inspections.
Test Cost
Based on questions I have asked in test measurement tutorials I have taught over the last four years, most organizations have little idea of their test cost. Again, there are easy ways to approximate these costs if you do not have the actual data. Most organizations can identify the start of the last test stage before shipping a product to the customer. Typically called System Test, it is conducted with more control and measurement than the Unit or Integration Test stages that precede System Test. There is usually some sort of problem-tracking system in place, and often there are closure codes that identify defects that are fixed as well as other closure reasons (user error, duplicate problem, etc.).

To develop the cost per defect found in this test stage, take the development and test resources on the project and divide that by the number of defects. There are several assumptions that are implicit in this calculation. First, when System Test starts, new functionality is not being added, or if it is, the development effort (cost) for the additions is tracked separately from the defect fixing activities of the development staff. I will save comments about adding new functionality during system test for a later article. The true test cost at this stage is the effort spent by testers finding problems and verifying fixes to defects, and development effort spent fixing the defects. I have seen developer effort erroneously logged to design or code (after all, "I'm writing code so I must be coding" is the wrong assignment here). It is important to "normalize" the cost number to "hours per defect" to allow use in later calculations where the number of defects is the variable.

The cost during System Test is (to a first approximation) the number of testers and developers, times the number of working days spent in System Test. This could be adjusted upward by any overtime, and downward by overhead activities such as vacation and training. (If you don't have an effort tracking system, this is the best you can do, and I have noticed that organizations that do not have effort tracking systems are the ones that also cancel training and vacations during the crunch period of test, which makes the calculation simpler!)

If you do not have this data, or cannot estimate it, fall back on the "10x" rule originally stated by Barry Boehm. He noted that the defect repair cost increased by a factor of ten for each successive stage of defect removal. A defect that costs one hour to find, fix, and retest in unit test, would cost ten hours in the next stage, 100 hours two stages later (system test) and 1000 hours after shipping. In my experience, I have not seen the producer costs increase at this rate, but have seen 30:1 to 50:1 ratios between inspection cost and system test or customer defect repair costs (where the inspection cost is in the two-hour range).

What these simplified calculations do not consider is that the defects from the earlier stages (especially requirements and design) have greater rework costs the later in the development cycle that they are discovered. I have seen requirements and design defects take 60-300 hours to find, fix, and retest. If you have these numbers, the calculations can be refined. (Also, if you have these numbers, you have probably already done these ROI calculations!)Example Calculations
There are two ways to calculate savings: in the large, and at a detailed level. In both examples below, concentrate on the method, and not the specific numbers used. Scale the example to your project size, and use your own data wherever possible. First, a higher-level view of savings can be developed from your last project. Determine the test cost per the above guidelines. Let's say you spent three months in System Test with a total project size of 15 people, or roughly 45 person-months (3.75 person years). Let's further assume System Test took 25 percent of the total project budget, so the total investment was 15 person-years (unless you have the total project cost—in which case, use that). This effort produced 45,000 source lines of code.

From the above rule of thumb, it would take 45 x 50 = 2250 hours (about 1.1 person-years) to inspect the 45,000 lines of code. If inspections reduce the defects found in System Test by ½, the savings would be about (3.75/2) = 1.87 person-years, or a 1.87/1.1 return on investment; and this is just counting the System Test savings, ignoring the Unit and Integration Test costs. This simplified analysis ignores the defect repair cost for those defects discovered in inspections. I have found this cost to be relatively low compared to the inspection cost, and well within the savings in this example. Keep in mind this example is only for code inspections. It could be extended to include both cost and savings for requirements and design inspections.

Now we'll look at an example, using the "What-If" Spreadsheet ("Defect Depletion and Cost Analysis Spreadsheet"—download available on this page in the above-right blue nav bar). In this example we will use 2000 lines of source code as the base size. Rather than top-level estimates, this example breaks down the removal cost per development stage and adds in both Unit and Integration Test defect removal costs. In the first case, we will assume there is no defect removal activity until testing starts, and that there are three test stages, with defect find, fix, and retest costs of six, 16, and 35 hours per defect. We also assume the defect removal effectiveness of each test stage is approximately 50 percent. In the second case, the inspection/review removal effectiveness is 50 percent, with a cost per defect of two hours per defect (detection and correction cost). In the third example, the inspection effectiveness is increased to 70 percent. This is displayed in Figure 1.

Figure 1. Comparative Defect Removal Costs
The x-axis represents development stage of requirements, analysis, high-level design, low-level (detail) design, code, and test stages. Although it may look like a "waterfall" development representation, this representation only requires that the development activity be recognized as the "source" of the defect. There are several conclusions that can be drawn from this chart: First, the cost of defect removal without inspections can be substantial, with up to 5:1 savings in System Test compared to a 70 percent defect removal in inspections; Second, there is also a significant savings if the inspection effectiveness can be raised from 50 to 70 percent. The numbers are easier to see when the total cost is displayed cumulatively, as seen in Figure 2.

Figure 2. Cumulative Defect Removal Cost
In Figure 2, you can barely see the additional cost for improving defect removal in inspections as a function of additional defects removed. The cost associated with improving the effectiveness should also be added. There is a warning here: while it is good to set goals to improve removal effectiveness, wishing will not make it happen. To increase the effectiveness, some overt action will be required, whether it is improved checklists, larger teams, or slower preparation rates.

All of these will increase costs, and should be added to the equation. One way to do this is to look at your current preparation and inspection rates, and if too high, recalculate the cost for the slower rate. Overall, this chart shows a 3:1 return on investment for using inspections (against the 70 percent effectiveness rate). It is hard to see in the chart, but the ROI for improving the inspection effectiveness from 50 percent to 70 percent is about 6.5 to 1. Let's recalculate this assuming that to get from 50 percent to 70 percent required slowing the preparation rate by 25 percent. That would raise the cost in the example by 25 percent, from 84 to 105 hours. This is 21 hours compared to savings in the range of 255 hours, or about a 10:1 payback for the increased effectiveness. This is a tradeoff most managers and developers will accept.

In these calculations we have made the assumption that 1000 lines of code can be inspected in 53 hours by a four-person team. So to reach the cost of two hours per defect, you need to be finding more than 25 defects per 1000 lines of code, since the two hours includes the rework cost. If you think 25 defects is too high, cut it in half and rerun the calculations. Inspections still provide a positive ROI, in this case only 2:1 rather than 3:1. If you plan three-person teams, lower the cost appropriately.

One other point—in these examples the inspection cost savings accrue only for "major defects," the term used to indicate the defect would cause the product to malfunction in either test or use. Spelling, grammatical, and code comment defects (called minor defects) are not included in the calculations. We are trying for an apples-to-apples comparison. I have actually seen some inspection analysis charts where major and minor defects have been lumped together. I'm not sure why anyone would do this (unless they were trying to make one of the defect removal numbers look better—a clear case of metrics malfunction).

Summary
I've tried to present a fairly straightforward way to calculate ROI scenarios for inspections, using a "what-if" spreadsheet template referenced above and located at this Web site. Keep in mind these examples are meant to show you how do to the ROI calculations, and that the data used, while reasonable, needs to be substituted with your own data or estimates. This method includes some simplifying assumptions, and you may be challenged because one or more items were left out. When that happens to me, I ask, "What is the overall impact of your item?" Just add the numbers of your objector in, and see what the impact is. Usually, the impact on the ROI is negligible. You may run into problems with the defect density numbers. All I can say is that numbers in the 20-25 defects-per-1000-lines-of-source-code range are not at all unusual when an organization starts doing inspections. If you were not seeing this defect rate, I'd begin to wonder how many defects are being fixed "under the table."

Calculations

Design Inspections

35 pages
7 pages per hour recommended preparation rate
5 hours per inspector
5 inspectors = 25 hours preparation
7 pages per hour recommended inspection rate
5 hours, 5 inspectors = 25 hours
Total time = 50 hours

Code Inspections

1000 source Lines
150 Lines per hour recommended preparation rate = 6.66 hours
4 inspectors = 26.66 hrs
150 lines per hour recommended inspection rate = 6.66 hours
4 inspectors = 26.66 hours
Total = 53.33 hours (50 for ease of calculation)

Note: Requirements inspections usually take longer

References:

• Strauss, Susan, and Robert Ebenau. Software Inspection Process. McGraw Hill: 1992
• Gilb, Tom, and Dorothy Graham. Software Inspection. Addison Wesley: 1993
• Radice, Ronald A. High Quality Low Cost Software Inspections. Paradoxicon Publishing: 2002
• Weller, Ed. Defect Depletion and Cost Analysis Spreadsheet