Many of us think about system risks when deciding what to test, but it seems that few have a standardized way to assess the risks of a given system. In this week's column, Rick Craig shares a simple method you can use to target your test efforts according to risk. This method comes from, and is developed more fully, in Systematic Software Testing, of which Rick is lead author.
It seems that risk-based testing is a hot topic. Yet many testers are not exactly sure what that term means or how to do risk-based testing. A poll of the audience at the recent STAREast software testing conference showed that less than half of the attendees
conducted any type of deliberate risk analysis.
If resources were unlimited (just imagine), perhaps using a risk analysis to prioritize the tests would not be necessary. But getting back to real life, a risk analysis can help us determine the priority of tests. In all fairness, many who do not conduct a risk analysis no doubt still prioritize testing based on what is important to the customer, what has failed repeatedly in the past, what is complex, and what has changed the most since the last release. All of these reasons are a sort of "risk analysis" conducted in the heads of the testers. But by formalizing this process just a little bit, we can get more consistent results and achieve credibility with our managers, developers, and colleagues.
Here's an outline of a simple ten-step process for conducting a usable (and reusable) risk analysis to help you prioritize tests. Use it as a starting point from which to add your own experiences to improve your own organization.
Step 1: Organize a brainstorming team.
This team should be made up of three to six stakeholders that possess some level of expertise about the nature of the system or how it will be applied in the business sense. Developers, business analysts, users, marketers, customer support, and testers are likely candidates.
Step 2: Compile a system-wide list of features and attributes.
For example, the brainstorming team for an ATM application would probably compile some of the following features: withdraw cash, deposit cash, check account balance, purchase postage stamps, transfer funds, etc. It may also be useful to include global "attributes" such as security and usability in the list.
Step 3: Determine the likelihood of failure of each feature or attribute.
The brainstorming team should assign a value for the relative likelihood of failure of each feature and attribute. For simplicity, you can use the scale of high, medium, and low. Our ATM risk analysis team might determine that the likelihood of failure of the withdraw cash function is relatively high since it requires the system to access and update the account, physically remove cash, etc. Transfer funds may be assigned a relative value of "medium" since, even though it requires the interfacing between accounts, it lacks the complexity of physically dispensing currency. Developers and systems architects are particularly valuable in this portion of the risk analysis, since the
likelihood is largely based upon the systemic characteristics of the software under test (SUT).
Step 4: Determine the impact of failure for each feature or attribute.
Use the same scale as in step 3 to assign a value for the impact of failure. A failure is when the system does not function or does not function correctly. Risk analysis team members who have a strong understanding of the business aspects of the SUT are particularly valuable here, since the impact is based upon the effect a failure will have on the users. Our team might assign a high value to the "withdraw cash" feature since they would rightfully conclude that this is the single most important feature to most users. One hint: Many users will insist that every feature should be assigned a "high" value for impact of failure. Obviously, if we are trying to use the risk analysis to prioritize tests, it doesn't help us if everything has the same rating, so it may be necessary to limit the team to a specific number of "highs."
Step 5: Assign numerical values corresponding to the relative values assigned in steps 3 and 4.
We typically assign a value of three for high, two for medium, and one for low. Other teams choose different scales such as ten for high, five for medium, and one for low. It is relatively unimportant which scale you use, but once you choose one, you must remain consistent throughout the exercise.
Step 6: Compute the risk priority.
Add the values assigned to the likelihood and impact together for each feature and/or attribute. For example, if the withdraw cash feature was assigned a value of high (or three) for likelihood of failure and high (three) for impact, then the risk priority would be six, which is the highest risk rating using this scheme. Some teams may choose to multiply the numbers together instead of adding them. This will have very little impact on the actual prioritization of the features, but once you choose a scheme, you must remain
consistent throughout.
Step 7: Review/Modify the values.
After the initial risk priority has been assigned, it can be modified based upon other knowledge, such as cyclomatic complexity, failure history, degree of change, etc. For example, if we had assigned a high likelihood of failure to the withdraw cash feature and a review of our defect logs shows that, in past releases, this feature has operated virtually without flaw, we may choose to lower the likelihood of failure and modify the risk priority accordingly.
Step 8: Reorganize the list of features and attributes in order of risk priority.
Step 9: Establish a "cut line."
If time and resources do not allow enough time to address all facets of the system, it ay be necessary to establish a "cut line" that separates features to be tested from features not to be tested (or maybe tested less). At the end of the day, software risk analysis is used to prioritize what to test first and, in some cases, what may not be tested at all.
Step 10: Consider ways to mitigate risk.
In some instances, you may decide that the risk rating of certain features is unacceptably high. In this case, you could consider some type of mitigating activity such as a code inspection, additional testing, etc., to reduce the likelihood of failure and hence reduce the overall risk.
Mitigated List of Priorities for ATM Features/Attributes
Risk analysis is an important part of testing. Few organizations have enough time and resources to address every feature to the level of detail that they would like. Risk analysis allows us to intelligently choose where to apply the bulk of our effort. Once a risk analysis has been conducted for a particular product, it is not necessary to completely start over for the next release, but rather just revisit the results of the previous risk analysis and update the features, likelihood, and impact columns.
Good Luck.
