Awareness of Risk Identification in Software Testing

We all assess risk every minute of our lives, whether we’re aware of it or not. The next time you go to cross the road, take a minute to think about the thought process you go through prior to stepping into the street.

It’s fair to say that many of us do the same thing while testing software. We identify risk seemingly effortlessly, but do we really understand the process we’ve undertaken? Do we know what methods we’ve called upon? Are we aware of how we’re identifying risks? And therefore, are we identifying all the important risks?

The Australian Workshop on Software Testing (OZWST) took place again in June this year, and the theme was “Experiences in Acquiring a Sense of Risk.” In essence, how do we go about identifying product risk; and further, how do we tell that story?

During the peer conference James Bach ran a workshop centered on risk identification. We were given a very broad specification (a few written lines on a whiteboard) of a product and asked to identify risks for it. After some time we were given further information on the product and then continued to identify risks in light of the new information, along with revisiting the risks we had previously identified in order to assess if they were still risks.

The real-time risk identification model below is the process our group followed during the workshop:

We began by analyzing the specification utilizing linguistic-based analysis, meaning we focused on the words in the specification to determine certain elements and functions of the product. We then listed questions and broke down, or modeled, the product, listing its components.

Given the very broad specification, our group quickly recognized that large assumptions were going to be made, as there was limited time for the exercise and we were not allowed to ask questions. Therefore, we listed all assumptions in order to ensure they were captured and declared to the group. Once the group was comfortable that we had established enough of a context, we began identifying associated risks and listing these also.

After we had quite a healthy list of risks, James entered the room and provided one more piece of information: a website URL. We began the process again, as the new information allowed us to conduct more experiential analysis, better break down and model the product, clarify our assumptions and declare new ones, and get a lot more specific with our risk identification.

It dawned on me after the workshop that what we had done was a repeatable process, and this is what I have captured in the model. As information is provided or gathered, you undertake further analysis in order to get more specific with your product model, assumptions, and risks.

For the purpose of the workshop exercise we had a deadline of ten minutes, so it was important for us to determine when we had enough information to begin risk identification. This is also true in the real world. We will always face constraints that mean we cannot infinitely question the context. We need to draw the metaphorical line in the sand and push on.

The following are brief descriptions of each element of the risk identification model:

• Information: Any information you are given or that you gather through the many methods available in our industry. This can be a specification, comparable products, business requirements, answers to questions, etc.
• Analysis: Analysis of the information you have been given or you’ve gathered.
• Analysis methods: There are many methods of analysis available. Some I have used previously include:
• Experiential analysis
• Linguistic analysis
• SWOT analysis
• State analysis
• Comparative analysis
• Breakdown/Model: The process of breaking down the information and your analysis in order to build a model of the product—whether a mental model or a written diagram, whatever works for you and your context
• Enough information: It can be difficult to determine when you think you have sufficient information to move forward with your risk identification, so it’s important to always remain open to further information coming your way.
• Assumption: I’ve heard many times that assumptions are dangerous, especially in the world of software testing. But assumptions will always be made because we simply cannot know everything. The important part is to be aware of them.
• Declaration of assumptions: You should be aware yourself of your assumptions, but in many contexts it will be important to also declare your assumptions to various stakeholders. Declaring assumptions can uncover further information and assist in risk identification when stakeholders add weight to the unknowns.
• Heuristics: These processes can assist you with identifying risks. The one I use most commonly is James Bach’s heuristic test strategy model. It provides many context elements that are important to all projects. Jakob Nielsen’s usability heuristics are also well-used items in my toolkit, along with many software testing mnemonics.
• Risk output: Your identified risks in the form of output that is most suited to your context. This may be an email, written on a piece of paper, or in a formal document that gains approval by stakeholders.

I wanted to develop a model that could fit many different contexts. In my experience, traditional risk assessment has taken place in a workshop where the focus was solely on risk identification. This model can be used in that context, but also on the fly while you’re testing. To highlight this point, let’s take an exploratory testing model and inject the real-time risk identification model.

On face value it looks like a lot of steps that need to be undertaken, but they don’t need to be extensive in duration or effort. That will be determined by the testing you’re performing.

The key lesson for me from OZWST this year is awareness. There are many things we do subconsciously in software testing that would serve us better if we thought about them more critically and were more aware of them. Risk identification—and the process we use for it—is one of those things. Once we become aware, we can record; once we record, we can share; and once we share, the community can learn. Let’s think more critically about what we do and maintain awareness.

Feel free to use this model to help you identify product risks, and please comment if you can improve on it through experience.

Thank you to all attendees of the OZWST peer conference.