Testing COTS: When, How, and How Much?

While there are no hard and fast answers to how much trust you should instill in commercial, off-the-shelf applications, there are some basic guidelines that can help. Start by asking yourself these questions: 

1. What is the risk that the application has defects?
2. What is the risk that your use of the application could cause errors?
3. What is the potential cost of a defect or an error?
4. If you do test, what type of testing should you do?

Depending on the answers, you may find that your COTS application-testing needs and approach will vary widely.

Application Risk
The risk inherent in the application itself is a function of its scope of functionality, its breadth of users, and its maturity. For example, most would agree that mass-market utilities such as word processors, spreadsheet programs, slide presentation programs, and graphics tools pose little inherent risk. Their scope is narrow, they have millions of users, and most are highly mature.

Then there are more specialized applications, such as accounting systems, that are still somewhat narrow in scope but whose maturity and customer base may vary widely. Consider that there are mass-market accounting systems, niche products that address smaller segments such as non-profit accounting, and even custom solutions for highly differentiated customers. These products may also have wide variations in maturity. A new product for a small market with few customers poses inherent risk.

And then there are massive applications, such as enterprise resource planning systems (ERP), that encompass multiple departments across many industries and support complex, critical business processes. These have a wide scope of functionality, may have many users, and may or may not be mature. Due to their scope and complexity, there is inherent risk in these systems. These risks may be greater for newer products with fewer customers or for customers in a new industry or using a new component.

Usage Risk
The way you use an application can also change the picture. Consider a spreadsheet used by a construction company to manage multimillion-dollar bids. In this case, the software itself may have low inherent risk, but if the template contains complex formulas and is tweaked often, the risk of errors rises.

Many desktop utilities also can be programmatically integrated with other software. For example, an internal stock trading application might integrate a spreadsheet for its graphing capabilities. In this case, the risk of the spreadsheet malfunctioning might be low, but the potential that the integration has errors is higher. The wrong data might be mapped to the formulas, for example.

The usage of enterprise applications is usually controlled by configuration settings and master data. However, the rules that govern a business process can vary significantly from one company to another, and the sheer complexity of these applications with their multiple points of integration raises the risk of error dramatically.

In other cases the user might actually modify an application to accommodate unique requirements. Clearly this type of usage poses the greatest risk, regardless of the stability of the original state of the software.Potential Cost
An essential part of any risk assessment includes the cost of failure. A garbled word processing document or scrambled slide presentation might be embarrassing, but a spreadsheet with template errors might cost millions if a bid is incorrect or if an improperly integrated graphing capability of a spreadsheet triggers the wrong buy or sell order.

Similarly, a poorly-functioning accounting system for your checking account might make it hard for you to balance your checkbook, but if used by a public company, it could create massive liability for inaccurate financial disclosures. Or, an ERP application that controls the manufacturing floor could literally shut down production if a business process goes awry.

Thus, the potential cost of a failure must be balanced against the cost of testing.

Type of Testing
Once you determine that some testing is needed, the next step is to decide what type to conduct. Unless you make modifications directly to the software, unit testing should not be needed. In fact, in most cases you won't even have access to the underlying source code.

But in those cases where you are integrating a purchased application with other systems, whether developed or purchased, integration testing of the seams is clearly required. The goal of this type of testing is not to verify the functionality of each application, but to assure that the information shared between them is correctly sent and received.

And in virtually all cases where substantial risk is identified, whether risk of inherent defects or those arising from usage, the most common form of testing is acceptance testing—that is, to test the software from a user's point of view to assure that it meets its critical requirements. This does not necessarily mean that you exercise all positive and negative cases or that you execute every possible path, but that you verify the functions with the highest risk.

As tempting as it is to dismiss the need for testing COTS applications, it's just not that simple. Take the time to analyze the application so you can balance the cost of testing against the potential risk and cost of failure. And then, after you test, all there's left to do is hope that it works.

Have you been burned by a COTS application? Let us know what you learned by posting your story below.