Many companies that elect to buy a packaged application rather than build a custom system do not maintain a traditional QA department. Acceptance testing of their own configuration is performed by users conscripted for the duration of the project who return to their day jobs as soon as possible.
The downside to this approach is that there is little reusability between projects. Borrowed resources don’t have the incentive, time, or skills to implement automation or invest in processes and documentation that will survive the project. The result is a laborious, expensive, manual testing process of uncertain coverage that becomes the long pole in any project.
This is a problem for both the customer and vendor. The resource and schedule drain discourages the customer from implementing upgrades or change packs as often as they become available. As a result, the customers often demand targeted patches for issues that may have already been resolved in a bundled package or they delay the purchase of new products.
This is a headache for the vendor, who can’t sell new products and ends up supporting multiple versions and patches in the field and can’t possibly test every combination. The chaos that inevitably ensues degrades quality and stability for the customer and diverts the vendor's development and support resources to constant fire drills. The irony is that less stability both increases the need for changes and decreases the appetite for them.
Converting Costs into Cash
Some software vendors have seen the customer’s testing challenges as an opportunity. After investing in test automation for their internal needs, they realize it can be packaged and implemented for their customers as a new product and service offering. This is a win-win-win for the customer, the vendor, and the vendor QA team.
For the customer, test automation reduces the demand for resources, compresses the implementation schedule, and increases the stability of the deployment. endor revenue increases from new products and services while support costs decrease due to greater stability. QA team members can monetize their efforts and also develop a closer relationship with their customers—sharing test cases and results in a common format and platform. This extends their understanding of customer needs and therefore their test coverage, which improves quality overall.
There is a lot to like in this model, and as long as automation has been around you have to wonder why this offering isn't already standard for most vendors. The answer can best be found by reviewing the following case study, which exposes the issues but also suggests solutions to making this a reality.
Case Study: Testing as a Product
A financial services vendor’s QA manager perceived the potential for marketing his automated test library to customers and sought to make it a reality. He presented the concept to the implementation services team, who loved it but noted that there was no documentation and training for the library. Because the automation assets were developed for internal consumption, they were not documented for external use, and neither was training formalized.
Producing documentation and training materials required additional investment the QA manager could not afford without a bigger budget, so he suggested that the services team should pick an "alpha" customer who would be forgiving of the rough edges in exchange for favorable terms. If the project was a success, then the QA manager would have a business case for the extra investment. In exchange, he also agreed to have one of his QA resources work with the services team to provide on-the-job training.
Eventually, a customer was identified and convinced, and so the project started. But as soon as the QA resource unveiled the test library, the customer became confused, explaining that they tested by simply performing the same tasks they did in their day job, with perhaps special emphasis on new or troublesome areas. The vendor QA tests were designed around the way the software was built, not the way it was used, and made no sense to the customer. It became clear there would be no reuse of the QA test cases.
Undaunted, the services team decided to create new acceptance tests, reasoning that they could scrub and reuse the project assets for future customers as well as use them internally in QA to extend their own coverage. They decided to partner with the customer's expert users to automate a representative sample of the most typical user scenarios. This process revealed yet another obstacle: The customer resources were non-technical and the services team was trained on the vendor's products but not on the test tool. The only resource that had the skills to develop new tests was the single QA resource.
Still the QA resource soldiered on and she was able to automate a passable subset of user acceptance tests to be scrubbed for internal reuse. But when the QA group attempted to incorporate the tests into its own environment, they realized their configuration—especially the data—was completely different from the customer's. Most of the tests would not work.
At the end of the day the project was scrapped.
But that doesn't mean it wasn't a great idea, it just means you have to avoid the pitfalls they uncovered. If you want to sell tests to your customers, you have to treat them as you would any other product. You must understand the customer's requirements and tailor your tests to meet their needs and skills. You must also have the commitment to providing the documentation, training and support that are needed to assure success in the field. And if you want to enjoy collaboration and reuse, you have to design your tests and test environment to support multiple configurations and data.
Judging from the number of companies who are either considering or actively pursuing this idea, I think it will become a reality. And, once a vendor shows success, competitors will be forced to follow. If that happens, software vendors may be able to convert the cost of test automation into cold hard cash as revenue and cost savings—not to mention higher quality and happier customers.
Have you or anyone you know done this? If so, please share!
