Six years ago, not long after the Sarbanes-Oxley Act introduced new levels of oversight to public companies, Linda Hayes speculated about what the legislation might mean for the state of software testing in large, public corporations. "Software QA is no longer an optional function primarily designed to protect developers from their mistakes," Linda wrote, "but is an essential one that protects them from SEC sanctions, civil damages, and an all-expense paid vacation to Club Fed." Now, she takes another look at her own forecast and how Sarbanes-Oxley has changed the testing environment.
When the Sarbanes-Oxley Act ("SOX") was passed in the wake of Enron and other examples of bad business behavior, I wrote a column for StickyMinds "Hello Up There! The Sarbanes Effect" speculating that maybe—just maybe—this legislation would elevate that status of testing in the corporation. After all, if corporate officers and directors had to accept potential liability for errors or omissions, they might look at testing as more of a necessity and less of a luxury. What a breakthrough that would be, from the bowels of IT to the rarefied air of mahogany row!
It may be coming true. In the past year, I have experienced more than one instance of corporate audit and compliance inserting itself into the testing area and vice versa. This is unprecedented in my experience and may portend the anointing of testing as the new accounting of IT.
Testing in the Audit Trail
In one case, the audit group declared itself a stakeholder in the user acceptance testing process and promoted a specific agenda around the goals and deliverables, requiring sign-off on test strategies, plans, test cases, and results. In another case, the compliance group performed an independent, internal survey to ascertain whether protected identity information was being exposed during test. In both cases the test organization was a direct beneficiary of the attention.
But the most interesting—and promising—case was when an enterprising manager was able to displace hundreds of thousands in annual audit fees by leveraging her automated test tools and processes and applying them to SOX. Having previously been a formal external auditor, she helped lead her own organization through year two of SOX. She gained hands-on knowledge and experience with the challenges of getting the business to test SOX controls. She and her team also had been heavily involved in transferring many of the first-year manual controls to application controls. Now, as the global testing director for a $4 billion manufacturing concern with a very complex, global SAP system, she saw on a daily basis how the impact of change could also impact the SOX-controlled environment they had put in place.
Thorough testing of new projects, as well as managing testing of upgrades and enhancements to the existing production environment, required the execution of hundreds of tests was almost a full time job for many of the business owners. In order to take off some of the manual testing effort and get back accurate results more quickly, she had already made the business case to automate the regression baseline testing, which covered over 400 of the most highly used transactions and a solid base of their critical processes. With the right combination of technology, skills, and effort and working together with the business, her team was able to automate and validate the more than 400 transactions and associated processes using seventy-five end-to-end scripts across seven primary functional areas. Pretty impressive.
Having tackled that elephant and looking at the changes yet to happen, she had an inspiration. There were over seventy individual financial controls managed by the software that had to be audited. Either the business had to execute the tests and the audit firm retest or a third party could be retained to independently test, but either option would cost more than $250,000 per year, every year, plus hundreds of internal man hours. Working with the external auditors and internal compliance team, she showed them how they could get higher quality results using an automated approach and still satisfy the requirements for independent validation of the software's internal controls.With that agreement in hand, she and her team implemented two things: one, a report that would show what had changed over a set period of time, so they only had to test what had changed; and two, integration of the controls tests as part of the automated test arsenal, including the right set of roles and process combinations in the test suite to be sure the more than seventy controls were executed each time. The test results were reported and saved in an "audit ready" format. What had taken months to do before could now be done in seventy-two hours and by only one person. For a relatively minor incremental effort, this approach saved her company a quarter of a million dollars a year and put big smiles on the business owners' faces.
Think Outside the Traditional Box
Think about it. We all have to scramble to justify investment in testing and automation, often producing laborious ROI analyses. The problem with most of these is that, unless your test coverage is comprehensive to begin with (and whose is?), at best you are going to increase quality without increasing testing costs. Reducing testing costs is hard to argue, since enough likely isn't being spent anyway. But, with this approach, you are displacing actual hard costs and tedious effort.
SOX applies only to public companies, of course, but that doesn't mean smaller or private companies don't care about compliance or have exposure to audit requirements.
The larger lesson is that we need to learn to think outside of the traditional feature/function box and realize that the systems we test may have far-reaching financial and operational consequences to our company or those that use the software we may sell. We can use this new perspective in the ongoing battle to justify additional investment in testing and automation and to get the right kind of attention from senior levels of management.
