Lessons Learned from Testing SAP R/3

SAP R/3 is the market leader in ERP installations and ERP sales. SAP has thousands of tables, multiple industry specific solutions, thousands of transactions, and connectivity to an unlimited number of legacy systems. Furthermore SAP can be configured differently from one company to another which creates a myriad of permutations for executing an SAP transaction. Installing and customizing SAP is a daunting challenge. Testing SAP R/3 is in and of itself another intractable challenge.

Many projects fail to test SAP correctly and consequently suffer staggering financial loses after deploying SAP into a live environment. The key to maximizing the value and ROI of SAP is to install and customize SAP correctly based on the documented requirements and to test it extensively based on the documented test cases and end-to-end business scenarios. Below some lessons learned are offered and identified to help organizations test SAP.

1. Not following methodology or No methodology at all
Some companies implementing SAP adhere to the ASAP methodology. Other companies have ad-hoc or ASAP-like methodologies for implementing SAP.

Even companies that are supposedly implementing SAP based on the ASAP methodology are not very strict and stringent in adhering to all the activities, deliverables, and tasks associated with ASAP. Consequently, but not surprisingly these companies have much confusion, obfuscation, befuddlement when they attempt to implement SAP. Compounding this problem is the fact that many large companies implementing SAP hire two or more implementation partners and multiple subcontractors that have incompatible approaches, methodologies, and lessons learned for implementing SAP.

The project manager and the steering committee should specify within the project charter how SAP will be implemented and what deliverables will be produced based on either ASAP or some other proprietary methodology. The objective is to have defined, proven, and repeatable processes for implementing SAP and that the project members have the knowledge or know-how for adhering to the methodology. The creation of an audit team or standards team would be helpful in enforcing compliance with the chosen methodology.

2. Inadequate test tools
SAP R/3 comes with an internal recording tool known as CATT (eCATT). One of the advantages of CATT (eCATT) is that since it is part of the standard SAP system it's free of charge. However CATT does have some limitations which impel many companies to procure other test tools.

There are many vendors offering commercial automated test tools and test management tools for testing SAP. Companies purchasing automated test tools expect and erroneously believe that the test tools will be the panacea to their entire SAP recording and testing needs. Unfortunately, this is not the case, since no two SAP implementations are exactly the same across two or more companies or at times even within different divisions of the same company. Consequently, a company implementing SAP might need to procure test tools from more than one vendor in addition to the CATT (eCATT) tool.

An SAP implementation could be implementing SAP add-ons such as BW (Business Warehouse), APO (Advanced Planning Optimization), SEM (Strategic Enterprise Management) or even modules such as PS (Project Systems) that generate graphs and charts that a recording tool does not recognize. Furthermore, a company may move its SAP GUI from the desktop (fat client) to running SAP as web-based (thin client) or through an emulated Citrix session which could render the existing test tools useless.

Companies that wish to move to an automated testing strategy should articulate and document what SAP modules and SAP add-ons they are installing in addition to any legacy applications integrating with SAP. This information should be provided to the vendors of automated test tools in order to determine what can actually be recorded and tested with the test tools. The company should further investigate with the vendor what additional benefits over CATT (eCATT) the automated test tools provide. The objective is to get the test tools that will maximize SAP recording.

3. Decentralized test teams
Some projects that I have consulted for, have a decentralized testing approach where each individual business process team, technical team (ABAP/4), and Basis (security) team conduct their own testing in the absence of a QA/testing team.

The decentralized method suffers from various drawbacks including redundancy, inconsistency, lack of standards, missing testing metrics, unreported test results, limited managerial visibility, limited test coverage, chaotic defect resolution, no independent verification of test results, etc.

Projects with a decentralized test approach perceive control as a significant advantage of decentralized testing. With decentralized testing each team has the independence and control as to how their particular area will be tested based on their own defined test procedures.

The perceived benefits from decentralized testing do not offset its drawbacks. A more robust approach for testing an ERP system is centralized testing where there is a QA team for defining the standards and procedures for the various testing cycles and documenting the test plan, lessons learned, UAT plan and the test strategy. The test procedures and test strategy include what test case templates will be used, how and what metrics will be reported, how peer reviews will be conducted, ensuring that all test requirements have been met, etc. In addition to the QA team under the centralized model the testing team provides expertise for executing test cases whether manually or with automated test tools. The testing team also provides independent verification for the test results since the testers were not associated with the configuration of the transactions, workflow development, creation of user roles, or the development of the RICE (Reports, Interfaces, Conversions, and Enhancements) objects, etc.

4. Working in Silos: Limited or no access to SMEs, Configuration Team
Since SAP is integrated and a change in one of its components can have a cascading effect on other aspects of the software it is often the case where the various SAP teams need to collaborate with one another.

This concept seems mundane and at first hand easy to grasp. But due to internal politics, deadlines, budget constraints, etc this concept is ignored and overlooked. When collaboration falls through the cracks the various SAP teams (configuration, RICE, Basis, testing, infrastructure, data, etc) start working independently in isolation or in a silo.

I have been in projects where the testing team members who arrived during the middle of the realization phase have been unable to get support from the SAP configuration team members and SMEs for drafting test cases, peer reviews, and sign offs.. In other projects the QA and testing team are almost considered a “bother”, “nuisance” to the other SAP teams.

The test manager and PM need to foster an environment of cooperation among the testing team and the various other SAP teams for the successful completion of testing artifacts and testing cycles.

5. Missing test bed
Within the SAP client landscape a dedicated QA environment should be allocated to the testing team for the various testing phases. The environment should be production sized, and contain production like data, have a stable baseline, and frozen configuration by the time the testing cycles commence.

What I have seen at many projects is a so-called “dedicated testing environment” that many SAP teams have access to for training, last minute configuration, etc during the middle of a testing cycle. This creates conflict among the teams and causes confusion over the ultimate ownership of the testing environment

The test manager should insist on a test environment where the testing team has complete control (for instance control over the accounting periods, running payroll, etc) and where the environment is not shared with other entities in particular during the test execution phase.

6. No controls for promoting/transporting objects into production
In many SAP projects that I have consulted for, there are very little controls for promoting objects into a live production environment. Other projects are very lax with their rules for promoting objects and have no means for auditing the approval process for transporting objects or do not include all the necessary stakeholders in transporting objects.

ERP systems like SAP R/3 offer benefits such as tight integrations across its modules and sub-modules and access to real-time data, but these benefits can backfire or even become otiose when a poorly tested or not even tested object is transported into a production environment. A faulty transported object such as a configuration change or code for an interface can have a deleterious cascading effect on other SAP functionality in the production environment. For instance a change to the HR module within payroll can affect functionality in the FI-CO module.

To mitigate the risk of transporting objects it is necessary to include approvals and set up a system of workflow where stakeholders from the testing team, Basis team, and configuration leads have input into objects that will be transported into production. A project should set priorities for transporting objects based on their criticality, document and approve objects that get transported after having been tested, and establish the frequency in which objects will be transported (i.e. every Friday at 4:00 pm) to avoid impacting end users as much as possible.

The company Mercury Interactive offers the tool Kintana which can help companies to move objects into various SAP clients and instances while maintaining a history log for auditing purposes of the objects that were transported and the entities approving the transports. Alternatively, I was a project where the organization created a home grown repository within Lotus Notes that provided managerial visibility for moving and transporting SAP objects which included notifications and approvals.

Whether a commercial tool is purchased or a home grown application is developed the main objective is to place stringent controls for moving objects into production with the necessary approvals while maintaining a history log.

7. Flaws with BPPs
For those projects adhering to the ASAP SAP methodology BPPs (Business Process Procedures) are artifacts or outputs from the realization phase. BPPs are documented based on stand alone or for single SAP transactions. The BPP provides detailed information in the form of instructions with screen shot print outs for how to execute a given SAP transaction (i.e. SAP transaction MM01–Create Material).

Admittedly, BPPs can assist a tester or end user on how to execute a given SAP transaction based on the project’s specific customizations. However, and this is often the case the SAP configuration team will document BPPs in a given environment such as the configuration environment with configuration data and then the SAP configuration team fails to update the BPPs to reflect the changes of the QA environment. The failure to update the BPPs diminishes their value for the testing team.

When BPPs are not updated they contain obsolete data, and may lack the necessary information to execute an SAP transaction for an SAP environment that has had customizations since the time the BPPs was initially created. Furthermore, many organizations do not perform any form of version control on BPPs which makes it difficult to discern whether a BPP is finalized, completed, peer-reviewed or still undergoing changes.

Poorly documented BPPs or outdated BPPs have a propagating effect on the creation of test cases and test scripts. Since BPPs are documented per stand alone SAP transaction, the testing team will need to link multiple BPPs for end to end SAP test cases (i.e. Hire–to–Fire test case) that involve multiple SAP transactions with pre and post conditions. A single poorly written BPP or outdated BPP can inhibit the creation of a test case containing several SAP transactions.

8. Missing peer reviews
Peer reviews help refine work-products and deliverables. Peer reviews also provide independent verification and give the end customers or SMEs an opportunity to provide feedback during the early stages of the SAP implementation. Peer reviews can occur at many junctures during the implementation of SAP for tasks such as filling out CI (Customer Input) templates, drafting test cases, creating functional design specs, development of business process flow diagrams, documenting BPPs, code walk-through, etc.

Inexplicably many SAP projects do not engage in the practice of peer reviews or have any templates or forms for documenting peer review feedback. The end result is that often times the end users have complaints about the quality of test cases during the UAT (User’s Acceptance Test) and about how a particular process was customized in SAP versus how the process was previously executed in the end user’s legacy systems.

Test managers should solicit the feedback, input and even the sign-off from the SMEs for the various testing artifacts as soon as possible or as the testing artifacts are produced. It is in the best interests of the test team to identify problems with the testing artifacts as soon as possible as opposed to weeks before the SAP cut-over or SAP go-live dates.

9. Problems obtaining valid test data
Arguably, the most prevalent risk to conducting an SAP test whether it is integration, functional, string, volume, smoke, or security test is obtaining valid SAP test data.

When dealing with SAP data invariable one or more of the questions below will arise before a testing cycle ensues:

Where will the test data come from? How will one produce new data for transactions that require unique data? How will the data be loaded into a new SAP client and instance? What happens when an interface or conversion needs to send data into core SAP R/3 from a legacy system and the interface or conversion is not executing properly or has yet to be developed? What happens when the transactional data has not been created?

Assuming that the testing team has a dedicated test bed or QA box the test manager will need to ensure that all the necessary data (master, transactional, test, etc) is properly loaded as part of the assessment for the test readiness review. Ideally an SAP test will be conducted within a test bed containing data that closely mirrors production data and where the testing environment is production-sized. The test manager should prioritize the test cases that will be executed and determine and document any work-around for test cases that cannot be executed due to missing test data.

10. Missing flow processes, diagrams
Diagramming or modeling how a business scenario will flow within SAP provides invaluable insights to the testing team and the configuration team. Diagrams and flow processes can illustrate the lifecycle, stages, sequences, activities, states and events associated with a particular business process. Unfortunately, many projects undergoing an SAP implementation or SAP upgrade fail to diagram their business processes leaving testers or newly hired resources in a bind to comprehend how legacy processes, or new customizations derived from gap analysis will be executed within SAP. SAP testers and end users participating in the user’s acceptance test should have access to diagrams depicting how business processes flow within SAP. The absence of such diagrams puts undue burden on the testing team and places the end users in a position of disadvantage.

The SAP business analysts in conjunction with the SAP configuration team, and possibly the SMEs (Subject Matter Experts) can develop diagrams in tools such as Visio, or the ARIS modeling tool that integrates and links with SAP’s Q&adb (Question and answer database) for those SAP implementations following the ASAP methodology. Alternatively, with Rational’s tools one can develop UML (Unified Modeling Language) diagrams to illustrate processes with interaction diagrams (sequence and activity diagrams).

11. No integration testing with external components
SAP integrates with many legacy systems via interfaces, IDOCs, conversions, BAPIs, connectors, or even middle ware such as Mercator. SAP can even be integrated with other commercial ERP systems, forecasting/planning systems, and CRM applications.

Although, the integration of SAP with other systems is critical to the successful implementation and deployment of SAP, many companies merely test during the integration test the interaction of SAP among its various modules and add-ons. Although the aforementioned type of integration testing is necessary it is also critically important to test the integration of SAP with non-SAP systems.

Some examples of integration testing that I have conducted between SAP and non-SAP systems are: SAP integrating via IDOCs to bar coding software, SAP properly sending outbound data to legacy system and the legacy system processing the data correctly, end to end financial reconciliations between SAP and data warehouses, integration between SAP and planning systems for MRP runs, etc.

12. Limited security access
I have seen SAP testing efforts come to a halt because the testers did not have the necessary privileges and permissions to execute certain SAP transactions or the necessary roles to submit electronic approvals. This problem is exacerbated when testers are executing test cases at odd hours or during weekend hours and there is no Basis support to augment the tester’s security role.

Successful execution of certain test cases in SAP whether manually or with automated test tools will require the test team members to have super user access. This is of utmost importance when conducting positive and negative testing for security roles. The testing team should coordinate and plan with the Basis team the necessary roles and permissions for developing test cases and for executing test cases in the designated QA environment or test bed.

Conclusion: Document these lessons learned as part of your test plan to mitigate the risk of repeating these blunders in your SAP installations.