If you’re like many of the leading enterprise testing teams today, you’ve already recognized the need for test automation and committed to ramping up your test automation efforts. You may have even made significant progress toward automating primary use cases that cover your top business risks. But then you reached an automation plateau.
Shortly after you built your initial automated test suite and started executing it regularly—potentially as part of a continuous integration effort—your dependencies created a roadblock. You need the dependent system components of the application under test (AUT) to be available in the test environment whenever your tests are executed.
However, with complex enterprise systems, at least some of the many required dependencies are commonly incomplete, unavailable, or operating incorrectly at the time of test execution. Some might have changed versions, and others might be using inaccurate or expired test data. The result is timeouts, incomplete tests, false positives, and inaccurate results—preventing you from delivering the fast quality feedback expected with test automation.
Service virtualization can help you overcome this plateau and increase test automation rates.
What Is Service Virtualization?
Service virtualization is a simulation technology that lets you automatically execute tests, even when the AUT’s dependent system components (APIs, third-party applications, etc.) cannot be properly accessed or configured for testing. By simulating these dependencies, you can ensure that your tests will encounter the appropriate dependency behavior and data every time they execute.
Service virtualization is commonly used when integration tests or end-to-end tests need to interact with dependent system components that are:
- Unreliable, evolving, or not yet completed
- Beyond your scope of control (e.g., operated by another company or division)
- Available for testing only in a limited capacity or at inconvenient times
- Challenging to provision or configure in a test environment
- Simultaneously needed by different teams with varied test data setups and other requirements
- Too restricted or costly to use for automated regression testing
Stabilizing Automated Tests
For automated tests to execute successfully, all the dependent systems must be available with the appropriate configuration, functionality, and test data—all at the same time, every time the automated test suite executes. This is a tremendous challenge.
When an automated test suite’s execution is impeded by timeouts, incomplete tests, false positives, or other testing problems, it’s often a symptom of test environment stability issues. With service virtualization, you can stabilize access to dependent systems so that tests can execute completely, reliably, and continuously.
For example, assume that you need to execute an end-to-end test that interacts with an order processing system beyond your immediate control. However, that order management system is continually being upgraded with new functionality that’s irrelevant to your tests. As a result of those frequent updates, that dependency is often unavailable or unstable.
If you use service virtualization to simulate the small sliver of dependency behavior and data that’s required to execute your tests, you eliminate the risk of dependency issues interfering with your automated test execution. The more your tests are isolated from the various dependencies they interact with, the greater the chance that your automated test execution will proceed as planned.
By simulating dependencies in this manner, you can also trust that your test failures stem from issues with your AUT, not problems with your test environment; and you can reliably re-create the test environment for defect reproduction or bug fix validation.
Automating Complex Test Scenarios
Once the initial automated test suite is running like a fine-tuned machine, the next goal is often to automate more advanced test cases. However, it can be considerably more challenging to eliminate dependency issues for complex scenarios that involve stateful transactions than for simpler test cases that simply search for data or add a new object.
For example, assume you are responsible for testing an account management system that interacts with a CRM beyond your scope of control. You need to test a scenario that:
- Loads an existing customer account and checks that pricing details are appropriate for their current address
- Pays the customer’s full account balance based on their current address
- Changes the customer’s address to a more expensive area
- Reloads the customer’s account details and validates that the pricing details are updated appropriately based on the new address
- Validates that a) an additional amount due is added to the customer’s account, and b) the account status changes from “paid in full” to “payment due”
With traditional service virtualization approaches, it would be difficult to simulate the dependencies involved in executing this test case. But this is where orchestrated service virtualization—a special type of testing that is driven from the perspective of the test—shines.
Here, it lets you accurately simulate the various stateful customer account updates so that you can test without having to actually interact with (or configure) the back-end CRM system. It can also eliminate any associated system delays (e.g., waiting for the address update to enter the system and the pricing details to update), which could otherwise introduce a bottleneck into the automated testing process.
Basic service virtualization scenario:
Advanced service virtualization scenario:
Orchestrated service virtualization is sometimes called test-driven service virtualization because it focuses on simulation from the perspective of the test and places the tester at the center of service virtualization asset creation and management.
Service virtualization also provides a simple way to test how your AUT behaves against edge cases and error conditions that would be difficult to configure in a staged test environment. For example, assume that your account management system interacts with multiple dependent systems (a CRM, location system, and order processing system), and you want to automate tests that validate how your AUT reacts when different combinations of dependent systems are down, delayed, or behaving incorrectly. Or, assume that you want to automate a test that validates how your AUT reacts when its expected messages are sent or received in an incorrect order. Service virtualization helps you simulate these conditions so that you can automate the broad range of tests required to effectively cover your risks.
Applying Service Virtualization to Boost Test Automation
For an idea of how these strategies are applied by testers in the field, consider the following example from one of the largest European communications providers.
Testers in this organization already had the challenging task of testing a rather complex sales-to-activation process. This process started with various touch points (e.g., a mobile application as well as a web interface) and processed the order through the full end-to-end system chain, which included back-end order processing systems, a CRM, and various billing and provision systems. These transactions involved more than two hundred dependencies communicating via HTTP and SOAP, with both ESB-driven and batch transactions. Further complicating testing, the process involved a huge number of variations, including private, small-business, and enterprise segments with distinctly different product characteristics.
As you can imagine, this was always a difficult system to test, even with four- to eight-week testing windows. However, once the team adopted two-week agile sprints and expected continuous testing, they realized they needed a new approach: a way to dramatically accelerate testing speed without compromising their high quality standards.
They knew that increasing test automation was essential for achieving this goal. However, their test automation efforts were limited due to test environment instability that caused false positives and interrupted test execution. To move past this barrier, they needed a reliable and cost-effective way to ensure that required dependencies were all available for testing—with the proper test data—every time an automated test executed.
Their first step in tackling this challenge was adopting service virtualization. They initially used service virtualization to simulate behavior associated with unstable systems in their test environment. They created service virtualization scenarios for their test case portfolio by recording the service requests—essentially, learning from the existing systems with minimal effort.
Committed to advancing automation, the team proceeded to apply a more advanced level of service virtualization. This time, they began by focusing their service virtualization efforts on all the dependent system behavior required to test. Starting with their two most important systems of engagement, they simulated all required dependency behavior based on their existing test case portfolio and service virtualization recordings.
By doing this, they fully decoupled each system, enabling a full integration test as part of their continuous integration and continuous deployment process. As a result, they could deploy rigorously tested systems to their staging environment for the final end-to-end tests.
Building upon the success of their initial efforts, they soon expanded the approach from the original two systems to forty-two systems. This enabled them to transition from their original four- to eight-week testing windows and complete testing within two-week agile iterations.
Ensuring Fast, Quality Test Feedback
If you can guarantee that all the dependent systems associated with your end-to-end tests will always be available, operating correctly, and configured with appropriate test data every time your automated tests execute, you might not need service virtualization. But for everyone else, it’s vital for achieving the sustainable, scalable test automation required for continuous testing, agile, and DevOps.
Providing the team with fast, quality feedback is one of the top goals of test automation. The goal of service virtualization is to ensure that test environment issues don’t impact the speed, accuracy, or completeness of that feedback, so you can satisfy business expectations for quality at speed.
User Comments
Nice article. Thanks for sharing.