Bringing Continuous Testing to Your Organization

Businesses are focusing more and more on customer experiences as part of their go-to-market strategy, and a key piece of the customer’s experience is their ability to traverse through your software in a quick and seamless way. Consumers are more tech-savvy than they’ve ever been, and less tolerant of defects, so applications suffering from bad behavior can have extremely negative effects on your brand.

To reduce these risks, organizations are doubling down on their quality initiatives, and the software development industry is embracing continuous testing as a mainstream activity.

What Is Continuous Testing?

Continuous testing is a principle of software testing in which all your tests are executing all the time, providing continuous feedback into the quality and health of your applications.

In order to achieve continuous testing, organizations must first adopt test automation. There are many types of test automation, spanning the breadth of an application starting at the UI layer, through the middleware systems, and even into back-end systems. Understanding how to bring in these different types of test automation practices as efficiently as possible enables you to move toward the path of continuous testing.

It All Starts with Building a Test Automation Strategy

The test pyramid, popularized by Michael Cohn and Martin Fowler, identifies that we should have different quantities of different types of testing activities. At the base, we want to establish broad coverage of the unit and API tests. These types of tests are easier to run in automation but often require technical skill to build. The top of the testing pyramid has automated UI tests and manual tests. We want this type of testing activity at the top because it’s the only way to ensure the customer experience.

Most people focus on the bottom and the top of the testing pyramid with an “automate what’s manual” approach for UI testing and a “developers should test” mentality for unit testing. While these practices are important, focusing on the top and bottom creates a gap in the middle, at the API layer, between the UI and the code.

This gap is only going to continue to become more problematic, with a recent report from ProgrammableWeb showing over 22,000 publicly available APIs in their directory, with the number steadily climbing. API testing is more critical than ever, and it needs to become an integrated part of the continuous testing strategy that you’re building.

Choosing the Right API Testing Solution

It all starts with choosing an API testing solution that can grow with you as your API testing maturity grows. Key features that you might need depend on your industry, application, and your organization’s unique requirements. But once you’ve chosen a tool, how do you get started?

There are three critical steps to achieve functional test automation faster.

Step 1: Establish broad test coverage of your existing APIs for automation

By building scriptless tests from web recordings and API contracts, and then using those tests to continuously validate the health of your APIs, you can ensure that the APIs are working as they were designed. Think of this as unit testing for APIs, but without the grunt work.

Not only is this a valuable testing technique, but it’s also one of the earliest types of functional test validation you can do, because service contracts for APIs are usually one of the first things that are written as new features or functionalities are created.

As an example, let’s say that the team has added some new functionality to our application. As a first step, the development team published a new service definition. In this case, a Swagger document was generated. The new service being added was a RequestLoan service, which takes a series of inputs and responds with a loan provider for the new loan.

To test this service, I could consume the Swagger YAML and create a series of clients for each individual operation.

One of those clients would be the request loan service. I could create a series of inputs, both positive and negative, to validate that the service behaved appropriately. I could then repurpose these tests for regression purposes.

Of course, while this type of testing is extremely valuable, it’s only half of the API testing puzzle, because it doesn’t validate how the APIs are actually being used. Enter step 2.

Step 2: Bridge the gap between UI and API

The second part of the API testing strategy is being able to model human usage of your application into complete API test scenarios.

You can start to bridge that gap in the testing pyramid by taking advantage of artificial intelligence to augment your ability to understand what’s actually happening behind the scenes when users are navigating your application, and interpret those behind-the-scenes transactions as API calls. This type of testing allows you align the user experience with your critical API tests.

AI is a key component of the strategy because we can reliably use AI to help us break down these communications into relationships and patterns, so we can understand the business rules of how our applications are being tested. We can bring this together with our unit-level API testing to get broad coverage of our API spectrum.

Continuing my example, you will notice that the request loan service requires inputs from other areas of the application:

While I could provide customer IDs from accounts arbitrarily, I really need them to exist in my application, so I will need to create a dynamic scenario where I first query the individual user to obtain the customer ID and account ID so that I can hand that information off to the request loan service and ensure that a dynamic scenario works as described. Ensuring that I’m working with real dynamic data ensures that that behavior that exists as a result of APIs interacting with each other can be fleshed out.

While these types of techniques will allow us to shift API testing practices left and create broad coverage of our applications in the earliest stages possible, there is a third critical component of this practice: our ability to understand and adapt to change.

Step 3: Ensure confidence through a maintainable change management process

I’ve spoken to so many people about their functional testing initiative that fell flat once their application changed. This is a common occurrence because testers spend the majority of their time building rich and great API tests, only to have them break when the application’s APIs change. This can have the cumulative effect of reducing confidence in the API testing strategy because testers are spending a large portion of their time maintaining their API tests instead of building new value.

Change management is a key piece of any functional testing strategy, and AI can be a critical enabler here as well. By automatically scanning service definitions (yes, the same service definitions used to initially create the test cases) to identify when your APIs have changed, you can understand when you’ll be impacted, and then build a template for migrating existing services to the new version.

In the first part of this example I made the statement that a new service was added to my application. This actually represents API change. Since I created the first round of baseline tests using the service definition, I can compare the different versions of the service definition to each other to identify what has changed and to build a map to update my existing test cases.

Upon reviewing the change template, it becomes clear that not only has a new service been added, but many of my existing services have been refactored. In the image above, you will notice that “GET customers” has a series of new fields that have been added. Using a change management workflow will allow you to proactively identify service changes while at the same time managing the updating of existing test cases, so you can recover from the change as quickly as possible.

This is arguably the most important practice you must establish when building your functional testing strategy. Understanding this and having a commitment to quality from the beginning will help you and your organization adopt this practice.

What about Flaky Test Environments?

It would be easy to stop there, with your excellent test automation enabling continuous testing. But say you’ve spent a good amount of time building this rich and powerful functional testing strategy, you run your tests in your environment as part of your automated nightly continuous testing process, and upon reviewing the results, you see that a large portion of your tests have failed due to a system that’s outside of your control.

Does this mean that your tests were bad? Are you now being held responsible for systems that are technically out of scope for your testing?

This is not an uncommon story. We know that functional tests can only be as effective as the test environments in which they execute. Unstable, unavailable, or just plain flaky test environments can reduce the return on investment we get from our functional testing tools. So I must, at least briefly, mention one of the best ways to stabilize your test environments: service virtualization.

Not to be confused with virtual machines (that’s hardware virtualization), service virtualization enables you to simulate the services that are communicating between different pieces of hardware. For example, think of an application calling a database. Do you actually need that database in your test environment? What if it doesn’t have the data that you need? With service virtualization, you can record the transactions with the database and then use that recording to create a simulated version of that database, with all the behaviors you want for your test environment. But of course, it doesn’t stop at just databases; it can be any type of service, such as a SOAP or REST API, or even TCP and microservices.

As part of building a sustainable API testing strategy, you’ll need to build a sustainable service virtualization strategy, and that starts with answering questions like:

• What services are good candidates for virtualization?
• How are virtual services created?
• How can I maintain a virtual environment?
• How can I deploy a virtual environment as part of my continuous testing strategy?

Service virtualization is a key enabler for a sustainable continuous testing strategy, but understanding where and when to bring it in and how to be as effective as possible is key to success.

Get Started with Continuous Testing

Now that you have a better understanding of how to integrate API testing as part of your continuous testing strategy, the next step is to get going! Get engaged with an API testing tool vendor, and start with step one. Understanding the end goal as you begin will help you make smart choices along the way.