Endgame Testing: Exploring Your Agile Product End to End

Unlike in waterfall, there is often no single test group in agile that tests the product as a whole. Each development team is multidisciplinary and is supposed to be autonomous and provide its own testing. If there are connections or APIs to software components being developed by other teams, emulators or mocks are often used to eliminate these dependencies so each team can complete their testing in isolation.

The problem is when a company delivers a product that consists of many components, or a set of products that are interconnected, the product isn’t likely to work properly without additional end-to-end testing. While there is usually some integration testing performed between components in agile, this testing is often limited to validating integration points and subsystems. It is not usually a comprehensive functional test.

My team solved this issue by assuring our agile products flow as expected end to end, a process called endgame testing.

What Is Endgame Testing?

The scope of endgame testing depends on your product, but it should include various kinds of testing that cannot be completed during sprints: functional end-to-end, load and performance, security, integrity, and usability testing.

As described in the book Agile Testing: A Practical Guide for Testers and Agile Teams, a big part of endgame testing is performing end-to-end exploratory testing, concentrating on overall product functionality. This process should “confirm that the application is working correctly, give you added confidence in the product, and provide information for the next iteration or release.”

I consider exploratory end-to-end testing performed as part of the endgame a new type of testing. In many waterfall processes, most end-to-end tests were scripted, such as acceptance testing against requirements instead of in an exploratory manner. While scripted testing may still be part of an endgame testing process, I have found exploratory testing to be very effective for identifying defects.

But even if a waterfall team executed exploratory testing end to end, the situation here is new because for a product built in autonomous groups, this is the first chance of overall coverage while the release is one step away.

The Goal of Exploratory Testing in the Endgame

The goal of endgame exploratory testing is to evaluate the value a product gives to the customer by testing the system end to end from the user's perspective. Testing will ensure the continuity of use between components developed by different agile teams, continuity in user experience, successful integration of new features, and identification of any remaining functional issues. Endgame exploratory testing will often identify gaps that are difficult to discover inside agile teams, including flows across the product.

For example, once during endgame testing I found that a browser unexpectedly lost its ability to run Flash after installing another plug-in. It wouldn’t have been easy to find this bug testing during sprints, as it was an issue that only occurred when functionality from two teams—the installer team and add-on team—was combined. Endgame exploratory testing is great at uncovering when two components work fine when tested separately but would stop working when used together.

Ideally, bugs are not supposed to be found during endgame exploratory testing, and to be fair, most of the bugs my team found were not showstoppers—mainly UI or usability issues. But on the other hand, because this is the last test before releasing, there is always a chance of finding critical bugs.

Now that we understand the purpose of endgame exploratory testing, let's talk about the process itself.

Criteria for Successful Endgame Testing

The need for endgame testing should be considered at each release. Any stakeholder can request endgame testing—the test or development architect, the agile team itself, or the product owner.

There are many good reasons for initiating endgame testing, including when a new or complex feature that has significant risk has been added to the product, a new feature has been added to a complex product that has many interfaces to other components or products, or a feature has been updated significantly but not tested as part of recent end-to-end testing activities.

To start endgame testing, features already should have been tested in isolation and passed the definition of done for all teams involved, and any additional integration testing required should be successfully completed.

As with any exploratory test, some preparation is necessary. This includes gaining access to product information from the product owner and talking to the agile teams about the new functionality. The key to this preparation is to focus on the customer's view of the product, not the detailed technical aspects—for example, knowing the table name in a database will not help you much as an endgame tester because that won’t be relevant from a user’s perspective.

It is also necessary to plan the scope of your exploratory testing in advance in accordance with the changes that have been made to the product and any known risks, and to prepare charters for anyone who is going to be testing the product. It is recommended that testers understand the product user stories and use cases to make sure the testing is user-driven. You also might use automation as a utility to reach certain scenarios.

Mainly, you want to think about how most users will use the product and where the risky areas are. Having said that, you do not want to heavily test in one area—detailed testing should have been done during earlier testing activities.

Conducting Endgame Exploratory Testing

Endgame exploratory testing should be performed in an environment as similar to production as possible, with servers similar to those in production, using the customer's database if you can, etc. This is particularly important for products that interact closely with the environment, operating system, or third-party products.

As usual, conduct your testing at a time to avoid conflict with other tests being run during endgame testing on the same environment, such as performance or security testing.

Participants in endgame testing are typically testing engineers, but make sure to include people from outside your typical agile teams. Your regular testers are liable to lose some of the user's point of view due to their knowledge of the product’s technical aspects, so an outsider’s perspective is helpful. Endgame testers shouldn’t have special understanding of how the product was created. Stakeholders and people from a variety of positions and places in the company can test at this stage, so try to get employees with domain knowledge about the product’s users. Talk to the product manager, study user personas to understand the customers, and if possible, ask some actual end-users to help.

It’s up to the test engineers to decide the scope of your tests. If there is a complex feature with many options, the stakeholders and the product owner can choose what paths are anticipated to be more common to include in the testing.

That being said, only the critical bugs should be corrected during or after the endgame. The team whose code is responsible for the bugs should fix them and verify the fixes. It is not always necessary to repeat the endgame after the correction, at least not in full.

What Endgame Testing Is and What It Isn’t

It’s important to realize that endgame testing is not regression testing. While you will go over the old features as well as the new ones, the emphasis is on the overall usability of the system, with a focus on features that have recent changes.

Endgame testing also isn’t integration testing. Integration testing focuses on testing the interfaces between components. End-to-end testing focuses instead on testing the product across use cases and business flows.

You can perform endgame testing in tandem with continuous delivery—there should be no conflict. The development cycles are short and the product can be deployed quickly at any point. Some end-to-end activities can be performed automatically, and exploratory testing can be performed related to the changes.

If your endgame testing doesn’t find any bugs, don’t be discouraged. This can happen in a mature product that was tested well. However, take this as a challenge for next time. Find new methods for testing—read articles, use your imagination, and review customer complaints to get new ideas for testing.

If you consistently don’t find any bugs, some might ask whether endgame testing is even necessary. While time is certainly something to consider, keep in mind that your job is to put the product through its paces, end to end. It only takes one critical bug that escapes into production to damage your company’s reputation.