Why You Need Continuous Testing in DevOps

The agile process is all about using short, flexible development cycles to respond quickly to customer needs. Doing this effectively these days involves building a DevOps software pipeline in order to quickly get high-quality software into the hands of your customers and receive feedback.

Most DevOps initiatives start with the adoption of continuous integration (CI) practices, where code is continuously integrated to make sure everything works together. Developers start the CI process by checking code into a shared repository many times a day. Each check-in is verified by an automated build process and some fast-running tests, allowing teams to detect errors and conflicts as soon as possible. Regression tests are run at least each night to make sure any changes made during the day did not break something else.

After CI is performed, a continuous delivery process is followed, where the application is further tested and, once it passes all the required tests, it's available to release into production. The upside of continuous deployment is that it delivers new functionality to users within minutes, as well as instant feedback to the team that allows rapid response to customer demands. Effective testing during your continuous deployment process is critical because without it, there is a big risk of continuously releasing buggy software into production. 

Don’t Let Testing Practices Slow You Down

Continuous testing, which is often called shift-left testing, is an approach to software and system testing in which testing is performed earlier in the software lifecycle. The goals are finding defects earlier, increasing software quality, shortening long test cycles, and reducing the possibility of software defects making their way into production code during deployments. Continuous testing is critically important if your company is trying to use DevOps to deploy software frequently into production.

Done right, continuous testing provides fast and continuous insight into the health of the latest build of your application. This information can then be used to determine if the app is ready to progress through the delivery pipeline at any given time. Because testing begins early and is executed continuously, bugs are exposed soon after they are introduced, which reduces the time and effort needed to find and fix them. Consequently, it is possible to increase the speed and frequency at which bug-free, high-quality software is delivered, as well as decrease technical debt.

Technical debt refers to the price organizations pay when releasing badly designed code. It's a way of calculating the cost of additional rework caused by choosing an easy and quick solution now instead of using a better, less buggy approach that would take longer. Just like financial debt, technical debt incurs interest that must be paid, such as increased maintenance, support, or legal costs. By shortening the time it takes to fix buggy software, continuous testing helps pay down your technical debt by keeping these interest costs from accruing.

DevOps is a cultural shift that promotes collaboration among all teams, including development, quality assurance, operations, and others, such as performance management, release management, and maintenance teams. Consequently, there is no single product that can be considered the definitive DevOps tool. Often, a collection of tools from a variety of vendors is used in the stages of a DevOps process. Continuous integration is often seen as the backbone of a continuous delivery pipeline, which explains the popularity of CI tools such as Jenkins and Bamboo to build, test, and deploy applications automatically when requirements change.

Companies using DevOps often ship new software into production hundreds of times every day. These companies are delivering smaller pieces of software, collaborating, and monitoring in production to create a continuous flow of code, from check-in to production. And they're using continuous testing technology to weave testing activities into every part of their software design, development, and deployment processes.

Let Tech Do the Heavy Lifting

To release high-quality code faster, your organization needs to let tech do the heavy lifting by adopting next-generation tools and practices that enable you to test early, often, automatically, and continuously.

By executing the right set of tests at the right stage of the delivery pipeline—without creating a bottleneck—these tools enforce agile principles by providing appropriate feedback at every stage of the process. This enhanced communication averts duplication of efforts and increases alignment among dev, ops, and testing teams, which will allow you to deliver software on tighter schedules.

But these streamlined schedules are only possible if test automation is seamlessly integrated into your software delivery pipeline and DevOps toolchain. Test automation works by running a large number of tests repeatedly to make sure an application doesn’t break whenever new changes are introduced. Manual testers are often still involved in DevOps projects, performing testing while an automation test suite is constantly running—but their role needs to shift toward a session-based exploratory testing approach, focused on areas with the most risk or where automation is not effective. 

The image below shows an example DevOps pipeline that incorporates continuous testing during check-ins, continuous integration, and continuous delivery.

A Continuous Testing DevOps Toolchain

While not an exhaustive list of all available DevOps products, here's a checklist of tools that together make up a viable continuous testing DevOps toolchain.

Planning Tools

If you're looking for a tool that makes it easy for different teams to collaborate, Jira is an agile project management tool that supports any agile methodology, be it Scrum, kanban, or your own unique flavor. From agile dashboards to reports, you can plan, track, and manage all your agile software development projects. Jira’s wide range of integrations also helps you connect to almost any other tool you're likely to need.

Dev Tools: Desktop or Cloud-based IDEs

While Eclipse and Visual Studio are the most popular desktop IDEs, Cloud9, developed by Amazon Web Services, and JSFiddle lead in the cloud.

Version Control Systems (VCS)

There are several web-based hosting services for DevOps version control, including Microsoft's GitHub, Atlassian's Bitbucket, and the open source GitLab service. All work within standard desktop or cloud IDEs to ease the processes of source code check-in and checkout. 

Build Tools

Jenkins is a CI/CD server that builds applications, runs tests automatically, and pushes code through your DevOps pipeline every time a developer checks new code into the source repository. Because of the rich ecosystem of plugins, Jenkins can be used to build, deploy, and automate almost any software project.

Bamboo is a CI/CD server from Atlassian. Like Jenkins and other CI/CD servers, Bamboo allows developers to automatically build, integrate, test, and deploy source code. Bamboo is a commercial software that is integrated and supported out of the box with other Atlassian products, such as Jira for project management and Hipchat for team communication.

Automated Testing Tools

Cucumber is a tool for specifying application features and user scenarios in plain text. Cucumber runs automated acceptance tests written in a behavior-driven development (BDD) style that encourages collaboration on software projects by writing test cases in a natural language that nonprogrammers and domain experts can read.

Selenium is a suite of different open source software tools that enable automated testing of web applications across various browsers and platforms. Most often used to create robust, browser-based regression automation suites and tests, Selenium, like Jenkins, has a rich repository of open source tools that are useful for different kinds of automation problems.

Agile teams can execute one-touch control of test automation from within the Zephyr platform with Vortex, Zephyr's advanced add-on that allows you to integrate with a growing suite of automated testing frameworks (including eggPlant, Cucumber, Selenium, UFT, and Tricentis) with minimal configuration. Besides being able to control the execution of thousands of automated test cases, Vortex makes it easy to automatically create test cases from test scripts and to apply insights from analytics on both automated and manual testing activities.

Session-Based Exploratory Testing

PractiTest is a test management system that supports session-based exploratory testing practices. Session-based exploratory tests are created and added to a test set during testing. These tests can be combined with other types of tests, including structured manual and automated, to maintain test suites, traceability, and test coverage.

Capture for Jira helps testers on agile projects create and record exploratory and collaborative testing sessions, which are useful for planning, executing, and tracking manual or exploratory testing. Session-based test management, a type of structured exploratory testing, is an extremely powerful way of optimizing test coverage without incurring the costs associated with writing and maintaining test cases. Like Zephyr for Jira, Capture for Jira has a deep integration with the Jira platform, allowing users to capture screenshots within browsers, record screens in Chrome, create annotations, and validate application functionality within Jira.

Deployment Tools

Longtime “movers and shakers” in the DevOps infrastructure-as-code space, Chef and Puppet are both automated configuration management and orchestration tools used to quickly spin up compute and storage instances on demand.

Test Continuously to Deliver Faster

DevOps is more than adopting the right set of tools; it's a cultural shift that incorporates testing at each stage of the agile project lifecycle. Continuous testing is key to unlocking this culture change because it helps everyone involved communicate more, collaborate better, and innovate faster.