Time to Think about an Open Source Automated Testing Ecosystem

In recent years, software development has evolved as a collaborative process not just internally but beyond the bounds of a given organization. It’s become the norm that products are now dependent on a variety of factors that each include numerous variables affecting the development process. For example, a product might be dependent on a specific app in order to function, the app itself might be dependent on a specific device, the specific device might be only available in a certain geographical locale, and so on.

In this new development model, software testing is increasing in importance as well as complexity. Today’s multitude of desktop and mobile operating systems, browsers, and devices, while on one hand, offer a great set of options for the end-user, also make it challenging for testers to ensure application and device compatibility. The industry has progressed well to handle testing’s growing compatibility needs, from the old era of physical machines to the current days of cloud-powered test environments. However, this is just one of several variables to consider. While this is a positive step, there are other unaddressed or partially addressed areas we need to consider, such as incorporating test debugging, productivity, and automation tools in order to meet today’s integrated testing needs.

Problem Statements
Companies still grapple with implementing an automated test framework that can run test scripts from multiple operating system/browser combinations with a single click of a button. An additional requirement to such a framework would be for an unattended (automated) solution to pick a new product build once it has been checked in so it can be deployed and the automation runs can be kicked off. This can occur whether or not the runs are build verification tests, a regression suite, or a set of pre-built sanity tests.

Although the cloud has largely eased test infrastructural pains, a comprehensive solution for testing across the combinations of supported devices and applications is still not very prevalent.

Oftentimes, the tester is switching between machines, running into test labs, or connecting to multiple sources to access resources such as IDEs, knowledge repositories, and test and defect management tools to report results, maintain test scripts, look at developer’s source code, develop new test code, etc. Doing so increases the number of interaction points and associated overhead.

Every tester or test group typically uses test productivity tools that specifically meet their day-to-day testing needs. While the value these tools provide is undeniable, the inconsistencies and dependencies of having them isolated in one-off local machines are also huge because a comprehensive framework that includes all tools in an integrated manner is not available as of today.

As for compatibility testing, we have come a long way since the days of the physical machine set up to the current days of the cloud. It’s become prevalent amongst product teams to leverage infrastructure as a service to pay for machines based on usage and to build machines on the go based on the requirements on hand. While this evolution is welcomed by all product disciplines, especially the testing community, have we maximized our opportunities to create an empowered environment for testers, taking into account all the variables mentioned above? Definitely not. What we really need is a tester ecosystem that supports not just automated compatibility testing but also associated automation tools to aid in the testing, debugging, troubleshooting, and test management efforts. This will create a seamless process that the tester can leverage to improve his productivity and the quality of the product under test. There are some commercial players trying to build such an ecosystem (at least partially) today, but with the evolution of open source solutions and their associated benefits, wouldn’t it be nice to have a comprehensive suite of integrated solutions built via the open source route, thus saving costs for testing organizations? In the true spirit of open source, imagine a group of testing organizations coming together to build onto existing, disparate solutions to put together this ecosystem and continue to enhance its offerings through their research and development efforts; everyone contributes either through hardware or software give-backs.

The Solution
Given the scale of operations and agility that even a small- to mid-sized organization has to achieve in the recent years, this automated testing ecosystem is slowly becoming the need of the day. There is so much raw information available within every organization and on the Internet that testers can easily become overwhelmed—regardless of their years of experience. Imagine a framework that weaves in the elements in figure 1 all under one umbrella—an umbrella that is available on the cloud, scalable, organized, cost effective when developed via open source, time saving, and helps test organizations become more agile and focus on delivering a quality product:

Figure 1 : An Integrated Test Framework

Such a framework will go a long way in alleviating the time-consuming and mundane overhead on a tester’s plate. This framework will not only empower the tester and the test team to focus on the core testing tasks but it will also improve the overall team morale, creating a win-win situation for all entities involved in the product development lifecycle and, ultimately, the end-user. If organizations begin to think along these lines and implement pieces internally through their research and development efforts, they not only take a big step towards building a next generation testing solution, but they also keep their testers occupied and motivated during the downtimes.

I truly believe that we are not far from seeing such a practical implementation, which will revolutionize the overall testing effort. We are already seeing some commercial players taking some steps and organizations, such as ours, are also kicking off such an initiative internally. However, it will take a lot of forethought, commitment, and investment across organizations to make this happen. But when done, this ecosystem will be here to stay for a long time.