In this article, Yamini Munipalli discusses the issue of whether the selection of automated tools should come before the development of QA processes in an organization. She presents her experience with this issue and hopes to shed some light on why automated tools should be secondary.
As quality assurance becomes a best practice of organizations seeking to minimize production risks, one of the questions that has to be answered is if it more important initially to establish a process for quality assurance or is it more important to select the right automated tool? In this article, the argument is that establishing a solid QA process is more important initially than selecting an automated tool.
The advantages to establishing a process first are twofold. First, it is easier to select the right tool when you have a clearer understanding of what the tool is used for and exactly how it will be used in the application. Establishing a QA process will allow the QA team to sort out this issue. The second advantage arises out of selecting the right tool. The advantage is that because the tool is selected after a process is in place, the organization is more cost effective with its use of money.
There is only one disadvantage to establishing a QA process before the selection of the right tool and that is that it takes time for a process to develop and become repeatable. Some organizations change more quickly than others so this will vary by organization.
There have been instances where there was a heavy investment in tools before an established QA process developed. In one company, the QA team was encouraged to use the tool yet there was no repeatable process for using the tool until much later. This led to lack of usage of the tool for years leading to a waste of valuable economic resources. During this time, an analysis of the tool on a client/server application was performed and revealed that there were problems with the tool. In fact, many of the well known record/playback tools have drawbacks when it comes to automating client/server applications.
When the client/server application goes through one round of record/playback, the script will playback fine. Regression testing with the same script results in object recognition problems when the tool is coordinate based. The coordinates change the nth time the script plays back resulting in script failures. Second, if the script is recorded with the start application feature, the position of the application changes every time the script plays back resulting in script failures. Finally, it is not possible to discern whether the application is hitting the actual API of the application or just the presentation layer in n-tiered applications.
Automated tools provide a wonderful solution for web testing but one should think carefully before investing in tools for client/server applications. This leads to the main point of the article which is that to deliver quality to an organization, the QA/QC team should focus on building a repeatable process first and then address the question of which tool to use later.
The QA manager should be just that and not also a development manager. These roles are very distinct and require different skill sets and different ideas. The QA manager should have some ideas about the steps necessary to deliver a repeatable process. The recommendation is that the following steps be taken: establish a QA strategy, hire or educate people with similar beliefs related to the strategy, establish what metrics will be used to measure individual and group performance, give project estimates, establish risk mitigation strategies, gather the right tools and resources, and communicate to the stakeholders the new QA process,
Establishing a QA strategy involves determining an overall role for the QA function within the scope of the SDLC at the organization. Quality assurance units in companies vary from being true gatekeepers to merely being a brief checking point within the SDLC. True gatekeepers have much more power within the company to control the release of software. They are influential and the politics are in their favor. This is the ideal situation for a QA Analyst.
The second step is to hire people with similar beliefs related to the strategy. These people would ideally have the same knowledge of testing and be familiar with the methodology designed by the QA manager.
The third step is to establish what metrics will be used to measure individual and group performance. In this step, communication is important. It is important to communicate to the team and to the necessary stakeholders what the metrics are that will measure individual and group performance. Some common metrics are the number of defects found, defect density ratios, and the number of defects found in production versus the number found in the QA environment. There are many other metrics out there.
The fourth step is to give project estimates as needed. Providing project estimates makes good business sense as it sets expectations about when deliverables are due.
The next step is to establish risk mitigation strategies. The purpose of testing is to minimize the risk the software will provide if implemented. One of the prevalent methods used to reduce risk is to do a software risk analysis. In the article entitled, "Measuring the Risk Factor," one of the methods of software risk analysis is discussed. In that article, the argument is made that four indicators should be considered when doing a risk analysis. These are complexity, severity, expected impact of failure, and likelihood of failure.
The sixth step is to gather the right tools and resources. Selection of tools should not come earlier than this. All other features should be in place before a discussion of the right tools can take place. When considering the purchase of tools, it is important to think about what the tool will be used for and how it will be used. A decision should be made with these considerations in mind.
Finally, it is important to communicate to all stakeholders what the QA approach will be going forward and set the expectations with all the stakeholders about what the QA team can accomplish given that everything is in place.
This article has argued that a repeatable QA process should come before the selection and implementation of automated tools. There are drawbacks to using automated tools in client/server environments and one should consider carefully the potential use of the tool. Many tasks should come before selecting a tool as it is not the cure for a weak QA process. The selection of a tool should come much later in the development of the team and the team's process and especially after the consideration of risk.
