This article describes the disadvantages of creating and designing your own testing tools as opposed to purchasing the test tools from a commercial vendor. Creating in-house test tools present a barrage of issues and problems that are often overlooked and consequently stifle the activities of the testing team. The article analyzes some of the most salient drawbacks of creating test tools.
Some companies need to create their own test tools due to certain project requirements situations. These situations occur when existing commercial vendors of test tools do not offer products that allow record and playback for a given IT environment. However, under most situations it's not practical or pragmatic to create, design, and develop test tools given the vast number of choices for commercial test tools. Companies that embark on a mission to create their own test tools for either test management, or record/playback will need to answer some key questions:
How long will it take to create the new test tool?
Who will manage the effort to create a test tool?
Where will the requirements come from to create the new test tool?
How will the creation and development of a new test tool detract attention from existing QA activities?
How will the new test tool be tested to ensure its dependability?
Who will create the training materials for this new test tool?
Is it more expensive to create from scratch a new test tool rather than purchasing a commercial off the shelf test tool?
In addition to reviewing the presented questions above, a company that is preparing to create its own test tools should also be aware of the other issues associated with developing test tools:
Maintenance Costs and Service Support
Vendors of commercial test tools, for a fee, will offer companies various types of support levels, new software upgrades, patches, installation support, consulting services, etc. However, with the creation of in-house test tools a company would now have to create a plan and framework for supporting the test tools for bugs, defects, upgrades, or patches. This option might be more expensive and less reliable than simply paying maintenance fee to a vendor. Furthermore, the quality of service for maintenance support from a commercial vendor is more extensive and comprehensive than a company's in-house test tool support. This is because commercial vendors have more service support engineers with access to the vendor's R&D department. Additionally, vendors of commercial test tools also have a repository of known issues or limitations with their test tools whereas companies creating their own test tools may not have such repository.
Missing Online Help
In-house created test tools often ignore features such as context sensitive help, online help search, and online help index. The creation of online help features for in-house test tools offer testers a smoother transition and learning curve to the test tools. The downside of developing online features for in-house test tools is that they are time consuming and resource intensive to create.
Test Tool Knowledge Limited to In-house Testers
A company should beware when creating their own test tools. When a test tool has been created from scratch, only the developers and testers of that tool will have the necessary knowledge to repair, patch, or upgrade it. This situation can deteriorate into a problem if the company has a high tester turnover and has to hire new testers.
Not Proven
In-house created test tools are not proven and have been widely used anywhere. While on the other hand, vendors of commercial test tools have already deployed and implemented their tools. Companies that are ready to begin or have initiated critical testing activities might not be well served by test tools that are unproven and have never been deployed to other projects. Another issue to consider is that many commercial vendors of test tools offer tools that interact with one another. In-house created test tools might not offer integration of different test tools or offer test tools whose integration have been effectively tested or demonstrated to work.
Obsolescence
What happens when the in-house created test tool or no longer meets the requirements of the QA team or support the company's newer IT systems for record/playback? Will there be a new development effort? Often times a company might change its underlying IT system. This may cause an in-house developed test tool to become obsolete and irrelevant. With a commercial test tool a company might have the flexibility to upgrade its license or switch to a different test tool with some price difference if its existing test tool no longer supports the new IT system. This option is not available for in-house developed test tools.
Missing Third Party Integration
Some commercial test tools have APIs that allow them to integrate with software from other vendors. However an in-house test tool may not enjoy integration with third parties. Furthermore, the cost of attempting to integrate the in-house test tool might be prohibitive since many resources would have to be allocated to this effort.
Not Enough Time
The development and gathering of requirements to produce a test tool might take several months. In addition a company with plans to create its test tools will first have to decide on a programming language, and back-end to develop its test tools, and secondly it will have to find the necessary resources to develop the test tools. Given a company's existing deadlines, it might not be possible to develop and produce a test tool in time for the various testing phases.
Requirements
Who is going to provide the requirements for the test tool? Many QA teams have testers, or SMEs that have little or no exposure to automated test tools. These testers do not have a firm grasp of what the automated test tool should do, what its use-cases are, or how it should function. Understanding precisely how the test tools will be designed can be a challenge. Another wrinkle to gathering requirements for the creation of the test tools might be that the documented requirements are ambiguous and untraceable. This makes the test tools more prone to customer dissatisfaction and/or defects.
Differences in Opinions and Political Battles
The QA manager, SMEs, and testers who are the end customer of the in-house test tool may have a different agenda or perspective on what the features of the test tool should be. This could introduce a conflict or strife between the development team and the QA team that may not be easily resolved. This would not be an issue for a purchased commercial test tool.
