Challenges in a Product Testing Environment

The software industry in India has gone through significant changes during the last two decades. From a largely on-site services driven market we have come a long way to be recognized now for rarer skills such as project management and product development engineering. test engineering is an extremely key aspect of product engineering and it has been a major challenge for many companies the world over to do this right so that they can get high quality products delivered to the marketplace. One of the major reasons why test engineering continues to be a challenge is related to how it is managed rather than due to technology factors.

This paper presents some experiences of Novell Bangalore in dealing with test engineering challenges as part of their product engineering delivery responsibilities for key Novell products. The paper identifies major challenges that were faced, some strategies that were used to overcome these challenges, experiences and results. A survey of test engineers done through an in-house designed questionnaire gave valuable inputs on people perspectives as well as insights into many of the management processes that are critical for success.

1. Introduction
The software industry is characterized by a constant ferment, fuelled by competition and yet the industry is only in its infancy. Being only average is not enough in an environment such as this, which is filled with constant innovation. Businesses want to be and need to be the best of the best. Needless to say quality is a key differentiation in achieving this and software testing has a major role to play here.

Software testing is an integral part of product engineering just as software design and development are. Further, it is also a complete engineering discipline with its own life cycle and so, should, perhaps be called software test engineering and not just software testing. However, this fact has not been recognized by many companies, both in India and around the world. This has contributed to less than satisfactory products being delivered to the marketplace as well as dissatisfied customers. The importance given to software design and development engineering, more often at the cost of test engineering, from a management perspective, has only made matters worse. There are obvious challenges to be overcome to get software test engineering its rightful place in the software industry today. These mostly relate to management issues rather than technology issues. This paper deals with some of these challenges as well as some experiences in dealing with them at Novell's software engineering center at Bangalore.

2. Background
Novell's software engineering center at Bangalore is a wholly owned subsidiary of Novell Inc. in the USA and is the largest engineering center for Novell outside of USA. It has complete responsibility for developing, delivering and supporting some of Novell's key products and for major components for some of the other products. It was established in late 1994 and has, in the last 6 years, established tremendous credibility for delivering complete products with a high level of quality to Novell's customers worldwide.

3. The Challenges
It is difficult to prioritize the many challenges involved in setting up and managing a successful test engineering organization as part of a product engineering organization. However, we have made an attempt to capture some of the most important challenges that we have faced. We believe there are issues that are common to many other product engineering organizations in India.

• Managing perceptions about Test Engineering and Test Engineers
• Building and nurturing a Test Engineering organization
• Developing Test Engineering as a career
• Recognizing and rewarding Test Engineers
• Test Engineering Practices

3.1 Managing perceptions about Test Engineering and Test Engineers
This was perhaps the biggest challenge (and in some ways continues to be a challenge) that we have faced. In this respect one needs to appreciate that the view of entrants into the industry plays a very important role, especially, if they are to be seen as part of carrying a product vision or on a larger scale, a company vision forward. There has been a general lack of awareness of the function of testing. What little was known by a person who just graduated from college was limited to textbooks or none at all, whereas a lot more awareness was there about the roles of developers. The role of a test engineer does not evoke the same response from a peer group as that of, say, a development engineer. Within an organization, the test engineer is very often not seen on par with a development engineer. And, finally, many new engineers use testing as a means to transition to another area of software engineering.

A key aspect of the problem is also caused by management in some organizations putting fresh engineers into testing without proper training or background. There are not many institutions in the industry that provide training on test methodologies to fresh engineers. This has given rise to an impression in the industry that anyone can do testing. Since this discipline is also not understood or appreciated, many view this as a thankless job.

The process and importance of test engineering as well as the fact that it is a core technology area has yet to be understood and appreciated fully. It is yet to be comprehended that the function of test engineering and a test engineer goes far beyond just executing test cases and finding defects. The fact that test engineering is a discipline with a complete life cycle, that includes test requirements planning and strategizing, test design, test development, test automation, test execution, test results analysis etc., is understood by very few people. test engineers also have a major role to play in determining and influencing the quality of the product. There is a great need for education at all levels on these aspects before one even attempts to change these perceptions.

There is also an impression in the industry that testing is the job of a separate team. This is the job of everyone involved in the product–for example, unit testing and responsibility for specific units of the product is the responsibility of the concerned developer. But this is not well appreciated.

These perceptions have contributed to significant peer pressure for test engineers in the industry forcing them to look at moving out of their test engineering roles as quickly as possible.

3.2 Building and nurturing a Test Engineering organization
In some ways, this challenge is imposing considering the first challenge discussed above. It is mandatory for a product engineering organization such as Novell to have dedicated and independent test engineering teams that comprehensively test products before they are delivered to their customers. The reporting structure plays a vital role in ensuring that the test teams think differently and feel empowered to stop product shipment if they believe that the product is not ready from a product quality perspective. Further, test engineering has to be developed as a core competence for the organization if they are to be successful in delivering quality products. As a result there is a need to build and nurture an organization that has complete responsibility for the test engineering function for a product. There are various aspects one needs to look into for this. The first aspect is related to recruitment of test engineers. This is a big challenge because of perceptions about test engineering that was discussed in 3.1. The other factor is getting people with the required skills and attitude. Domain and product knowledge are very critical for testing products and such skills are not readily available in the industry.

The second aspect is related to test engineers' motivation. The work done by test engineers is often viewed as being monotonous, repetitive and something anyone else can do (but not me!). Team motivation is a challenge for both test team leads and project managers at all times. Typically, for a product, there is the initial excitement and enthusiasm of testing something new. The momentum is there for sometime during the first few cycles of testing. Once the product stabilizes, the testing tends to get repetitive. At this point of time the challenge is to force people in the team to see things differently, calling for creativity and a renewed energy towards testing the product. Diversity in team composition plays a vital role here. Actions that team leads take to check regressive trends can put the team back on track.

Other aspects that are critical for ensuring success include the independence of the test organization, the organization's involvement and integration with the product life cycle activities, and their say in determining readiness of a product for release based on test results. Finally, another critical aspect is related to providing a career for test engineers. This has been discussed as a separate item in the next section.

3.3 Test Engineering as a career
The most widely asked question by a test engineer is, what value addition do I get in being a test engineer? The answers to this question are varied. Most persons with experience, typically 8+ years, in the industry appreciate this better than others since they have seen the whole life cycle of a product. The most common reason cited is that it helps them gain a good insight into a product both technically and from a customer perspective. It helps in building up domain knowledge as well as test environment knowledge.

To someone who is relatively inexperienced, typically 3 years or less, the perception is that little or no value addition is done to one's personal self in being a test engineer. In terms of technology and in terms of the nature of work, test engineering is not viewed as an appropriate career choice.

The real challenge is to appreciate that test engineering is a viable career option in a product engineering set-up. As mentioned before it is a core competence for the organization and one that provides the organization a competitive advantage in the marketplace. This fact is not understood by many organizations making it very difficult for them to recruit and retain good test engineers.

3.4 Recognizing and rewarding Test Engineers
Recognition both to the test engineer and to the importance of test engineering has not been very encouraging in the industry today. Many engineers feel left out at the end of a product release with the spotlight focussed on the other groups. The fact that test engineers do at least an equally critical job as development engineers do is not appreciated. If test engineers are not recognized and rewarded in a timely manner, it will be very difficult to motivate and build good and strong test engineering teams.

3.5 Test Engineering Practices
Having to understand a product for a competitive market in its entirety – both technically and from an end user perspective is a challenging task for a test engineer, in order to do justice to both the work profile (of testing) and to the individual. Rapid changes in products in terms of requirements and the technical environment of testing, challenges test engineers to identify these changes fast enough and adapt even faster. To do their role effectively, test engineers need to keep pace with new features added, knowing them in depth, visualizing appropriate test scenarios, developing test cases and implementing them.

Test engineering is a complete life cycle that includes test planning, test design, test development and automation as well as test execution. There is a need for processes and best practices for all these aspects. Most companies focus on test execution resulting in poor quality products coming out of testing as well as de-motivated test engineers. It is not only important to have good processes and practices in place but there is also a need for mechanisms to institutionalize them.

There is also a major challenge in some of the softer aspects and processes. For example, communication between development and testing teams is a key factor and is often hampered by factors such as lack of involvement of test engineers in the early stages of product development, physical separation between development and test teams, lack of teaming initiatives at a product level etc.

4. Organizational initiatives to address the challenges
To address the challenges identified Novell Bangalore has initiated and implemented several initiatives during the last few years. These have helped keep the motivation levels of test engineers high as well in ensuring that product quality expectations have been exceeded.

Major initiatives taken by Novell Bangalore in building a strong test engineering organization to support our product development initiatives include:

• Staffing key senior personnel (specifically some of the test leads) from within and making them as role models for other test engineers.
• Involvement of test engineers in making release decisions–this has helped in ensuring that the test engineers take complete ownership for the product and for them to feel responsible for the decisions and quality of the product
• Major teaming initiatives, both within the test teams as well as between test teams and other engineering teams such as development engineering, configuration management and so on. The organization has implemented a specifically tailored team building program for this purpose that has been very successful.
• Knowledge sharing initiatives through informal means as well as periodic formal events.
• Encouraging innovation in test strategies that has helped in making test execution less monotonous and in optimizing both schedule and resources needed for testing.
• Customer site visits by test engineers–this has helped in promoting an excellent knowledge of how Novell's customers deploy our products and what their production environments are so that we will be able to recreate them in our labs in Bangalore. In this respect it must be mentioned that Novell Bangalore has built up a very sophisticated test lab with 900+ systems networked to provide test set-up that is comprehensive enough for a good majority of Novell's customers world wide.
• Identifying test engineering as a core competence (a bundle of skills, technologies and assets critical for our success as a product engineering organization) and defining and implementing a plan for the same.
• Defining specific job responsibilities for test engineering roles at various levels to promote a career progression plan for engineers taking up these roles.
• Treating test engineers on par with development engineers in the organization–incidentally test engineers consider this as a major recognition of their criticality to the organization.

5. Results
As a result of the several initiatives undertaken, Novell Bangalore has seen results that are both tangible and intangible. For example we have noticed major improvements in quality of products delivered to the marketplace as evidenced by feedback from customers received on the product as well as post-release defect densities for products engineered and released by us. Novell Bangalore's credibility as an engineering organization with our parent organization in the US has seen a very steady and significant raise over the years. The proof of this is the increasing levels of responsibility that has been bestowed on us from inception. From just delivering some non-critical components to Novell's products when we began in 1995 to complete engineering ownership for several products and major components for other products today, we have come a long way and are today viewed as an extremely critical software engineering center in Novell.

While the above results are gratifying, we felt the true test for our management initiatives would be to check what people perceive about the organization and the roles. We did this through a survey of about 35 odd test engineers in the company. The survey questions focussed mainly on perceptions of test engineers and test engineering roles. The questions elicited one of 5 choices as follows (1) Strongly Disagree (2) Disagree (3) Neither Agree nor Disagree (4) Agree (5) Strongly Agree. Please see the Appendix for a complete description of the questions and % responses for each option for each question. Our analysis of the responses to these questions revealed the following interesting observations:

• Test engineers, in general, are able to identify themselves with their roles and believe strongly that test engineering is very critical to ensure quality of products delivered
• Doing product testing greatly enhances one's knowledge about the product
• Test engineering is seen as at least as critical as development engineering
• There is good communication within the test teams as well as with other teams responsible for the product
• Test engineering provides good value addition to the careers of test engineers
• Test engineers have the freedom to exercise their creativity in their jobs

While these are quite positive, there were important areas that need attention as well.

• Test engineers are not actively involved right through the product life cycle as they should be
• Training provided to test engineers is not adequate
• The quality of product that enters testing is an area of concern for many test engineers implying that we need to tighten up the engineering and review processes prior to testing.

6. Moving forwards
Moving forward, we as an organization have decided that we need to focus on the following areas–basically those where the survey has shown us to be needing improvement.

1. Training test engineers regularly and adequately in Testing methodologies, new and emerging testing technologies as well as on the domain areas of relevance to them.
2. Involving test engineers pro-actively at all stages of product development.
3. Improving the recognition given to test engineers
4. Tightening up software engineering processes before testing to ensure we have better quality products entering testing–this also will mean better enforcement of entry and exit criteria across various phases in the life cycle
5. Improving the communication process across the various engineering teams (test engineering teams and other engineering teams) in order to create better understanding and working relationships

In addition we have felt the need to extend the type of survey we have done with test engineers to some thing we do across the whole organization so we get a comprehensive perspective of the entire function and its impact on the organization.

Acknowledgements
We would like to thank all the persons who took time to respond to the survey and express their views on the topic. We would also like to thank Srinivasan Desikan, one of the senior test managers in the organization for taking time to review the paper and give us his critical inputs.