How to Hire a QA Person

What criteria do people use to select QA engineers? It's natural to think that the right kinds of people to hire are people just like you-but this can be a mistake. In fact, every job requires its own unique set of skills and personality types, and the skills that make you successful in your field may have significant differences from the skills needed for the QA job. 

If you read many job posting specifications for QA roles, you'll find that they commonly describe skills that are much more appropriate for a developer, including specific knowledge of the company's unique technology. Some specifications are so unique and lofty that it seems the only qualified candidates would be former heads of development!

Realistically, the QA person you seek should have the adaptability, intelligence, and QA-specific skills that will enable them to come up to speed on your project quickly. Relevant experience includes testing procedures, test writing, puzzle solving, follow-through, communication, and the "QA mindset."

Unless they are testing a programming interface or scripting language, a QA person's role is to test the product from the end user's perspective. Contrast this with developers, who look at the product from a code perspective. Consider the difference between being focused on making the code perform in a very specific way and wondering what would happen if you did "this" instead of "that" through the user interface.

It's remarkable that the people who are assigned to interview QA candidates tend to be anything but QA people themselves. Most often, developers and HR people do the bulk of the interviewing. Yet QA is a unique discipline, and candidates need to be evaluated from a QA point of view, as would accountants, advertising staff, and other specialized professionals. QA people often have the feeling that they need to have two sets of skills: those that interview well with development engineers, and those that they actually need once they get the job.

What Not to Do
The first mistake you can make is to assume that you don't really need a QA person. Code-based unit tests do not represent the end user's interaction with the product. If you tell your boss that you "just know" it works, or base your assumptions on unit tests, she probably won't feel reassured. Before the big rollout, she is going to want metrics, generated by a professional.

The second mistake is to conduct the interview as you would for a development position. Even though more and more QA people are getting into programming, most of them aren't developers. If you give most QA people a C++ test, they will fail.

Quite often, developers are tagged and thrown into a room with a QA candidate just to round out the interview process and make sure that everyone on the team feels comfortable with the choice. But many developers only know how to interview from a developer's perspective. When asked to interview someone, they will usually give them a programming test, which might eliminate candidates who have the best QA skills.

Unless they are testing from the API level, most QA people don't go near the code. They approach the product from a user's perspective. You are not looking for a programmer; you are looking for someone to represent the user and evaluate the product from their perspective.

What QA People Do
If the actual requirements of QA almost never involve any experience with the programming language, environment, and operating system, and very little to do with the type of program being created, what criteria should we be looking for? If QA people aren't programmers, what do they do?

1. They Are Sleuths. Perhaps most important, a QA person needs to be an investigator in order to define the job and understand the project. There may or may not be a product specification (spec) available defining the project. Too often the spec is nonexistent, bare bones, or woefully out of date. Furthermore, the difference between the original bare-bones spec and the current but undocumented reality is known and discussed only in private development meetings at which QA people are usually not present. QA is usually not deliberately excluded, just overlooked because development's focus is to accomplish the task, not necessarily to share their information with everyone.

Thus a QA person needs to have the investigative skills to seek out information through all available means: manuals, specs, interviews, emails, and good old trial and error. What is the product really supposed to do? What is the customer expectation? How will management know when the product is ready to ship? What measurable standards must be met? What are the individual developers working on now and what are they most concerned about? This investigation is the job of all QA people. Some experienced developers may find this in conflict with their experience, as some organizations set development tasks in a hierarchical way, with job specifications coming down from the architect and individual contributors dealing with specific focused subsets. It may seem natural to expect QA to work the same way, but in fact each QA person needs to be an independent investigator with a broad picture. Where developers are making code conform to specifications, QA people are looking for the needle-in-a-haystack problems and unexpected scenarios, in addition to verifying that the product actually works as expected.

2. They Know How to Plan. A QA person needs to plan and set priorities. There is a definable project to test. Given all the possible combinations of expected uses, as well as all the potential unexpected scenarios including human and mechanical failure, one can imagine an infinite number of possibilities. Organizing one's activity to get the most effective results in the (usually) limited time available is of paramount importance.

Further, this is an ever-changing evaluation. In ideal circumstances, QA is on pace with or even ahead of development. QA should be included in the earliest planning so that at the same time developers are figuring out how to build the code, QA is figuring out how to test the code, anticipating resource needs and planning training. But more likely, QA is brought to the project late in its development and is racing to catch up. This requires planning and prioritization with a vengeance.

Consider also that each new build represents a threat to established code. Code that worked in previous builds can suddenly fail because of coding errors, new conflicts, miscommunication, and even compiler errors introduced in the latest build. Therefore, each new build needs to be verified again to assure that good code remains good. A useful prioritization of tasks would be to

• spot-check the new build for overall integrity before accepting it for general testing
• verify that new bug fixes have indeed been fixed
• exercise new code that was just added, as this is the area most likely to have problems
• revalidate the established code in general as much as you can before the next build is released

Outside of straightforward functional testing, there may be requirements for performance testing, platform testing, and compatibility testing that should run in environments separate from the standard development and test environment. That's a lot of testing to manage. QA people have to be able to react at a moment's notice to get on top of sudden changes in priority, then return to the game plan again after the emergency has been met.

3. They See the Big Picture. A QA person needs the "QA mindset." Generally, a development engineer needs to be a focused person who drives toward a specific goal with a specific portion of the larger program. Highly focused and detail-oriented persons tend to do well at this. QA, however, is not a good place for a highly focused person. QA in fact needs to have multiple perspectives and the ability to approach the task at many levels from the general to the specific, not to mention from left field. A highly focused person could miss too many things in a QA role by exhaustively testing, say, the math functions, but not noticing that printing doesn't work.

4. They Know How to Document. A major portion of the QA role involves writing. Plans need to be written, both the master plan kind and the detailed test script kind. As the project evolves, these documents need to be updated as well. A good QA person can write testing instructions so that any intelligent person with basic user skills can pick up the script and test the product unaided. Bug reports are another major communication tool and QA people need to have the ability to define the bug in steps that are easy to understand and reproduce. It would be good to ask a candidate to bring samples of bug reports and testing instructions to the interview. Lacking these, look for any technical
writing samples that show that the candidate can clearly and economically communicate technical subject matter.

5. They Care About the Whole Project. It's also important for the candidate to have a passion for getting things right. Ultimately, QA is entrusted with watching the process with a big-picture perspective, to see that it all comes together as well as possible. Everyone has that goal, but most are too busy working on their individual trees to see how the forest is doing. QA candidates should exhibit a passion for making the project successful, for fighting for the right thing when necessary, yet with the practical flexibility to know when to let go and ship the project.

How to Hire Right
So how do you evaluate a complete stranger for QA skills?

Here's one idea. Find a simple and familiar window dialog such as a print dialog, and ask your candidates to describe how they would go about writing a test for it. Look for thoroughness and for the ability to approach the items from many angles. A good QA person will consider testing that the buttons themselves work (good QA people don't trust things that are supposed to work without question), then that the functions are properly hooked up to the buttons. They should suggest various kinds of print jobs. They should suggest testing the same dialog on various supported platforms and exception testing if the network is down or a printer is out of paper. They should mention the appearance and perhaps the working of the dialog. Performance testing may also come up, as well as the handling of various kinds of content. The more variations on a theme they come up with, the stronger a candidate they are.

Another idea is to present them with a function-testing scenario in which there is no specification from which to develop a test plan. Ask them how they would learn about the function and the test. Their answers should include documentation, old tests, marketing people, conversations with the developers, reading the bug database, trial and error, and writing up presumptions to be presented to developers for evaluation and correction. Again, look for variety and creativity in finding solutions.

QA people need to be creative problem solvers. They like to grab onto a problem and figure out the solution by whatever means they can. They will peek at the answers of a crossword puzzle to break a deadlock. They will come up with a new solution to an old problem. They are aware of the details when finding a solution, yet they have the ability to think outside the box, to appreciate new and experimental aspects. Some successful interviewers keep one or two "brain teaser" types of puzzles on hand for the candidates to work out. Candidates are asked to solve the problem and explain their thinking as they go. Whether they find the answer is not as important. Listen to their thinking process as they work. If they are able to attack the problem from many directions and not give up after the first failures, they are showing the right thinking style. Particularly look to see if they dig into the problem with real enjoyment. A true QA person would.

Of course, QA people need to be intuitively technical. They can usually program a VCR and use most technical equipment without needing the instructions (at least for basic functionality). Other people go to them for help with technical things. Listen for examples of this in their conversation. For example, if they are computer inquisitive, they don't just use software, they tinker with it. They inquire into the details and obscure corners of functionality and try things to see how they work. They may have stories of some creative accomplishment using software differently than intended by the developers, such as using a spreadsheet to write documents.

Good QA people are always learning, whether advancing their technical skills or learning something entirely new. Listen for signs of self-directed learning in the interview.

Conclusion
Good QA people have a sense of ownership and follow-through in their work. They are directly responsible for their work and its contribution to the overall process. They are good at taking a general instruction and fleshing out the details of their work on their own. They will work long and hard at it. Let them tell stories of their achievements and successes in overcoming bad situations. Look for the passion, the ownership, and the pride.

The key thing to remember is that the kinds of skills and mindset needed for QA work is different from those needed for other roles. Spend some time getting to know good QA people in your organization and getting to know what characteristics make them successful. Seek out their opinions on what to look for. Develop a consistent interviewing approach that you use over and over so that you become familiar with the range of responses from various candidates. And for goodness' sake, use your own QA people, even the new ones, to evaluate new candidates.