Managing Outsourced Testing (On time and On Budget)

Outsource Definition
Outsource testing tasks are those which are carried out by a third party. These tasks can include test strategy, planning, execution, and or automation. Tasks are typically done offsite but may also be done on site. Additionally there is the option of integrating tasks into a current test process at any point in the project lifecycle.

Managers—Know the Options
As a QA manager it is important to know all resource options. The software world can change quickly; projects can come in and out before you know what hit. Maintaining a top-notch team and or equipment is not always possible. There is a very real shortage of talented test engineers and project managers, so building a relationship with a good third party test vendor can be a lifesaver. Outsource test engineers can also be integrated into an existing QA organization. This integration allows the chance to ramp up the test effort and get the use of additional hardware. Once the project is over, there is an opportunity to scale back the test group and the excess hardware.

Besides using the test lab vendors, test tool manufacturers may be of some help. Some test tool companies provide full service consulting with the purchase of the tool. If ramp-up time is an issue, their engineers are already up to speed with the tool and can get right to your application. Regardless of the outsource option, each effort is going to take time to get budgeted and staffed. It is important for you to allot enough time to evaluate solutions.

Determine Need —Create outsource criteria
When a project starts, ask the following questions. This will determine which areas need attention.

• Are the test personnel available to complete this project?
• What does the delivery schedule look like?

Is there enough time to hire and train staff for the project? If staffing is an issue, a testlab with strong testers can fill in an immediate need.

• Is the hardware correct?

Hardware can be a huge capital expense. In some cases, the bulk of the functional testing can take place in one or two target platforms. If the testplans are well written, then there is an option to outsource hardware configuration testing. This allows access to the latest machines, and additional test resources to test in parallel with the current staff.

• Are there product requirements or specifications?

Requirements are essential for good test planning and strategizing. This is especially true in an outsource effort where vendors have little or no knowledge of the application. Good specifications also allow for better testplans and testcases.

• Is there a test strategy or plan?

A test strategy is a document, which provides the testing approach. It typically is the first document a test group will create and is the basis for other QA deliverables. A good strategy will cover test styles, tools used, summary of test cases and what is in and out of test scope. If possible create this document in-house. If the vendor does it, then control of project shifts to the vendor. Only do this if the entire QA effort is to be outsource.

• Are there any special skill sets needed to complete my project?
• Is there a need for testers with expertise in a product area (e.g., finance etc.)?

Some software applications require explicit subject matter expertise. If the current staff does not have this knowledge, there may be a need to outsource.

• Is there a need to automate?

Automation is still an uncommon skillset, in many organizations. Even if there are experienced automation personnel, they may have time constraints. . If automation is a requirement, then this task is a candidate for outsourcing. Like hardware testing, strong documentation is important to get the most for the outsource dollar. When the project is over, you can take ownership of the testscripts and make them part of your regression suite. Check with test tool vendors in this case. They may have engineers who can do these tasks.

• Do I need to execute any stress / load testing?

If testing a client / server or web application, it is imperative there is some sort of stress or load testing. This testing will simulate user load and give you an idea of how the system will behave when the number of users increase. Stress / Load testing is typically carried out by senior technical engineers. This talent can be hard to find. Stress testing usually occurs once the application is close to shipping, which means the schedule might be very tight when the opportunity arises. This testing usually requires extra hardware as well.

• Do I have funding for outsourcing or can I arrange for funding?

Money can always be an issue. If money is tight, then oustource just the most critical items in the criteria. One creative alternative maybe to borrow resources from another department. Another is to strike a deal with a nearby college to get some interns.

Research labs using your criteria
Start the lab search by calling on peers or colleagues. Word of mouth is still the best reference. If this route does not turn-up any prospects, then take to the Internet or trade publications. The newly formed Software Quality Magazine can be a great resource. Software test labs come in many varieties with different specialties. This is why is so important to have defined search criteria. Narrow the candidates to two or three and then setup some phone interviews.

Q.V.C. (Question, Visit and Cost) Shop around for a lab

Now that you have your candidates, it is time to question, question and question. This is a key step in determining which lab to use. Here are some key points to get answered:

QUESTIONS

Ask for the following sample deliverables:
Testplan and testcases
Defect reports
Project summaries
Sample deliverable output is critical. When work is being done offsite, the written and verbal communication takes on greater importance. Face to face meetings will be few and far between. Look for items such as clearly detailed defect reports, test cases with defined outcomes and very detailed project summaries.

Ask about hardware / software
Will the project share machines with others?
What is the availability of project-specific equipment?
If using a high volume test lab, chances are they share equipment across projects. It is in your best interest to find this out. If there are going to be issues, find out if the test lab will lease whatever equipment needed.

How much hardware \ software do they have?
Get an inventory of hardware and software. Any reputable lab should have all the latest equipment. Additionally labs should have a good supply of older machines that still proliferate in the marketplace. The goal is to get as close as possible to the machine class the target audience is using. Look to get a list of the following:
CPU’s
Printers
MODEMS
Graphics cards
Network Components

Is their hardware organized and available?
A lab may have a lot of equipment in inventory, but how well organized is it? In some cases I have seen labs with parts everywhere, and machines split between labs. Ask if machines travel from lab to lab and how they keep track of the components. A good sample question may be, If they had to find a video card how would they go about doing it?

Do they use out of the box machines?
Some labs add software to machines, which may be native to their company. This software may mask problems, which an average at home user would not have. A good way around this is find out if the lab has an “out of the box” machine. These machines are ones which consumers get directly from computer manufacturers and are free from company specific software. Remember, not every machine is blessed with spreadsheets and word processors. If there is DLL the software requires from a popular application or operating system component, then the home consumer may crash and burn. If testing a widely used product, this sort of testing is mandatory.

Do the testers maintain the machines or does an I.T. staff?
If testers are always configuring machines, then they do not have time to test. If the test lab has an I.T. squad, they can keep the machines up and running while the testers are focused on testing. An I.T. team can also be quicker in turning around a machine.

Ask about personnel and view tester resumes. What are the testers Lab Technology cumulative experiences?
Be sure to see the resume of any prospective tester. The sales staff will promise all kinds of skills but the testing is still being done by individuals. Review reports and project summaries created by the testers. Look for similar testing experience or products they have tested.

Are the testers shared across projects?
If the lab is to be used sporadically, the known testers may not be available at a later date. Find out if this is the case and plan accordingly. Once you locate a good tester request that person for future projects. In fact, ask for particular testers as part of the contract negotiations.

Will they sign non-disclosures? Can testers be bonded?
Since work is done offsite or with multiple clients a non-disclosure is always advised. If your software is financially sensitive be sure to get everyone involved on your projects bonded. This can be another contract point. Don’t lose good testers due to bonding issues.

Do they have Project Management skills?
Someone in the testlab will be responsible for coordinating the test work and communications with your company. Find out if the proposed lead has any of this experience. It makes communication flow a lot easier.

Have they worked on similar project technologies?
Have the testers worked on similar projects? This can be a big help, especially if the testers in your organization are not familiar with the new technologies.

VISIT
Once the candidates have been narrowed again, it is time to do an onsite visit. The visit is critical to see firsthand how the lab is setup, what the work environment is like.

Take a lab tour.

Does the lab look like a comfortable work environment? This is an area that typically gets taken for granted. The testing is being carried out by individuals. People need an environment, which is comfortable to work. Some labs are small and do not have any windows. The better the environment, the better the quality of work.

Does the hardware seem well organized? Look around the lab. Look for different hardware. Ask them to locate some pieces. See how long it takes them.

Does the lab meet security requirements? Does the organization have specific security requirements? Make sure the lab can meet those requirements. Some requirements could be the project does not have access to the internal LAN, or no dialup access to the outside world.

Interview the testers These are the folks doing the actual work. Interview them. Find out if their previous test experiences are a match for your needs.

COST
This is where it gets tricky. Vendors like to charge for everything. Be sure to get an itemized bid to work from. The bid should contain personnel; hardware and task related cost. The bid should also include the proposed start and end dates.

Is there a flat fee or is it a per day cost? The software may be late being delivered to the lab. Is there a charge for downtime or pay only when they are working? Find out if there is a charge for testers when they go on vacation. Do they have backup if a tester goes on vacation?

If commitment is long term can you negotiate a more attractive contract? If several projects are going to the lab, try to negotiate a lower rate. Doing one project at a time is more costly.

Are there additional hardware charges? Some labs will only include basic machines for testing. If your test situation requires more than the basic find out before signing the contract.

Will the testers be working onsite or offsite? If the testers are offsite and need to be flown out, negotiate who pays the expenses.

Agree on project end deliverable Decide up-front what the project end deliverable will contain. Generally, there should be copies of the bug reports, recommendations and all correspondence between the two companies.

Is there an extra charge for a project manager? Find out if the project manager is part of the test team. Do you pay for the PM? If so, see if one of the testers can double as the PM. This can reduce cost.

Getting started and establishing a Workflow
Once the vendor is chosen, it is time to get down to testing. Setting up project workflow or framework is the key to success. Here are some tips on getting started:

Test a low risk area. If there are some serious concerns about outsource then try a low-risk area. Choose a project piece that will not disrupt the schedule if there is a bad experience. As there are success experiences, then outsource as needed.

Establish a Primary contact on both sides. If work is being done offsite, it is important that communication is done frequently and consistently. The only way to do this is establish a single point of contact on both sides. This will help eliminate miscommunications and keep the workflow going. Email is good for keeping track of every exchange. Phone calls are OK, but do not leave an audit trail. In some cases, ask for a business contact. This person can handle the non-testing issues such as the contract or personnel.

Have entire team meet face to face. It is important for the team get to know each other, especially if the work is done offsite. By getting to know one another up front it helps build team unity once testing begins. If this isn’t possible, then try to get the key contact out to the site.

Provide Documentation to the lab Any and all documentation should be given to the lab. They usually do not have any prior knowledge of the environments or application. If the project has already started, then they will need to come up to speed quickly. If you cannot provide documentation, then have someone from the team travel out to the lab to help bring the testers up to speed. This technique can really bring a team along quickly. In providing testplans or testcases to the lab, be as thorough as possible. It is important that testcases clearly layout what to test and what the expected outcome should be. Try to not leave any test outcomes open to interpretation.

Review their Testplans / Testcases. If the lab is going to provide your group with testplans, review them thoroughly. Make sure the level of detail is enough to alleviate any chance of misinterpretation. Once again every test case must have a defined result. Test case outcomes cannot be open to interpretation. Look for some basic areas to be covered such as positive, negative, boundary, and documentation test cases.

Establish daily / weekly progress meetings. Set up a regular time to get updates from the testlab. Use the time to answer questions they may have come up with, or problems they may have uncovered. In addition, be sure to get a weekly written update. These updates include defect reports, testing status and any other issues. Get a pager, in case the environment goes down while they are testing. Testers should never be idle.

Establish defect-reporting mechanism. If possible have the testers enter defects directly into the defect database. A web based tracking system may be a best bet. Another method is to send issues via email, but these can sometimes get lost. Have someone from your staff verify the bugs before they are reported to development. Be sure the bugs are reproducible before getting development involved. This will also help get development buy-in to outsource test. Developers do not like hard to reproduce bugs. If the vendors are sending the defects this way, this will reduce confidence in the vendor ability.

Project End Deliverables As the project comes to an end there should be clearly defined project end deliverables. The deliverables may include the following:

Project Summary Recommendations
Completed testcases with results
Copies of all correspondence Bug / Incident reports

Treat the testers like they are part of your staff Keep in mind that the testers are really an extension of your staff. Treat them the way would treat your own people. Some folks treat third party testers as expendable, but keep in mind these testers may be doing some of your most critical work. Treat them well.