In a recent interview with Leslie Segal, we learned why seemingly attractive cloud-based performance and load testing tools aren't as beneficial as having your very own cloud-based test lab. You may rethink what's easiest or best for your team after hearing what Leslie has to say.
Software quality professional Leslie Segal recently sat down with us to explain the numerous benefits that building your own test lab in the cloud can have for your testing operations. From the ability to work around your own schedule, to added security controls – Leslie helps explain why your own personal lab just makes sense.
Noel : When setting up a performance test lab in the cloud, what kinds of difference in results may there be compared to setting up actual, physically machines.
Leslie : The main difference between a physical machine and a virtual machine (VM) is that the VM has an extra layer that it uses to talk to the underlying physical hardware; so excessive requests for CPU, memory, the network and I/O can skew the results of the performance test. Knowing that this is a potential issue, there are many steps that can be taken to ensure valid results from your tests.
Noel : With cloud-based testing, testers are able to set up machines in different parts of the world - what are the benefits of doing this, and are there any potential negatives involved?
Leslie : If you have a website that is accessible in different parts of the world, it's important to test from the different physical locations to understand the true performance from all of your users perspectives. If your website is hosted on servers in Toronto, but you have a lot of users accessing it from California, testing it just from machines in Toronto may present you with a limited (and incorrect) view of actual performance.
Conversely, you may have content delivery networks with multiple distributed locations, so you will want to test not only that the performance is consistent from strategic locations, but that the content itself is consistent and in sync at all of those locations. The biggest issue with performance testing from multiple locations is that the results of the tests will vary based on time of day and network conditions (of which you have no control over).
Noel : For those who may not feel ready to set up their own performance test lab in the cloud, and are thinking of using a vendor - what kinds of things should they look out for, or try to avoid?
Leslie : The biggest issues that I've had with using a vendor for performance testing in the cloud are availability of the test environment (you often need to schedule weeks in advance); flexibility of the load test environment (from how many load agent machines you can use, to needing to install any application specific client-side code, to getting the scripts you developed in your local environment to run in the cloud environment) and accessing the (behind the firewall) application under test from the vendor's test environment.
Noel : Security in the cloud has been an issue for some who are hesitant to adopt this fairly new technology. How have you seen security improve over the years, and do you see even more room for an increase in the future?
Leslie : Security in the cloud is definitely as issue, more so for applications running in the cloud, than test machines in the cloud. While it has improved, there is always room for better security. Having your own performance lab in the cloud is infinitely more secure that using a vendor's lab. With your own cloud machines, you can manage the permissions for each user, set the security as needed, set firewall rules as needed, establish a dedicated VPN tunnel, and turn off the machines when they are not in use. If there is an issue with test data stored on the machines, it can be stored on a dedicated drive that is unmounted when not in use.
Noel : What are the main takeaways that you hope those who attend your session are able to bring home to their own companies?
Leslie : I hope to provide everyone who attends the session with the information and confidence that they need to go back and set up their own performance test lab in the cloud. Even if they don't end up using their cloud performance test lab, they will see how easy it is to set up and find other uses for the cloud machines - we are setting up new machines all the time for test management, bug tracking, test automation, etc. See you there!
Founder and principal consultant at Testware Associates, Inc., Leslie Segal has consulted and taught about software quality for more than twenty years. Specializing in performance testing and test automation, Leslie has worked with organizations worldwide to implement valid, reusable, efficient, and effective performance testing and test automation and to abolish record-and-playback. Contact Leslie at [email protected].
