If you have automated GUI tests, it's likely that you run them serially, which takes a long time. Imagine how fast they would be if you could run UI tests concurrently.
Concurrent Is Faster than Serial
I recently finished writing a test launcher that concurrently runs four sets of automated tests for four different users in four
different test environments. Doing this means that my tests run in about one third of the time as they do when run serially.
(While this idea isn't new, I hadn't considered trying it until a presentation at the 2007 Google Test Automation Conference inspired me to make this work.)
Who Am I?
When a user logs in to a computer in my test environment, he has to give a user name and a password. Each user on a machine launches processes with a unique identifier, has a defined set of places from which he may run programs, and has space allocated on the machine solely for his use. In the course of getting my test-launcher to work, I had to learn how to manage all of this information for all of my test users.
My four users are named qa9, qa10, qa11, and qa12.
There is even a command "whoami":
$ whoami
qa11
Being Someone Else: Sudo and Root
Normally it is impossible to operate as a user other than yourself on these computers, but running tests in multiple user environments from a single controller demands this. On Unix-like systems, the "sudo" command makes this happen.
Because of how my test environments are engineered, I need to be able to use sudo to get root privileges on my test machine. Your automated tests may not require root privileges. Whatever your test environment requires, it is good practice to use only the minimum privileges required.
Effective User ID, aka "euid"
Whenever a user launches a process, that process is associated with the user running it by a "uid" (user id) value and an "euid" (effective user id) value. Both uid and euid values are used by the system to allow or forbid the processes from accessing appropriate resources on the machine. These IDs are interchangeable, but for my purposes I need to be able to manipulate euid, not uid.
I'm testing a Web application with Selenium-RC (Remote Control), but my automated tests also run a command-line utility used for setting up and tearing down test data. This "st-admin" utility is intended for use by administrators of the application, but it comes in very handy for testers, since it allows us to do some powerful things that the regular UI interface does not. Some of the things it can do are "st-admin create-user" and "st-admin delete-page," and much more.
The tricky part is that when st-admin runs, it examines its own euid to discover where it is supposed to run. If user cmcmahon launches st-admin, st-admin will run from cmcmahon's environment and in cmcmahon's environment. If the test launcher (cmcmahon) wants to run st-admin for user qa11, cmcmahon has to launch st-admin from qa11's environment with the euid for user qa11.
Programming languages represent euid in different ways. My st-admin tool is written in Perl, and euid in Perl is represented as "$>". My test launcher is written in Ruby, and euid in Ruby is "Process.euid". The shell command id() shows all the id information for the user.
Launching Simultaneous Processes with fork()
Unix-like systems are designed to run multiple processes for multiple users. The ability to manipulate euid covers the part about multiple users; now we need to cover the part about multiple processes.
The utility fork() allows us to create new processes and have those processes run concurrently. If we have sudo rights, every forked process can run as a different user in a different environment.
Find the User's Stuff with PATH
Both unix-like systems and Windows have an environment variable called PATH. PATH is simply a list of directories. The PATH on my Mac (which is a unix-like OS) is
/usr/local/bin:/usr/local/bin/ruby:/bin:/sbin:/usr/bin:/usr/sbin
The PATH on a Windows machine is found in Start/Control
Panel/System/Advanced/Environment Variables
Whenever a user launches a program with a single word, like "ruby" or "notepad", the OS tries to find the particular program in the list of directories in PATH. For my testing, I need to have the correct copy of the st-admin utility found in the PATH of the user running the tests.
My Test Launcher
My test launcher has to do three things:
- It has to send HTTP requests to a port. The port is related to the euid of the current user.
- It has to find the command-line script "st-admin" in the PATH of the particular user running the test.
- It has to launch st-admin actions. The proper environment in which st-admin is to run is determined by the euid of the process that calls the st-admin utility.
A Little Code
Here are some examples of how to look at what we've discussed so far, with user "qa11" as an example:
#echo $PATH
/home/qa11/stbin:/home/qa11/bin:/hom
e/qa11/local/bin:/usr/local/bin:/home/qa11/src/st/current/nlw/bin:/home/qa11/src/st/current/nlw/dev-
bin:/home/qa11/src/st/trunk/nlw/bin:/home/qa11/src/st/trunk/nlw/dev-
bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bi
n/X11:/usr/games
Effective userid (euid)
#perl
my $euid = $>;
print "$euid = ¥n"
>> 2010
#ruby
euid = Process.euid
puts euid
>> 2010
#shell
id
uid=2010(qa11) gid=2010(qa11)
groups=4(adm),2010(qa11),65532(machine
A Little Code
My test-launcher has these methods:
get_testcases
set_users
set_ports
set_failure_port
add_tests_to_queues
run_all_tests
Most of these methods set up data for the tests to run. They're pretty boring until you get to run_all_tests. Here is a simplifed version of my code:
@queues = ['cases1', 'cases2', 'cases3', 'cases4']
@ports = ['22209', '22210', '22211', '22212']
@users = ['qa9', 'qa10', 'qa11', 'qa12']
def run_all_tests
0.upto(@queues.length - 1) do
|index|
@queue = @queues[index]
#FOR EVERY QUEUE OF TEST CASES
@env_port = @ports[index]
@dev_user = @users[index]
fork do
#FORK A NEW PROCESS
Process.euid = 0
#BE ROOT...
fork_euid =
@env_port[1..4]
Process.euid =
fork_euid.to_i
#...SO WE CAN SET THE euid FOR THE FORKED PROCESS
path =
'/home/dev_user/stbin:/home/dev_user/bin:/home/dev_user/local/bin:/usr/l
ocal/bin:/home/dev_user/src/st/current/nlw/bin:/home/dev_user/src/st/cur
rent/nlw/dev-bin:/home/dev-
user/src/st/trunk/nlw/bin:/home/dev_user/src/st/trunk/nlw/dev-
bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin
/X11:/usr/games'
#SET A TEMPLATE PATH VARIABLE
ENV['PATH'] = path.gsub!(/dev_user/,@dev_user)
#REPLACE THE dev_user VARIABLE IN PATH WITH THE REAL USER
@queue.each do |@testcase|
#RUN EVERY TEST IN THE QUEUE INSIDE THE FORKED PROCESS
def run_test
puts "running #{@testcase}
on #{@env_port}"
puts "effective
user id: #{Process.euid}"
puts "user name:
#{@dev_user}"
puts
@content = `~/stbin/run-wiki-tests
--timeout
50000 --test-server "http: // machine.socialtext.net:#{@env_port}"
--test-
workspace test_data --plan-page "#{@testcase}" 2>&1`
#SHELL OUT TO RUN THE TEST ITSELF
end #def run_test
run_test
#RUN EACH TEST IN THE QUEUE
end #queue.each
end # fork do
end # queues do
end #run_all_tests
Make it Fast
Ten years ago the few tools that existed for testing at the UI were poorly made and very expensive. They worked, but just barely--some of the time. (Remember, first you have to make it work.):
Open source UI test tools, like SAMIE, then Watir, then Selenium, raised the quality and lowered the cost of UI testing. (After you make it work, then you make it good.)
As testing tools become more powerful and achieve better quality, it's becoming expected that some programming skill is necessary for effective automated testing. System administrator skills help a lot too.
Notice that my test-launcher code doesn't care that my UI tests are written in Selenium. It doesn't matter that my test data is in a wiki. It doesn't even matter that my test launcher is written in Ruby--it could as well have been a shell script (although Ruby has a lot of features that make this kind of scripting convenient.) All that is trivial. What matters is that I can use a program to run multiple tests in different environments simultaneously.
Knowing about computer environments and being able to do a modest bit of programming enables us to achieve the final step of making software: You make it fast.
