Don't Debug Selenium Test Failures Blindly

Selenium lets you automate testing a web application end-to-end using a real web browser.  It's a great tool for smoke tests, especially when run periodically on a continuous integration server.

However, when a test fails, diagnosing the failure can be a real challenge.  Your testing framework might report a vague failure message such as "Expected true but was false".  If you've written custom failure message for your assertion, you may get a better message like "Expected the Save button to be visible."

Of course, then next question is, "Well, if the Save button wasn't visible, then what was?"  The painful, yet all too frequent, way to answer this question is to watch the test run and observe the failure first-hand.  If you don't watch the test run, then fixing the failure becomes an exercise in "debugging blind".  This is a waste of a human's time - the test already ran and failed on the continuous integration server!

However, with a little tweaking you can make Selenium capture PNG-format screenshots and save the browser's HTML source.  These two artifacts are helpful debugging tools to have when diagnosing a failed test.  Here's why and how:

Capture a Screenshot - Most failures can be diagnosed by a quick glance at the web browser; sometimes the test was on the wrong page, a validation rule caused a failure, or the application just plain blew up with an exception.  See the Selenium interface's captureScreenshot() or captureScreenshotToString(String) methods for details.  (Note that I'm referencing Selenium RC for Java, but other languages should have the same features.)

View the Browser's HTML source - Sometimes a page looks correct, but has a problem with the underlying HTML.  In that case, use the Selenium interface's getHtmlSource() method and write the current DOM contents out to a file.  This even works for AJAX frameworks like GWT that dynamically build the UI at runtime.

Modify your test framework to capture these files whenever a test fails.  Be sure to give them meaningful names (the test name is a good start) and place them in a location that your continuous integration server will capture as build artifacts.  Then when your test fails, you'll have your assertion failure message, a stack trace, a screenshot, and a copy of the HTML in the browser to help diagnose the problem.  By investing a little bit of time to make Selenium work better in your continuous integration environment, you'll be much more productive when debugging a test failure.

Note that the base test cases included with Selenium RC for Java (SeleneseTestCase and SeleneseTestBase) include the ability to capture screenshots on failure via the protected isCaptureScreenShotOnFailure() method, however it doesn't capture the browser's HTML source.  You'll need to modify or make your own to get that functionality.