Recently about Testing
Last April I attended the first annual Selenium conference in San Francisco. It was an awesome conference with a great line up speakers that included many of the original creators of the Selenium project! It was a neat opportunity to have a lot of perspective into the history of Selenium and what they want it to be in the future.
One great talk I heard while I was there was by Jason Huggins the original creator of Selenium IDE. I had just had an email discussion with a person on another project in our company about whether or not they should standardize on Selenium IDE or Selenium RC and I was interested to hear what Jason had to say about the future of Selenium IDE.
When asked whether or not one should use Selenium IDE over RC he replied with something to the effect of "choose Selenium RC hands down". He went on to say that there was even discussion among the project creators about whether or not to remove Selenium IDE from the project completely.
Very intriguing coming from the creator of Selenium IDE himself! And I have to agree with him.
Note: For those new to Selenium, Selenium IDE is a Firefox browser plugin that allows you to record a test script that includes actions (such as clicks or typing text into forms) and assertions (such as verifying text on a page) to be replayed it at a later date. Selenium RC on the other hand is an API with Java bindings (and several other languages as well). It allows you to start a browser from code and issues commands to it or make assertions from it via a Java program.
On the surface Selenium IDE is appealing. It is fast and easy to get started. Testers with no programming background can create test scripts with relative easy. In a short time you can have a suite of tests to show for your efforts.
But unfortunately the returns are very short lived. The fundamental problems with Selenium IDE (which Jason pointed out in his talk) are the inherent brittleness of the scripts, the lack of variables, conditionals, or looping support, and the lack of ability to reuse portions of the scripts.
(1) Selenium IDE scripts are brittle. As soon as the page layout changes, the scripts will break. The Selenium IDE plugin does its best to locate page elements using id values or xpath selectors, but if the structure of the page changes at all the tests break.
Selenium RC is not immune to this problem, but since the programmer is writing the selector it is easier to use selectors that are more resilient to change. For example, instead of the Selenium IDE generating an xpath selector like "//div/span[3]/div[2]/span[4]/div[2]/a" which is bound to break as soon as the page structure changes, Selenium RC allows a programmer to write something like "$('foo').down('bar')" which is much more resilient.
(2) Selenium IDE lacks the support that a programming language has for variables, conditionals and looping. Imagine you have a list on a page with a large number of items in it. You might want to assert that each expected item appears on the page. But SeleniumIDE has no capacity for repeating the same action over and over again unless you manually record that action multiple times you need.
If you have an application where the appearance of the application can vary based on the environment it is deployed to then you would have to write one Selenium IDE script for each environment. You have no way of performing "if in this environment then check this condition" kind of logic.
Other kinds of simple assertions are difficult. Imagine trying to assert that a timestamp is no less than a day old. In Selenium IDE you have no way of providing the logic to parse a timestamp and verify it falls within the expected range.
Selenium RC is executing in the context of a full featured programming language. Loops, conditionals, and variables are all at the disposal of the test writer and can be used to make rich assertions about the state of the application without needing to create many scripts that are variations on the same thing. The power and expressiveness you have in Selenium RC lets you write tests in entirely different ways and much less tediously than recording long scripts over and over again.
(3) Selenium IDE lacks the ability to reuse portions of your test. In most applications there are common steps (such as logging in or searching for an item) that are repeated in many different tests. In Selenium IDE you have to repeat all those steps for each test script you write.
Selenium RC is again executing in the context of a programming language so common portions of the test can be refactored into methods that can be reused. This dramatically reduces maintenance effort because when an application changes the tests only need to be updated in one or two places instead of being recreated altogether.
These three comparisons are really just part of the debate. There are other things that make Selenium RC attractive in the long run (including easy deployments, and the ability to interact with the application in more ways than just via a web browser). The debate boils down to the fact that while Selenium RC requires a higher learning curve and requires test writers to be programmers, the results are far less brittle, far more powerful, and dramatically easier to maintain than a massive collection of Selenium IDE scripts.
In an organization where developers have the trust and respect of management and application testing is not strictly assigned to a test team, Selenium RC is the clear choice of technology.
One great talk I heard while I was there was by Jason Huggins the original creator of Selenium IDE. I had just had an email discussion with a person on another project in our company about whether or not they should standardize on Selenium IDE or Selenium RC and I was interested to hear what Jason had to say about the future of Selenium IDE.
When asked whether or not one should use Selenium IDE over RC he replied with something to the effect of "choose Selenium RC hands down". He went on to say that there was even discussion among the project creators about whether or not to remove Selenium IDE from the project completely.
Very intriguing coming from the creator of Selenium IDE himself! And I have to agree with him.
Note: For those new to Selenium, Selenium IDE is a Firefox browser plugin that allows you to record a test script that includes actions (such as clicks or typing text into forms) and assertions (such as verifying text on a page) to be replayed it at a later date. Selenium RC on the other hand is an API with Java bindings (and several other languages as well). It allows you to start a browser from code and issues commands to it or make assertions from it via a Java program.
On the surface Selenium IDE is appealing. It is fast and easy to get started. Testers with no programming background can create test scripts with relative easy. In a short time you can have a suite of tests to show for your efforts.
But unfortunately the returns are very short lived. The fundamental problems with Selenium IDE (which Jason pointed out in his talk) are the inherent brittleness of the scripts, the lack of variables, conditionals, or looping support, and the lack of ability to reuse portions of the scripts.
(1) Selenium IDE scripts are brittle. As soon as the page layout changes, the scripts will break. The Selenium IDE plugin does its best to locate page elements using id values or xpath selectors, but if the structure of the page changes at all the tests break.
Selenium RC is not immune to this problem, but since the programmer is writing the selector it is easier to use selectors that are more resilient to change. For example, instead of the Selenium IDE generating an xpath selector like "//div/span[3]/div[2]/span[4]/div[2]/a" which is bound to break as soon as the page structure changes, Selenium RC allows a programmer to write something like "$('foo').down('bar')" which is much more resilient.
(2) Selenium IDE lacks the support that a programming language has for variables, conditionals and looping. Imagine you have a list on a page with a large number of items in it. You might want to assert that each expected item appears on the page. But SeleniumIDE has no capacity for repeating the same action over and over again unless you manually record that action multiple times you need.
If you have an application where the appearance of the application can vary based on the environment it is deployed to then you would have to write one Selenium IDE script for each environment. You have no way of performing "if in this environment then check this condition" kind of logic.
Other kinds of simple assertions are difficult. Imagine trying to assert that a timestamp is no less than a day old. In Selenium IDE you have no way of providing the logic to parse a timestamp and verify it falls within the expected range.
Selenium RC is executing in the context of a full featured programming language. Loops, conditionals, and variables are all at the disposal of the test writer and can be used to make rich assertions about the state of the application without needing to create many scripts that are variations on the same thing. The power and expressiveness you have in Selenium RC lets you write tests in entirely different ways and much less tediously than recording long scripts over and over again.
(3) Selenium IDE lacks the ability to reuse portions of your test. In most applications there are common steps (such as logging in or searching for an item) that are repeated in many different tests. In Selenium IDE you have to repeat all those steps for each test script you write.
Selenium RC is again executing in the context of a programming language so common portions of the test can be refactored into methods that can be reused. This dramatically reduces maintenance effort because when an application changes the tests only need to be updated in one or two places instead of being recreated altogether.
These three comparisons are really just part of the debate. There are other things that make Selenium RC attractive in the long run (including easy deployments, and the ability to interact with the application in more ways than just via a web browser). The debate boils down to the fact that while Selenium RC requires a higher learning curve and requires test writers to be programmers, the results are far less brittle, far more powerful, and dramatically easier to maintain than a massive collection of Selenium IDE scripts.
In an organization where developers have the trust and respect of management and application testing is not strictly assigned to a test team, Selenium RC is the clear choice of technology.
In general Selenium and JUnit mix very well. One of the few areas where I have had problems getting them to play well with each other is when I wanted JUnit to notify Selenium when a test failed.
JUnit has a couple mechanisms for catching test failures but all these methods fire after the test is torn down. Since I typically shutdown Selenium in the test tearDown() method (or its @After annotated equivalent) it is too late to ask Selenium to capture a screenshot.
My solution has been to capture a screenshot after every test instead of just trying to do it on failures or errors.
Initially I had trouble capturing the screenshots because there is also no easy way to get JUnit to tell you which test just completed. Naturally I wanted to name the screenshots after the tests.
Thankfully in JUnit 4.7 a set of test "rules" were added (see the release notes) which included a mechanism for capturing the currently executing test name.
The code for this is below. I recommend putting this code in a test base class that all your Selenium tests can inherit.
Note: I find it helpful to print the method name first followed by the class because the JUnit report is displayed method name first making the screenshots easier to cross reference.
Note: If you are running your tests on Windows and your workstation screen is locked you will get a black screen for a screenshot. I have searched long and hard and there appears to be no workaround for this on Windows. It is considered a security vulnerability. If you are running on Linux you can use virtual frame buffer to simulate a screen while running headless.
JUnit has a couple mechanisms for catching test failures but all these methods fire after the test is torn down. Since I typically shutdown Selenium in the test tearDown() method (or its @After annotated equivalent) it is too late to ask Selenium to capture a screenshot.
My solution has been to capture a screenshot after every test instead of just trying to do it on failures or errors.
Initially I had trouble capturing the screenshots because there is also no easy way to get JUnit to tell you which test just completed. Naturally I wanted to name the screenshots after the tests.
Thankfully in JUnit 4.7 a set of test "rules" were added (see the release notes) which included a mechanism for capturing the currently executing test name.
The code for this is below. I recommend putting this code in a test base class that all your Selenium tests can inherit.
@Rule
public TestName testName = new TestName()
@After
public void tearDown() {
try {
selenium.windowFocus()
selenium.captureScreenshot("<YOUR PATH>\\${testName.getMethodName()}-${getClass().getSimpleName()}.png")} catch (Exception e) {
// Swallow the exception to prevent double error reporting in the JUnit report.
}
selenium.stop()
}
Note: I find it helpful to print the method name first followed by the class because the JUnit report is displayed method name first making the screenshots easier to cross reference.
Note: If you are running your tests on Windows and your workstation screen is locked you will get a black screen for a screenshot. I have searched long and hard and there appears to be no workaround for this on Windows. It is considered a security vulnerability. If you are running on Linux you can use virtual frame buffer to simulate a screen while running headless.
Several months ago a coworker and I were working on converting manual test cases into Selenium RC tests. We were running Selenium RC on IE 7 with singleBrowserMode set to false. In that situation Selenium uses two browsers: one that executes the JavaScript of your test and one that browsed to the website under test. The former is the parent of the later, and makes calls into its DOM to perform the steps of the tests.
As we were developing we started to realize that performance was degrading severely. Tests were hanging at certain steps for thirty to forty seconds.
After some investigation my coworker determined that the main culprit was the isVisible() method in the Selenium RC API. It was taking upwards of 4 or 5 seconds per execution.
Some online research yielded the root cause. Since Selenium is running the code of your test in a separate browser (presumably to avoid cross site scripting restrictions), each DOM operation is a round trip between the two windows. This is exceedingly slow if there is a lot of chatter between the windows.
Implementing an isVisible() method is more difficult than it looks because there is not good browser support (especially in IE) for finding the "computed style" of an element (the style of the element and all nonoverlapping styles from its entire string of ancestors.)
A quick peek into the Selenium source code showed that Selenium was using its bundled version of Prototype to traverse the DOM to find each parent of the element under consideration and examine their style attributes. All these round trips on a page with a huge DOM was causing the performance problem.
We circumvented this by implementing our own isVisible() method in native JavaScript and injecting it into the page using Selenium's setExtensionJs() method:
Note: The above code is written in Groovy. Hence the multi line String.
Note: The setExtensionJs() method must be called before starting the selenium client.
Note: This version of isVisible() was written specifically for IE since that is all we are required to test against. A cross browser version would require more elaborate logic.
We had already subclassed the DefaultSelenium object in our test suite to implement several bug fixes and add some extended functionality. It was simple to then overwrite the isVisible() method with the following:
This improved performance by roughly a factor of ten and reduced the test suite's run time by 20 to 30 percent.
This technique could be extended to other methods as well if you have a similar problem.
Acknowledgments: Kudos to Jesse Lentz for his detective working in figuring out that isVisible() was the problem and suggesting a solution!
As we were developing we started to realize that performance was degrading severely. Tests were hanging at certain steps for thirty to forty seconds.
After some investigation my coworker determined that the main culprit was the isVisible() method in the Selenium RC API. It was taking upwards of 4 or 5 seconds per execution.
Some online research yielded the root cause. Since Selenium is running the code of your test in a separate browser (presumably to avoid cross site scripting restrictions), each DOM operation is a round trip between the two windows. This is exceedingly slow if there is a lot of chatter between the windows.
Implementing an isVisible() method is more difficult than it looks because there is not good browser support (especially in IE) for finding the "computed style" of an element (the style of the element and all nonoverlapping styles from its entire string of ancestors.)
A quick peek into the Selenium source code showed that Selenium was using its bundled version of Prototype to traverse the DOM to find each parent of the element under consideration and examine their style attributes. All these round trips on a page with a huge DOM was causing the performance problem.
We circumvented this by implementing our own isVisible() method in native JavaScript and injecting it into the page using Selenium's setExtensionJs() method:
selenium.setExtensionJs('''
seleniumCustom = {};
seleniumCustom.isVisible = function(locator) {
var visible = true;
var element = selenium.browserbot.findElementOrNull(locator);
if (element == null) {
return false;
}
// Check if this is a hidden input element
if (element.type && element.type == "hidden") {
return false;
}
// Check this element and all parents for hidden style
while (element != null) {
if (element.currentStyle) {
if (element.currentStyle['display'] == 'none' ||
element.currentStyle['visibility'] == 'hidden') {
visible = false;
break;
}
}
element = element.parentNode;
}
return visible;
}
''')
Note: The above code is written in Groovy. Hence the multi line String.
Note: The setExtensionJs() method must be called before starting the selenium client.
Note: This version of isVisible() was written specifically for IE since that is all we are required to test against. A cross browser version would require more elaborate logic.
We had already subclassed the DefaultSelenium object in our test suite to implement several bug fixes and add some extended functionality. It was simple to then overwrite the isVisible() method with the following:
/** Overwritten with an alternate implementation to improve performance. */
public boolean isVisible(String locator) {
return getEval("seleniumCustom.isVisible(${locator})").toBoolean()}
This improved performance by roughly a factor of ten and reduced the test suite's run time by 20 to 30 percent.
This technique could be extended to other methods as well if you have a similar problem.
Acknowledgments: Kudos to Jesse Lentz for his detective working in figuring out that isVisible() was the problem and suggesting a solution!
At work we maintain a suite of Selenium RC tests that numbers 150 strong. Before each of our releases we execute the entire suite multiple times in multiple environments against Internet Explorer (IE) 7 and 8.
Selenium scales surprisingly well in most respects but one issue stymied our team early on. Right as the selenium client would start up and launch the browser it would hang on this popup:
Access is denied? Line 857? RemoteRunner.hta? Huh?
As you probably know Selenium is unable to control a popup, like the one above, that originates from the browser. When you run selenium tests you often have to configure your browser to supress many different kinds of popups. Unfortunately IE offered no way to disable this particular type.
This popup would happen perhaps 1 out of 100 test runs. It would interrupt the test run and force us to kill the selenium process which would lose the test results for that run. It originated in the selenium codebase so there was no way to trap the error. Eh gads...
We run our tests with singleBrowserMode set to false (due to a bug with Selenium not handling cookies properly with HTTPS.) This causes selenium to launch two windows: a "master" window that runs the test code and a "slave" window that browses to your application.
Long story short our team tracked this error down deep into the selenium JavaScript code base (htmlutils.js) to a function it calls to create the "slave" window:
function openSeparateApplicationWindow(url, suppressMozillaWarning) {
// resize the Selenium window itself
window.resizeTo(1200, 500);
window.moveTo(window.screenX, 0);
var appWindow = window.open(url + '?start=true', 'seleniummainapp_window');
if (appWindow == null) {
var errorMessage = "Couldn't open app window; is the pop-up blocker enabled?"
LOG.error(errorMessage);
throw new Error("Couldn't open app window; is the pop-up blocker enabled?");
}
try {
var windowHeight = 500;
if (window.outerHeight) {
windowHeight = window.outerHeight;
} else if (document.documentElement && document.documentElement.offsetHeight) {
windowHeight = document.documentElement.offsetHeight;
}
if (window.screenLeft && !window.screenX) window.screenX = window.screenLeft;
if (window.screenTop && !window.screenY) window.screenY = window.screenTop;
appWindow.resizeTo(1200, screen.availHeight - windowHeight - 60);
appWindow.moveTo(window.screenX, window.screenY + windowHeight + 25);
} catch (e) {
LOG.error("Couldn't resize app window");
LOG.exception(e);
}
if (!suppressMozillaWarning && window.document.readyState == null && !seenReadyStateWarning) {
alert("Beware! Mozilla bug 300992 means that we can't always reliably detect when a new page has loaded. Install the Selenium IDE extension or the readyState extension available from selenium.openqa.org to make page load detection more reliable.");
seenReadyStateWarning = true;
}
return appWindow;
}Aha! Notice that when they try to resize and move the "appWindow" they put a try/catch block around the resizeTo and moveTo function calls. These methods can throw an access is denied error if the browser security settings are violated or if the page is not fully loaded. But in the first two lines where "window" is resized and moved there is no try/catch!
The root cause was that there was a race condition in which the "master" window would not always be fully loaded before this code was executed.
The fix I did was to unzip the selenium server jar file, add a try/catch around the first two lines of the method above, and then zip it up again.
Hopefully the upcoming release of Selenium 2.0 will avoid this problem, but if you are running 1.0.1 or below this may be a useful workaround.
Continue reading Selenium Mojo : "Access Is Denied" .
Recently we have been using the excellent FakeFS (fake filesystem) gem in some specs to test code that reads and writes files on the filesystem. We are using the latest release version of this gem which is 0.2.1 as I am writing this. Some of the code under test uses the IO each_line method to iterate lines in relatively largish files. But we found out quickly that is a problem, since in version 0.2.1 the FakeFS::File class does not extend StringIO and so you don't get all its methods such as each_line. (The version on master in GitHub as I write this does extend StringIO, but it is not yet released as a formal version.)
As an example suppose we have the following code that prints out the size of each line in a file as stars (asterisks):
def lines_to_stars(file_path)
File.open(file_path, 'r').each_line { |line| puts '*' * line.size }
end
Let's say we use FakeFS to create a fake file like this:
require 'fakefs/safe'
require 'stringio'
FakeFS.activate!
File.open('/tmp/foo.txt', 'w') do |f|
f.write "The quick brown fox jumped over the lazy dog\n"
f.write "The quick red fox jumped over the sleepy cat\n"
f.write "Jack be nimble, Jack be quick, Jack jumped over the candle stick\n"
f.write "Twinkle, twinkle little star, how I wonder what you are\n"
f.write "The End."
end
So far, so good. But now if we call lines_to_stars we get an error:
NoMethodError: undefined method `each_line' for #<FakeFS::File:0x000001012c22b8>
Oops. No each_line. If you don't want to use an unreleased version of the gem, you can add each_line onto FakeFS::File using the following code:
module FakeFS
class File
def each_line
File.readlines(self.path).each { |line| yield line }
end
end
end
Basically all it does is define each_line so that it reads all the lines from a (fake) file on the (fake) filesystem and then yields them up one by one, so you can have code under test that iterates a file and work as expected. So now calling lines_to_stars gives a nice pretty bar chart containing the line sizes represented by stars:
******************************************** ******************************************** *************************************************************** ******************************************************* ********
Since we're using RSpec, to make this work nicely we added the above code that defines each_line into a file named fakefs.rb in the spec/support directory, since spec_helper requires supporting files in the spec/support directory and its subdirectories. So now all our specs automatically get the each_line behavior when using FakeFS.