Kerry Buckley

What’s the simplest thing that could possibly go wrong?

Integrating Selenium with Exactor

8 comments

We use Exactor to automate acceptance/regression testing for our web application. This is fine for most things, but once we started introducing DHTML and AJAX features, we found that Exactor (or more specifically the version of JWebUnit that it uses behind the scenes) couldn’t handle testing them.

The obvious candidate for testing these enhanced features was Selenium, which uses a real browser to perform its tests. We wanted to stick with Exactor though, for a few reasons:

  • Rewriting existing tests would waste a lot of time, and running two separate sets of tests isn’t ideal;
  • Some tests need to also perform operations outside of the web interface, like inserting test data into the database and running mock instances of external systems that we connect to;
  • Exactor’s simple text format for test scripts is easier to follow than tests written in Java, particularly for non-developers.

The obvious solution was to wrap Selenium within an Exactor command. We could have written (or more likely generated) an Exactor command for each Selenese command, but it seemed simpler to just use the one, passing the Selenese command as a parameter.

Of course you have to have the Selenium server running before you can use Selenium RC commands, and it seemed to make sense to do this in the Exactor test runner. Our runner has some other customisation too (mainly to make it return a non-zero status if any regression tests fail, and to report progress – eg ‘running test 42/150 (1 failed)’ – to CruiseControl’s status file as it runs), but here’s just the Selenium stuff, with everything else stripped out:

package com.bt.csam.exactor;

import java.io.FileNotFoundException;

import org.openqa.selenium.server.SeleniumServer;

import com.exoftware.exactor.Command;
import com.exoftware.exactor.ExecutionSet;
import com.exoftware.exactor.ExecutionSetListener;
import com.exoftware.exactor.Runner;
import com.exoftware.exactor.Script;
import com.exoftware.exactor.listener.HtmlOutputListener;
import com.exoftware.exactor.listener.SimpleListener;
import com.thoughtworks.selenium.CommandProcessor;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.HttpCommandProcessor;
import com.thoughtworks.selenium.Selenium;

public class CsamRunner extends Runner {

    public static final String SELENIUM_COMMAND_PROCESSOR_KEY = "seleniumCommandProcessor";

    private static final int SELENIUM_SERVER_PORT = 4444;

    private static CommandProcessor commandProcessor = null;

    private static Selenium selenium = null;

    private static boolean useSelenium;

    private static String browser;

    public static void main(final String[] args) {
        browser = System.getProperty("selenium.browser");
        useSelenium = !(browser.equalsIgnoreCase("none"));
        System.out.println(useSelenium ? "Running Selenium using browser: "
                + browser : "Ignoring Selenium tests");
        try {
            if (useSelenium) {
                SeleniumServer.main(new String[] {});
            }
            Runner runner = new CsamRunner(args[0]);
            runner.addListener(new SimpleListener());
            runner.addListener(new HtmlOutputListener(runner.getBaseDir()));
            runner.addListener(new SeleniumListener());
            runner.run();
        } catch (Exception e) {
            System.err.println(e.getMessage());
        }
    }

    public CsamRunner(final String fileName) throws FileNotFoundException {
        super(fileName);
        if (useSelenium) {
            String baseUrl = System.getProperty("base.url");
            commandProcessor = new HttpCommandProcessor("localhost",
                    SELENIUM_SERVER_PORT, browser, baseUrl);
            selenium = new DefaultSelenium(commandProcessor);
            selenium.start();
        }
    }

    public void run() {
        try {
            super.run();
        } finally {
            if (useSelenium) {
                selenium.stop();
            }
        }
    }

    public static class SeleniumListener implements
            ExecutionSetListener {

        public void executionSetStarted(final ExecutionSet es) {
            if (useSelenium) {
                es.getContext().put(SELENIUM_COMMAND_PROCESSOR_KEY,
                        commandProcessor);
            }
        }

        public void commandEnded(final Command arg0, final Throwable arg1) {
            // ignore
        }

        public void commandStarted(final Command arg0) {
            // ignore
        }

        public void executionSetEnded(final ExecutionSet arg0) {
            // ignore
        }

        public void scriptEnded(final Script arg0) {
            // ignore
        }

        public void scriptStarted(final Script arg0) {
            // ignore
        }
    }
}

Here are the important bits:

  • Line 34: read in a property which is set by our ant script, to tell us which browser Selenium should use. If this is set to the special value of ‘none’, Selenium is not used (this is really only there until we sort out getting Firefox onto our CruiseControl server).
  • Line 40: create a new Selenium server.
  • Lines 55—59: start a Selenium RC interface with the right base URL (also passed in from ant). We create our own command processor, so that we can run arbitrary commands rather than using the methods for individual commands – for some reason there’s no DefaultSelenium.getCommandProcessor().
  • Line 68: stop the Selenium server when we’ve run all the tests.
  • Line 45: add an execution set listener, so we can…
  • Line 78: …put a reference to the Selenium interface in the execution set context map when the execution set starts.

On to the command wrapper itself:

package com.bt.csam.exactor.command.selenium;

import com.bt.csam.exactor.CsamRunner;
import com.exoftware.exactor.Command;
import com.exoftware.exactor.Parameter;
import com.thoughtworks.selenium.CommandProcessor;
import com.thoughtworks.selenium.SeleniumException;

/**
 * Command wrapping a Selenium command. Specify the Selenium command and its
 * arguments as arguments to the Exactor command, eg:
 * 
 *  Sel assertValue "id=logTypeFilter" "%"
 * 
 * @author Kerry Buckley
 */
public class Sel extends Command {

    public void execute() throws Exception {
        CommandProcessor commandProcessor = (CommandProcessor) getScript()
                .getExecutionSet().getContext().get(
                        CsamRunner.SELENIUM_COMMAND_PROCESSOR_KEY);
        if (commandProcessor == null) {
            fail("Selenium is not running. Ensure the selenium.browser property is set.");
        }
        Parameter[] parameters = getParameters();
        String command = parameters[0].stringValue();
        String[] arguments = new String[parameters.length - 1];
        for (int i = 1; i < parameters.length; i++) {
            arguments[i - 1] = parameters[i].stringValue();
        }
        try {
            commandProcessor.doCommand(command, arguments);
        } catch (SeleniumException e) {
            fail(e.getMessage());
        }
    }
}

Again, the interesting lines:

  • Line 22: Get the command processor that the runner put in the context.
  • Lines 29—33 : extract the Selenium command and its arguments from the Exactor command arguments.
  • Line 35: Run the command, throwing an assertion failure if the command fails.

Now you can turn a Selenium script like this:

open http://www.google.com/
type "hello world"
click btnG
assertTitle "hello world - Google Search"

into an Exactor script like this:

Sel open http://www.google.com/
Sel type "hello world"
Sel click btnG
Sel assertTitle "hello world - Google Search"

Integration with Selenium IDE

That's all very well, but wouldn't it be nice if Selenium IDE could understand your Exactor scripts, so you could record them in the IDE and save them to a .act file, or open a .act file in the IDE and run them? What you need is a custom format, and here's one I prepared earlier:

/**
 * Parse source and update TestCase. Throw an exception if any error occurs.
 *
 * @param testCase TestCase to update
 * @param source The source to parse
 */
function parse(testCase, source) {
  var lines = source.split(/\r?\n/);
  var commands = [];
  for (var n = 0; n < lines.length; n++) {
    commands.push(parseLine(lines[n]));
  }
  testCase.setCommands(commands);
}

function parseLine(line) {
  var rtn;
  if (line.match(/^#/)) {
    rtn = new Comment(line.replace(/^#\W*/, ""));
  } else if (line.match(/^Sel\W/)) {
     rtn = new Command();
     rtn.command = line.replace(/^\W*Sel\W*(\w*).*/, "$1");
     var remaining = line.replace(/^\W*Sel\W*(\w*)/, "");
     target = nextValue(remaining);
     rtn.target = stripQuotes(target);
     remaining = remaining.replace(/^\W*/, "").substr(target.length);
     rtn.value = stripQuotes(nextValue(remaining));
  } else {
    rtn = new Comment(line.replace(/^/, "[exactor]"));
  }
  return rtn;
}

function nextValue(str) {
  if (str.match(/^\W*'/)) {
    return str.replace(/\W*('.*?[^\\]').*/, "$1");
  } else if (str.match(/^\W*"/)) {
    return str.replace(/\W*(".*?[^\\]").*/, "$1");
  } else {
    return str.replace(/^\W*(\w*).*/, "$1");
  }
}

function stripQuotes(str) {
  return str.replace(/^(['"]?)(.*)\1$/, "$2");
}

/**
 * Format TestCase and return the source.
 *
 * @param testCase TestCase to format
 * @param name The name of the test case, if any. It may be used to embed title into the source.
 */
function format(testCase, name) {
  return formatCommands(testCase.commands);
}

/**
 * Format an array of commands to the snippet of source.
 * Used to copy the source into the clipboard.
 *
 * @param The array of commands to sort.
 */
function formatCommands(commands) {
  var result = '';
  for (var i = 0; i < commands.length; i++) {
    var command = commands[i];
    if (command.type == 'command') {
      var target = command.target;
      var value = command.value;
      result += "Sel " + command.command;
      if (target != "") {
        result += " \"" + target.replace(/\"/, "\\\"") + "\"";
      }
      if (value != "") {
        result += " \"" + value.replace(/\"/, "\\\"") + "\"";
      }
    } else if (command.comment.match(/^\[exactor\]/)) {
      // non-selenium exactor command
      result += command.comment.replace(/^\[exactor\]/, "");
    } else {
      // normal comment
      result += "# " + command.comment;
    }
    result += newline();
  }
  return result;
}

function newline() {
  if (navigator.appVersion.lastIndexOf('Win') != -1) {
     return "\r\n"
  } else {
     return "\n"
  }
}

Obviously there's some fairly unpleasant regular expression hackery going on here, but thanks to JSUnit I was able to test-drive the development of the conversion functions. Here's the test suite:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
	<head>
		<title>Exactor import/export assertion tests</title>
		<script language="JavaScript" type="text/javascript"
			src="../jsunit/app/jsUnitCore.js"></script>
		<script language="JavaScript" type="text/javascript"
			src="../jsmock/jsmock.js"></script>
		<script language="JavaScript" type="text/javascript"
			src="../../../src/javascript/selenium-ide/exactor.js"></script>

		<script language="JavaScript" type="text/javascript">

// stub implementations

function TestCase(commands) {
	this.commands = commands != null ? commands : [];
	this.setCommands = function(commands) {
        this.commands = commands != null ? commands : [];
    }
}

function Command(command, target, value) {
	this.command = command != null ? command : '';
	this.target = target != null ? target : '';
	this.value = value != null ? value : '';
	this.comment = '';
	this.type = 'command';
}

function Comment(comment) {
	this.comment = comment != null ? comment : '';
	this.command = '';
	this.target = '';
	this.value = '';
	this.type = 'comment';
}

// tests

var command, command2, commands;

function setUp() {
    commands = [];
	command = new Command("command", "target", "value");
	command2 = new Command("command2", "target2", "value2");
}

function testFormatHandlesTwoSimpleCommands() {
	// assumes format() delegates to formatCommands(), so only
	// a single test is provided.
    commands.push(command);
    commands.push(command2);
	var result = format(new TestCase(commands)); 
	assertEquals("Sel command \"target\" \"value\"" + newline()
			+ "Sel command2 \"target2\" \"value2\"" + newline(), result);
}
	
function testFormatCommandsHandlesSimpleCommand() {
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"target\" \"value\"" + newline(), result);
}

function testFormatCommandsHandlesTwoSimpleCommands() {
    commands.push(command);
    commands.push(command2);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"target\" \"value\"" + newline()
			+ "Sel command2 \"target2\" \"value2\"" + newline(), result);
}

function testFormatCommandsHandlesCommandWithASpaceInTheTarget() {
	command.target="foo bar";
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"foo bar\" \"value\"" + newline(), result);
}

function testFormatCommandsHandlesCommandWithASpaceInTheValue() {
	command.value="foo bar";
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"target\" \"foo bar\"" + newline(), result);
}

function testFormatCommandsHandlesCommandWithSpacesInTheTargetAndTheValue() {
	command.target="foo bar";
	command.value="wibble wobble";
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"foo bar\" \"wibble wobble\"" + newline(), result);
}

function testFormatCommandsEscapesQuotesInTheTarget() {
	command.target="foo\"bar";
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"foo\\\"bar\" \"value\"" + newline(), result);
}

function testFormatCommandsEscapesQuotesInTheValue() {
	command.value="foo\"bar";
    commands.push(command);
	var result = formatCommands(commands); 
	assertEquals("Sel command \"target\" \"foo\\\"bar\"" + newline(), result);
}

function testFormatCommandsHandlesComments() {
	var comment = new Comment("This is a comment.");
    commands.push(comment);
	var result = formatCommands(commands); 
	assertEquals("# This is a comment." + newline(), result);
}

function testFormatCommandsRestoresNonSeleniumCommandsFromMarkedComments() {
	var comment = new Comment("[exactor]web.ClickButton\t\tfoo");
    commands.push(comment);
	var result = formatCommands(commands); 
	assertEquals("web.ClickButton\t\tfoo" + newline(), result);
}

function testParseHandlesASimpleCommandAndAComment() {
	// assumes parse() delegates to parseLine(), so only
	// a single test is provided.
	var testCase = new TestCase();
	var source = "Sel command \"target\" \"value\"" + newline()
			+ "# foo";
	parse(testCase, source);
	var cmds = testCase.commands;
	assertEquals(2, cmds.length);
	assertEquals("command", cmds[0].type);
	assertEquals("command", cmds[0].command);
	assertEquals("target", cmds[0].target);
	assertEquals("value", cmds[0].value);
	assertEquals("comment", cmds[1].type);
	assertEquals("foo", cmds[1].comment);
}

function testParseLineHandlesSimpleCommand() {
	var line = "Sel command target value";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesSingleQuotedTarget() {
	var line = "Sel command 'target' value";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesDoubleQuotedTarget() {
	var line = "Sel command \"target\" value";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesSingleQuotedValue() {
	var line = "Sel command target 'value'";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesDoubleQuotedValue() {
	var line = "Sel command target \"value\"";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesSpaceInSingleQuotedTarget() {
	var line = "Sel command 'target with space' value";
	var result = parseLine(line); 
	assertEquals("command", result.type);
	assertEquals("command", result.command);
	assertEquals("target with space", result.target);
	assertEquals("value", result.value);
}

function testParseLineHandlesComments() {
	var line = "# This is a comment.";
	var result = parseLine(line); 
	assertEquals("comment", result.type);
	assertEquals("This is a comment.", result.comment);
}

function testParseLineCommentsNonSeleniumCommandsWithSpecialMarker() {
	var line = "web.ClickButton\t\tfoo";
	var result = parseLine(line); 
	assertEquals("comment", result.type);
	assertEquals("[exactor]web.ClickButton\t\tfoo", result.comment);
}

function testStripQuotesDoesNotChangeStringWithNoQuotes() {
	var str = "foo bar";
	assertEquals(str, stripQuotes(str));
}

function testStripQuotesDoesNotChangeStringWithInternalSingleQuote() {
	var str = "foo\'bar";
	assertEquals(str, stripQuotes(str));
}

function testStripQuotesDoesNotChangeStringWithInternalDoubleQuote() {
	var str = "foo\"bar";
	assertEquals(str, stripQuotes(str));
}

function testStripQuotesStripsSingleQuotes() {
	var str = "\'foo bar\'";
	assertEquals("foo bar", stripQuotes(str));
}

function testStripQuotesStripsDoubleQuotes() {
	var str = "\"foo bar\"";
	assertEquals("foo bar", stripQuotes(str));
}
		</script>
	</head>
	<body>
		<h1>Exactor import/export assertion tests</h1>
		
		<p>This page contains tests for the Exactor import/export functions. To see them, take a
			look at the source.</p>
	</body>
</html>

Technorati Tags: , , ,

Written by Kerry

November 16th, 2006 at 2:09 pm

Posted in Agile,Software

8 Responses to 'Integrating Selenium with Exactor'

Subscribe to comments with RSS or TrackBack to 'Integrating Selenium with Exactor'.

  1. Hi Kerry,

    I was investigating this very idea lately, and I was just about to implement my own Exactor/Selenium framework, when I came across your version here, and I must say I am impressed!

    Have you made any further developments with this since this article was published in November?

    I’d also like to know your future plans for this. Seeing as you freely display your code are you planning on making it an open-source project? Is there more code than what is shown on this page? If you could send me what you have developed (along with any other information you have) it would be greatly appreciated, I hope to be able to develop this idea further for my own company’s needs.

    Regards,
    Adam Hynes
    DeCare Systems Ireland

    Adam Hynes

    24 Jan 07 at 4:17 pm

  2. Adam,

    Glad you liked it!

    I’ll have to double-check that I’m not breaking any [major] corporate rules by open-sourcing this code – watch this space. What I’ll probably do is tidy it up a bit (I’ve changed projects recently, so haven’t made any more progress since this post) and submit it to Brian Swan to see if he’d like to roll it into Exactor, but I’ll post a copy here too.

    Kerry

    27 Jan 07 at 4:22 pm

  3. That would be great Kerry, thanks a lot.

    Adam Hynes

    29 Jan 07 at 9:47 am

  4. Hi Kerry,
    Se├ín Hanley of Exoftware will be giving a talk here in Cork tomorrow. I believe this is the company behind Exactor. I’d be happy to chat with him about your idea of adding your Selenium wrapper to Exactor, if you felt it would serve any purpose.

    Regards,

    Brendan Lawlor (also of Decare Systems Ireland).

    Brendan Lawlor

    29 Jan 07 at 4:39 pm

  5. […] while ago I cobbled together some code to drive Selenium from Exactor (which was the acceptance testing framework we were using at the time). That project was offshored […]

  6. Hi Kerry,

    I went through steps you have mentioned except I don’t want to have command wrapper. After all setup when I run any of the .act file I get error message “No command found for:”. I tried running act files with “exactor -lib classes” as well. I have added all jars. Classpath and JAVA_HOME are working fine for other projects. Any suggestion? Is there any document which explains steps to verify exactor/framework setup.

    Thanks for all your help.

    KW

    8 Jul 11 at 11:41 pm

  7. Sorry KW, I’m afraid I can’t really help – it’s been four years since I last did any real development in Java, let alone used Exactor. If I had to do it again now I’d probably use Cucumber (or cuke4duke instead.

    Kerry

    9 Jul 11 at 7:40 am

  8. Hi Kerry,

    Thanks a lot for your reply. I will try Cucumber. I went though their website/wiki/forums on GitHub but I was not able to find any help reg. full installation and how to use it. Any suggestions?

    Can you please answer one more question about your code here? What argument you expect for your main in CsamRunner? Path of .properties file/.act file path which contains bunch of commands.

    Thanks,
    Kedar

    KW

    11 Jul 11 at 8:56 pm

Leave a Reply

You must be logged in to post a comment.