Kerry Buckley

What’s the simplest thing that could possibly go wrong?

Driving Selenium from the RSpec Story Runner (RBehave)

7 comments

[Update] The story runner is now included in the 1.1 release of RSpec, so a lot of the hackery mentioned below is no longer required. See http://rspec.info/ for details.

Background

A while ago I cobbled together some code to drive Selenium from Exactor (which was the acceptance testing framework we were using at the time). That project was offshored shortly afterwards, and I’m pretty sure Selenium never actually got integrated into the build (sneaking a peek at the continuous integration server reveals that even the pre-Selenium acceptance test build hasn’t run successfully for months), but I was convinced of the value of Selenium for testing non-trivial web applications.

On my next project (the Web21C SDK portal) we used Selenium on Rails heavily for automated acceptance testing. This worked well, although the fact that the tests are deployed with the application limits what you can do – you can’t interrogate the database after a test, for example, or dynamically set up stubs for systems you interface with (unless you also deploy the stubs as part of the application, and drive them from the browser).

With Mojo there’s an additional complication: as well as being accessible through a browser, most functionality is also available as a RESTful API, using digest-based authentication. To keep our acceptance tests in one place, we wanted to be able to drive a browser using Selenium, alongside other tests which talked to the server directly. The obvious answer was to use Selenium Remote Control, and I also liked the look of RBehave, which has now been incorporated into RSpec (but not released yet).

The RSpec story runner

The story runner in the upcoming release of RSpec is an evolution of Dan North’s RBehave, which in turn is based on JBehave (from the same author). In the past couple of weeks, David Chelimsky has done some excellent work on extracting the text of stories from the code, leading to the plain text story runner.

Read on for the gory details of my solution to running Selenium tests from the story runner.

Putting it all together

Here’s what I did to get Selenium working from the story runner (you need to check out RSpec trunk – I’m currently using r2791 of rspec and r2814 or rspec_on_rails).

I’m sure when the next release of RSpec comes out a lot of this will be unnecessary, and there will probably be standard places for the stories and ways of running them, but for now I’ve just hacked something together that’s simple and works. Apologies for any pieces of string or duct tape that are still showing!

The files described below are laid out as follows, relative to the root of the Rails project:

+-- lib
| +-- rspec/
| +-- rspec_on_rails/
| +-- selenium-ruby-client-driver/
| | +-- selenium.rb
| +-- selenium-server/
| | +-- selenium-server.jar
| +-- tasks/
|   +-- acceptance.rake
+-- stories/
| +-- all.rb
| +-- helper.rb
| +-- steps/
| | +-- login.rb
| | +-- selenium.rb
| +-- stories/
| | +-- login.rb
| | +-- login.txt

Here’s an example story:

stories/stories/login.txt

[ruby]
Story: Login
The front page should contain a login form

Scenario: Loading the home page when not logged in
When the user goes to /
Then the title should be ‘mojo’
And the page should contain the text ‘Welcome to mojo’
And the page should contain the text ‘Sign in using your OpenID’
And there should be a field named ‘openid_url’
And there should be a submit button named ‘login’, with the label ‘Sign In’

Scenario: Logging in with an unregistered OpenID URL
When the user goes to /
And the user types ‘some.openid.provider’ into the openid_url field
And the user clicks the login button
Then the page should contain the text ‘Could not find OpenID server’
And the page should contain the text ‘Sign in using your OpenID’

Scenario: Logging in successfully
Given a test user
When the user goes to /
And the user types ‘http://dummy.openid/’ into the openid_url field
And the user clicks the login button
Then the page should contain the text ‘Hi test’
[/ruby]

Each ‘Given’, ‘When’ and ‘Then’ corresponds to a step (an ‘And’ is just syntactic sugar for a repeat of the same type of step). The steps are defined in separate files, which can be included as required (see below). This story uses two sets of steps:

stories/steps/login.rb

[ruby]
steps_for(:login) do
Given “a test user” do
User.delete_all
User.create!(:username => ‘test’, :openid_url => ‘http://dummy.openid/’,
:fullname => ‘Test User’, :email => ‘[email protected]’,
:mobile => ’0123456789′)
# ActiveRecord::Base.connection.commit_db_transaction
end
end
[/ruby]

stories/steps/selenium.rb

[ruby]
steps_for(:selenium) do
When “the user goes to $path” do |path|
$selenium.open path
end
When “the user types ‘$text’ into the $field field” do |text, field|
$selenium.type field, text
end
When “the user clicks the $button button” do |button|
$selenium.click button
$selenium.wait_for_page_to_load 5000
end
Then “the title should be ‘$title’” do |title|
$selenium.title.should == title
end
Then “the page should contain the text ‘$text’” do |text|
$selenium.should have_text_present(text)
end
Then “there should be a field named ‘$field’” do |field|
$selenium.should have_element_present(field)
end
Then “there should be a submit button named ‘$name’, with the label ‘$label’” do |name, label|
$selenium.should have_element_present(“//input[@type='submit'][@name='#{name}'][@value='#{label}']“)
end
end
[/ruby]

Note the use of placeholder parameters, which are parsed out of the lines in the story. The Selenium steps use a global instance of Selenium::SeleniumDriver, which is created in the helper (again, see below). I originally created this in a “Given a browser” step, but it seemed simpler to just create it up front, before running the stories.

stories/stories/login.rb

[ruby]
require File.expand_path(File.dirname(__FILE__) + ‘/../helper’)
run_story_with_steps_for :selenium, :login
[/ruby]

This is just a wrapper for the textual story, which specifies which steps are required.

stories/all.rb

[ruby]
dir = File.expand_path(File.dirname(__FILE__))
require “#{dir}/helper”

Dir["#{dir}/stories/**/*.rb"].uniq.each { |file| require file }
[/ruby]

Another simple wrapper, this time to run all stories.

stories/helper.rb

Warning: This file is where all the ugly code’s hiding!
[ruby]
ENV["RAILS_ENV"] = “test”
require File.expand_path(File.dirname(__FILE__) + “/../config/environment”)

# Hack to use trunk rspec without affecting the rest of the project
$:.delete_if {|dir| dir =~ /rspec/}
$: << File.dirname(__FILE__) + '/../lib/rspec/lib'
$: << File.dirname(__FILE__) + '/../lib/rspec_on_rails/lib'

# Not sure why, but this is required when rspec_on_rails not running as plugin
require 'spec/rails/matchers'

require 'spec/rails/story_adapter'
require File.dirname(__FILE__) + '/../lib/selenium-ruby-client-driver/selenium'

Dir[File.dirname(__FILE__) + "/steps/*.rb"].uniq.each { |file| require file }

unless $selenium
$selenium = Selenium::SeleniumDriver.new 'localhost', 4444,
'*firefox', 'http://localhost:3000/', 10_000
$selenium.start
end

# Don't add an ActiveRecordSafetyListener, or it'll roll stuff back
class Spec::Story::Runner::ScenarioRunner
def initialize
@listeners = []
end
end

# Runs the story in the file with the same name as the calling ruby file,
# but with a .txt extension instead of .rb, using the specified steps.
def run_story_with_steps_for *steps
with_steps_for *steps do
# Pull the filename of the caller out of the stack. Must be a better way.
run caller[3].sub(/\.rb:.*/, '.txt')
end
end

# Alias some more ruby-like methods on SeleniumDriver, to make it play
# better with rspec matchers.
module Selenium
class SeleniumDriver
alias_method :original_method_missing, :method_missing
def method_missing method_name, *args
if method_name.to_s =~ /^has_.*\?$/
real_method = method_name.to_s.sub /has_(.*)\?$/, 'is_\1'
if respond_to? real_method
return send(real_method, args)
end
elsif respond_to?(real_method = "get_" + method_name.to_s)
return send(real_method)
end
return original_method_missing(method_name, args)
end
end
end
[/ruby]

Most of this code doesn't really belong 'loose' in a helper file, but I didn't want to expend too much effort on getting everything working tidily with a pre-release version of RSpec.Hopefully there are enough comments to give a vague idea what's going on, but here are some more notes:

  • Lines 5–10: There still seem to be some bugs in trunk RSpec which cause existing specs to fail (or possibly bugs in our specs that don’t show up in 1.0.8). I’ve installed the trunk versions of the plugins under lib, and here I pull the real plugins out of the load path and insert the trunk versions instead.
  • Lines 17–21: Create a selenium driver, and open a browser session.
  • Lines 23–28: By default, RSpec adds an ActiveRecordSafetyListener to the story runner. This rolls back database changes between scenarios, which is great if your calling your code directly, but obviously means that if you write to the database, the server that Selenium’s talking to can’t see them. There’s probably a cleaner way of disabling it.
  • Lines 30–37: A convenience method to remove some code from the story wrappers. Basically just replaces ‘.rb’ with ‘.txt’ and runs that file with the specified steps. However, I can’t find a clean way of finding the file name of the code that called a method, so I’m pulling it out of the call stack. Ugh.
  • Lines 39–56: The method names in the ruby interface to Selenium RC are rather java-like (eg get_title and is_text_present). I’ve used a bit of method_missing hackery to wrap these methods with more ruby-like names (title and has_text_present? for those specific cases), which allows us to use rspec matchers like $selenium.title.should == foo and $selenium.should have_title bar.

lib/tasks/acceptance.rake

Finally, I’ve created some rake tasks to wrap everything up, and also to start and stop the selenium server.

[ruby]
desc “Run the acceptance tests, starting/stopping the selenium server.”
task :acceptance => ['selenium:start'] do
begin
Rake::Task['acceptance:run'].invoke
ensure
Rake::Task['selenium:stop'].invoke
end
end

namespace :acceptance do
desc “Run the acceptance tests, assuming the selenium server is running.”
task :run do
system ‘ruby stories/all.rb’
end

namespace :selenium do
desc “Start the selenium server”
task :start do
pid = fork do
exec ‘java -jar lib/selenium-server/selenium-server.jar’
exit! 127
end
File.open SELENIUM_SERVER_PID_FILE, ‘w’ do |f|
f.puts pid
end
# wait a few seconds to make sure it’s finished starting
sleep 3
end

desc “Stop the selenium server”
task :stop do
if File.exist? SELENIUM_SERVER_PID_FILE
pid = File.read(SELENIUM_SERVER_PID_FILE).to_i
Process.kill ‘TERM’, pid
FileUtils.rm SELENIUM_SERVER_PID_FILE
else
puts “#{SELENIUM_SERVER_PID_FILE} not found”
end
end
end
end

private

SELENIUM_SERVER_PID_FILE = ‘tmp/pids/selenium_server.pid’
[/ruby]

Still to do

There’s plenty of scope for improvement here. For a start, I haven’t figured out how to generate a report for the test run, apart from the plain text output from the console (which is only usable if you start the selenium server in a separate shell). Also, I can’t see an obvious way of returning a success/fail status at the end of the run, which makes it difficult to use in a continuous integration process.

The story runner looks like a very promising tool though, and I can’t wait for the stable release.

Technorati Tags: , , , ,

Written by Kerry

November 7th, 2007 at 11:57 am

Posted in Agile,Ruby,Software

7 Responses to 'Driving Selenium from the RSpec Story Runner (RBehave)'

Subscribe to comments with RSS or TrackBack to 'Driving Selenium from the RSpec Story Runner (RBehave)'.

  1. Thanks Kerry for the great post. I was looking for a good summary of the plain text stories and this was a great one. The Selenium integration seems like a very cool idea. I’ll have to look into Selenium.. Do you know if you can have Selenium run on multiple browsers and have them interact with the JS checking the DOM at each intermediate step?

    Ben Mabey

    8 Nov 07 at 4:25 am

  2. Hi Ben, glad you found it useful.

    You can definitely drive multiple browsers from Selenium, but I’m not sure about checking the DOM.

    Kerry

    8 Nov 07 at 8:15 am

  3. [...] week I stumbled upon acceptance testing in rSpec and a very pretty way of writing those [...]

  4. [...] Earlier I wrote about the new plain text story runner in RSpec and the integration of these stories with Selenium. [...]

  5. [...] forefront of my mind since that time and it was great to stumble upon Kerry Buckley’s blog on selenium-rc and rbehave.  In my opinion, the type of approach to testing that he discusses is an absolute necessity if we [...]

  6. [...] website, using plain text stories from David Chelimsky together with the Selenium runner from Kerry Buckley When I wanted to make the stories as clean as possible And I wanted to click a rails generated [...]

  7. Hack again?!

    Blackdanhil

    27 Jan 09 at 1:50 pm

Leave a Reply