Kerry Buckley

Over the years I've tried a number of approaches for managing gem dependencies in a Rails project. Here's a quick round-up of what I've tried, and the pros and cons of each.

Just use what's on the system

This is probably most people's default approach when first starting with Rails. Just sudo gem install whatever you need, require the appropriate gems (either in environment.rb or in the class that uses them), and you're away.

This mostly works OK for small projects where you're the only developer, but you still need to make sure the right gems are installed on the machine you're deploying the application to.

Worse, though, is what happens when you come back to the project after a while, various gems have been updated, and things mysteriously don't work any more. Not only do you have to mess around getting the code to work with the latest gem versions, but you probably don't even know exactly which versions it used to work with.

Freeze (unpack) gems

I think I first came across this technique in Err the Blog's Vendor Everything post. The idea is to install copies of all your gems into the project's vendor/gems directory, meaning that wherever the code is running, you can guarantee that it has the correct versions of all its dependencies.

This got much easier in Rails 2.1, which allowed you to specify all your gems using config.gem lines in environment.rb (you can also put gems only needed in specific environments in the appropriate file, eg you might only want to list things like rspec and cucumber in config/enviroments/test.rb). You can then run sudo rake gems:install to install any gems that aren't on your system, and rake gems:unpack to freeze them into vendor/rails, and be sure that wherever you check out or deploy the code, you'll be running the same versions of the gems. There's even a gems:build task to deal with gems that have native code (but more on that later).

Subsequent versions of Rails have improved on the original rake tasks – dependencies are now handled much better, for example – but there are still a few problems. The main one is the handling of gems that are required by rake tasks in your project, rather than just from your application code.

When you call a rake task in your Rails project, this is more-or-less what happens (I may have got some of the details slightly wrong):

1. The top-level Rakefile is loaded.
2. This in turn requires config/boot.rb, but not config/environment.rb.
3. It then requires some standard rake stuff, and finally tasks/rails (which is part of Rails – specifically railties). This finds and requires all the .rake files in your plugins and your project's lib/rake directory.

The problems start when you have a task depends on the rails environment task, and also requires a gem which is listed in environment.rb. Because the gem-loading magic only happens when the environment is loaded, the rake task will be blissfully unaware of your frozen gems, and will load them from the system instead.

If the system gem is newer than the frozen one, you get errors like this:

can't activate foo (= 1.2.3, runtime) for [], already activated foo-1.2.4 for []

If you work on two projects that use different versions of a gem like this, you end up having to uninstall and reinstall them as you switch from one to the other, which gets tedious fairly quickly.

Specify gems, but don't freeze

You can get round the wrong-version problem to some extent by specifying version numbers in environment.yml as '>=x.z.y' (or by not specifying them at all). If you're doing that, though, there's not really much benefit in unpacking the gems, and you may as well just use rake gems:install to make sure they're on the system. Of course the downside of this approach is that you can't be sure that everyone's running the exact same versions of the gems. Worse still, you can't be sure that what's on your production box matches your development and test environments.

GemInstaller

GemInstaller solves most of the problems with the built-in Rails gem management by running as a preinitializer, meaning it gets loaded before the other boot.rb gubbins.

GemInstaller uses the gems installed on the system rather than freezing them into the project, but because it gets to run first it ensures that the correct versions are used, even if there are newer versions installed. By default it checks your project's gem list and installs anything that's missing every time it runs (which is whenever you start a server, run the console, execute a rake task etc). You create a YAML file listing the gems you need (dependencies are handled automatically), and other options such as an HTTP proxy if necessary.

Of course on Unix-like systems, which is most of them (although I hear there are still people developing Rails projects on Windows), gems are generally installed as root. GemInstaller can get round this in two ways – either by setting the --sudo option and setting a rule in /etc/sudoers to allow the appropriate user(s) to run the gem commands as root without having to provide a password, or by using the built-in gem behaviour that falls back to installing in ~/.gem.

Personally I like to keep all my gems in one place, accessible to any user, so I went for the sudo approach. The only problem with this is that it uses sudo for all gem commands, rather than just install or update, which means it runs a sudo gem list every time your app starts up. Depending on the way you have Apache and Passenger set up this may mean granting sudo access to what should be a low-privileged user.

I ended up disabling the automatic updating of gems, and just warning when they're missing instead. In fact later versions of GemInstaller don't try to handle the update automatically anyway.

I created a separate script to do the update, which can be run manually, on a post-merge git hook, or as part of the Capistrano deployment task.

Because GemInstaller needs to go out to the network to fetch any new or updated gems, things get a bit more painful (as always) if you are unfortunate enough to be stuck behind a corporate HTTP proxy. Actually it's easy enough to configure if you're always behind a proxy, but it gets slightly trickier if your web access is sometimes proxied and sometimes direct. Nothing that can't be solved of course.

Unfortunately you can still end up with version conflicts if a gem is required by one you have specified, then you explicitly require an older version, but these can usually be resolved by shuffling the order of the gems in geminstaller.yml.

Bundler

Bundler is the newest kid on the gem management block, and looks to have solved pretty much all the problems faced by the other approaches. It's based on the gem management approach from Merb, and can be used in any Ruby project (not just Rails).

Bundler works by unpacking gems into the project (I recommend using a directory other than the default vendor/gems to avoid confusing Rails – this can be configured by setting bundle_path and bin_path in the Gemfile), but the intention is that you only commit the .gem files in the cache directory to source control. Gems are then installed locally within the project, including any platform-specific native code as well as the commands in bin.

Because Bundler resolves all dependencies up-front, you only need to specify the gems you're using explicitly, and let it handle the rest, which hopefully means an end to version conflicts at last.

Here's an example Gemfile:

Note the two additional sources (rubyforge.org is configured by default), the path overrides, and the last line, which removes the system gems from the paths, avoiding any potential confusion.

I've put this in config/preinitializer.rb to update from the cached gems on startup (this doesn't hit the network):

To avoid any startup delays after an upgrade, I also call system 'gem bundle --cached' from the after_update_code hook in the capfile.

Finally, to make sure only the .gem files are checked in, add these lines to .gitignore (you'll still need to explicitly git add the bundled_gems/cache directory):

vendor/bundled_gems
!vendor/bundled_gems/cache

[Update 3 November] Yehuda Katz just posted an article all about Bundler, including features coming in the imminent 0.7 release.

Technorati Tags: ruby, rails, gems, rubygems, geminstaller, bundler

I tend to agree that comments are, in most cases, evil (or at least mildly malevolent), but I did come across one of the exceptions to the rule today.

While doing a bit of drive-by refactoring while fixing a bug, I reflexively changed this line:

to this:

Then reading the comment above the line, expecting to delete it, it all came flooding back:

Now you could maybe argue that this unexpected behaviour is because httparty uses just a little too much of that old method missing proxy magic (which of course isn't really magic at all), but that's not the point of this post.

In the end I pulled it out into a private method to make it clearer what was going on, but decided to leave the comment in.

Naresh Jain recently posted a refactoring teaser. The original code was in Java, but I thought I'd have a go at refactoring it in Ruby instead. I deliberately didn't look at Naresh's solution beforehand, so mine goes in a rather different direction.

My code's here (with the specs in the same file for simplicity), and you can step through the history to see the state at each step of the refactoring. The links in the text below take you to the relevant version of the file, and the Δs next to them link to the diffs.

Firstly, I converted the code to Ruby, and made sure the tests (which I converted to RSpec) still passed. It's a fairly straight conversion, although I made a couple of changes while I was at it – mainly turning it into a module which I mixed into String. I changed the name of the method to phrases, to avoid conflicting with the built-in String#split. The regular expression to split on is much simpler too, because Ruby didn't understand the original, and I had no idea what it was supposed to do anyway.

Here's my initial Ruby version:

I didn't change the specs at all during the refactoring (because I didn't change the API or add any new public methods or classes), and made sure they all passed at each step.

The first thing Δ I changed was to simplify that big iterator in phrases that loops through the list of phrases, formatting the output string. Basically all this does is to put each phrase in quotes, then stitch them all together separated by a comma and a space. The first of those tasks is a simple map, and the second is a join. The whole method collapses down to this (ruby methods automatically return the result of their last statement):

Next Δ I remembered that by default String#split splits at whitespace anyway, so I did away with the regular expression. Then Δ I renamed the strings variable to words to make its purpose a little clearer, leaving the phrases method looking like this:

The section of single_double_triple_words that extracted the single words seemed redundant, as we already had that list – the original words. I removed it Δ, and initialised all_strings to the word list instead (not forgetting to rename single_double_triple_words to match its new behaviour):

That has_enough_words method seemed a bit odd – particularly the way it was only called once, rather than after extracting each set of phrases. I decided it was probably a premature and incomplete attempt at optimisation, and removed it Δ for now.

My next target was the duplication in the blocks that calculate double- and triple-word phrases. First Δ I extracted them into separate methods:

I decided that the num_words variable in the two new methods wasn't really necessary (it was only used once, and I think strings.size expresses intent perfectly clearly), so I inlined it Δ (and the same in triple_words):

Most of the code in double_words and triple_words was obviously very similar, so I created Δ a general extract_phrases method, and called it from both. The new method uses the start position and length version of Array#[] to extract the appropriate number of words, then Array#join to string them together separated by spaces:

At this point double_words and triple_words have become just dumb wrappers around extract_phrases, so I removed them Δ and just called extract_phrases directly:

Rather than hardcoding the calls for two and three words, I changed it Δ to use a loop:

I decided this was the point to put the optimisation back Δ and stop looking for phrases once we had enough:

I decided that number_of_words wasn't particularly clear, so changed it Δ to phrase_length, then Δ made the iterator in extract_phrases more ruby-like by using map:

I then noticed that I hadn't been consistent in changing strings to words, so fixed that Δ.

Lastly Δ, I decided that even though it was only one line, the code that formats the output deserved pulling out into a method.

Here's my final version of the code:

Just a couple of things that have caused a bit of head-scratching lately when writing RSpec specs using the built-in mocking framework.

Catching StandardError

Watch out if the code you're testing catches StandardError (of course you're not catching Exception, right?). Try this:

Nothing particularly exciting there. Let's run it and check that it passes:

$ spec foo.rb
.

Finished in 0.001862 seconds

1 example, 0 failures

However, what if we change the example to test the opposite behaviour?

$ spec foo.rb
.

Finished in 0.001865 seconds

1 example, 0 failures

Wait, surely they can't both pass? Let's take out the rescue and see what's going on:

$ spec foo.rb
F

1)
Spec::Mocks::MockExpectationError in 'Calling a method that catches StandardError does NOT call Bar.bar'
 expected :bar with (no args) 0 times, but received it once
./foo.rb:6:in `foo'
./foo.rb:18:

Finished in 0.002276 seconds

1 example, 1 failure

That's more like it.

Of course, what's really happening here is that Spec::Mocks::MockExpectationError is a subclass of StandardError, so is being caught and silently discarded by our method under test.

If you're doing TDD properly, this won't result in a useless test (at least not immediately), but it might cause you to spend a while trying to figure out how to get a failing test before you add the call to Foo.foo (assuming the method with the rescue already existed). Generally you can solve the problem by making the code a bit more selective about which exception class(es) it catches, but I wonder whether RSpec exceptions are special cases which ought to directly extend Exception.

Checking receive counts on previously-stubbed methods

It's quite common to stub a method on a collaborator in a before block, then check the details of the call to the method in a specific example. This doesn't work quite as you would expect if for some reason you want to check that the method is only called a specific number of times:

$ spec foo.rb
.

Finished in 0.001867 seconds

1 example, 0 failures

I think what's happening here is that the mock object would normally receive an unexpected call, causing the expected :bar with (any args) once, but received it twice error that you'd expect. Unfortunately the second call to the method is handled by the stub, so never triggers the error.

You can fix it, but it's messy:

$ spec foo.rb
F

1)
Spec::Mocks::MockExpectationError in 'Checking call counts for a stubbed method only calls a method once'
 expected :bar with (any args) once, but received it twice
./foo.rb:23:

Finished in 0.002542 seconds

1 example, 1 failure

Does anyone know a better way?

The full example code is in this gist.

Just came across an interesting error message from rspec. I had a spec that looked like this:

Obviously it failed, as I hadn't written the code yet, but there was more in the error message than I expected:

..........F

1)
RuntimeError in 'Blog should not mass-assign 'confirmed''
'should_not be  true' not only FAILED,
it is a bit confusing.
It might be more clearly expressed in the positive?
.../spec/models/blog_spec.rb:20:

Finished in 0.06192 seconds

11 examples, 1 failure

In fact, rewriting this as should be_false wouldn't work, as the expected value is nil. I took the hint though, and rewrote it as should be_nil.

World's-most-famous-twitterer Stephen Fry has a system for handling follow requests: you tweet using the #followmestephen hashtag, and he wades diligently through them, manually following people.

This seems an odd sort of thing to do – most people choose whom to follow based on whether they know them or like what they say, rather than on request – but I suppose when you have over a quarter of a million followers things work a little differently. It also creates lots of work , and looks like an ideal candidate for automation.

I thought I'd have a quick play with the Twitter API this morning (no doubt I'm not the only one), and cobbled together the script below, which you can also download as follow_me_stephen.rb (although if you're not Mr Fry I'm not sure why you would want to). Save the file, and run using ruby follow_me_stephen.rb.

I wanted to avoid having too many dependencies, so I didn't use the twitter gem, or the excellent httparty, but I was too lazy to figure out all the XPaths to handle the Atom version of the API. This means you need to have the JSON gem installed, which is as simple as sudo gem install json (omit the sudo on Windows).

The script's pretty dumb, in that it grabs the whole set of search results every time, and blindly requests to follow everyone, regardless of whether you're already following them.

For those outside the Rails community who have no idea what I'm on about, some people got a bit upset about Rails 2.2 defining Array#second up to Array#tenth, so for example you can call foo.fifth instead of foo[4] (you can already call foo.first instead of foo[0]). One of the last changes to be committed before 2.2 was released was to slim the list down to just second, third, fourth and fifth, but adding Array#forty_two (the ultimate answer) instead.

I've been a fan of the RailsEnvy guys (Gregg Pollack and Jason Seifer) ever since their "Hi, I'm Ruby on Rails" spoof of the "I'm a Mac" ads, and have been listening to their podcast ever since. I even got a mention on it once. Well that's not strictly true – I wasn't actually mentioned, but a patch I'd contributed to Capistrano was, which is close enough.

Recently, Gregg and Jason have branched out into screencasts, but I hadn't actually watched one because (understandably) they charge for them, and I was too tight to cough up the cash. £1200 for a new MacBook, no problem. A fiver for a screencast? What am I, made of money?

Anyhow, when I saw that they were looking for people to review their Ruby on Rails Envycast, covering the latest goodness in Rails 2.2, I jumped at the chance to get a free copy. A wonderful example of cognitive bias, given that I wouldn't have agreed to write a review just to be paid $16.

What do I get for the money?

The basic $9 gets you the screencast and a set of code samples to go with it, or for $16 they'll throw in Carlos Brando's Ruby on Rails 2.2 PDF too. Alternatively the PDF is available on its own, also for $9.

Video

The video is available in Quicktime or Ogg formats at a resolution of 569×480, as well as a version optimised for iPhones and iPods. Total running time is just under 45 minutes, and incredibly the first 39½ of those go by before Jason makes any claims about Rails's scalability.

I don't know whether it's unique, but the Envycast style of having the presenters chroma-keyed onto the Keynote presentation generally works very well. The visuals themselves are professional, although sometimes the 'sparkle' effect is a bit overused for my taste. The presentation style is much as you'd expect if you've listened to the podcasts, with plenty of cheesy humour to keep things interesting. I think having two people present in a conversational style is a big help.

The screencast is split into sections, each covering the new features for a different component (ActiveRecord, ActiveSupport, ActionPack, ActionController, Railties, internationalization and performance). The Quicktime version (not sure about the others) has bookmarks, making it easy to jump to a particular section. The whole thing is set against a variety of city skylines to liven the background up a little – by the way guys, that's Tower Bridge, not London Bridge.

Each new feature is introduced with an example, generally contrasting the 'old' way of doing something with the equivalent in 2.2. There's enough detail to get the idea of what's changed, without dwelling too long on each one. One tiny gripe with the code snippets on screen: the pedant in me hates seeing curly quotes in code, because I know if I typed puts ‘foo’ into irb instead of puts 'foo', it wouldn't work.

Code samples

The screencast comes with a set of code samples to illustrate all the features discussed in the screencast. These take the form of sample classes with Test::Unit test cases, along with rakefiles to run them. The sample directory contains a frozen installation of Rails 2.2, so all you need to do to run them is add the appropriate values to database.yml. I had trouble running them initially because they were inside a directory with a space in its name, but other than that it all worked nicely.

PDF

The PDF that comes with the $16 bundle is by Carlos Brando, well-known for his free Rails 2.1 book. It's available in the original Portugese, or translated to English by Carl Youngblood. The book weighs in at 118 pages, and as you would expect goes into more detail than the screencast. It claims to cover all the major changes in Rails 2.2 (I haven't checked!), and contains clear descriptions with examples.

Conclusion

So is it worth it? On balance, I think the answer is yes, although I wonder whether they'd sell more at $5 rather than $9 – after all, I can buy (to pick an example at random) the entire Naked Gun trilogy on DVD for roughly the same amount, and Gregg and Jason aren't that funny. The value is in collecting all the information in one place – you could trawl through the release notes and lighthouse tickets to get all the same information, but if you value your time at all, the screencast and PDF pay for themselves many times over.

Should you buy the PDF, the video or both? If you just want the hard facts, go for the PDF, but if you want to be entertained too (assuming you find the Rails Envy podcasts entertaining), get the video as well. The next episode, Scaling Ruby, is out now, and I might buy it just to see if Jason finally admits that Rails might actually be able to scale.

Technorati Tags: railsenvy screencast rails2.2

I consider myself fortunate that the previous two projects I worked on (the BT Web21C Portal and Mojo) were Rails-based (actually it wan't just luck in the former case, as I had a part in selecting the framework). I love the expressiveness and flexibility of the Ruby language, the power and relative simplicity of the Rails framework, and the all-round awesomeness of tools like RSpec and Capistrano, and I don't particularly relish the thought of going back to Java (although I'm told that Spring is much nicer than last time I used it).

At our recent release planning session, I was assigned to a new project, which involves (among other things) exposing a CLI-based configuration interface as a web service. In our initial discussions, the four of us more-or-less agreed on a few initial decisions:

• The exposure should be REST. Fortunately the people developing the upstream system shared this opinion.
• Rails is an ideal framework for RESTful web services.
• Ruby seemed like a good fit for parsing the command responses too.
• Asynchronous behaviour would be handled using queues (probably ActiveMQ).

Since we'd all been working on Rails projects when the new team was formed, we assumed that this wouldn't be a particularly contentious route to go down, but unfortunately our director/architect/boss didn't see things quite the same way. He had two main objections:

• Rails may make sense for GUI applications, but why on earth would you use it for a service? All our other [SOAP] services are written in Java.
• At some point the application will need to go into support, and we don't have support/operations people with Ruby or Rails experience

I think the first point's easier to address, as I'd argue it's based on a misunderstanding: Rails isn't really anything to do with GUIs, but is a framework for creating MVC web applications. Virtually all the heavy lifting Rails takes care of is in the controller and model areas, with the creation of the actual visible GUI being left to the developer to take care of with the usual mix of HTML, CSS and Javascript. The only thing Rails adds is the ability to insert dynamic content using ERB – similar to the role of JSP in Java EE.

A RESTful web service is, to all intents and purposes, the same as a normal web application, but (potentially) without the HTML. All the power that Rails brings to web application development is also harnessed when creating RESTful services.

The second point represents a much more fundamental strategy choice. If the company makes the decision that all development is going to use Java (the language as well as the platform), then we inevitably lose the flexibility to choose what may appear (in a local context) to be the right tool for the job. Personally I think that would be a shortsighted and ill-informed decision: if that were the strategy, we'd presumably all still be developing in C, or COBOL, or Assembler. Or we'd have gone bust. But then I'm not an architect (incidentally, according to Peter Gillard-Moss, that's reason number 10 why I don't deserve to be fired), so what do I know?

However, if Ruby is considered an acceptable technology choice for "normal" web applications, we'll still need people with appropriate skills to support those, so the problem doesn't go away. I suspect even for a Java specialist, supporting a well-written Rails application with good test coverage is probably easier than supporting some of the spaghetti-coded Java I've seen.

Anyway, our arguments obviously weren't totally unconvincing, because we were given a couple of weeks to show what we could produce before getting a final decision. That time runs out on Monday, so if I'm unnaturally grumpy after that it'll be because we've been told to chuck all our work so far away and start from scratch in Java. Or possibly FORTRAN.

Update, 16 June Well we made our case, and we get to stick with Rails. Celebration all round!

For ages I've been getting an intermittent problem with RSpec, where occasionally I'd see the following error on a model spec:

You have to declare the controller name in controller specs. For example:
describe "The ExampleController" do
controller_name "example" #invokes the ExampleController
end

The problem seemed to depend on which order the specs were run in, and for rake it could be avoided by removing --loadby mtime --reverse from spec.opts. It was a real pain with autotest though, and today (my original plan of "wait for RSpec 1.1 and hope it goes away" having failed) I finally got round to looking into it properly.

It seemed that the error was being triggered by the rather unpleasant code I wrote a while ago to simplify testing of model validation. Digging into the RSpec source to see what was happening, I found that that error message only gets returned when (as you'd expect) you don't declare the controller name in a controller spec (specifically in an instance of Spec::Rails::Example::ControllerExampleGroup). The code that decides what type of example group to create lives in Spec::DSL::BehaviourFactory, and according to its specs, there are two methods it uses to figure out what type of spec it's looking at:

I began to suspect that the problem was caused by the fact that my specify_attributes method wasn't declared in a file in spec/models, so I thought I'd try specifying the type explicitly. So instead of this:

I changed it to this:

Sure enough, it worked! Not sure whether anyone else is likely to see the same problem (unless they're foolish enough to use my validation spec code), but hopefully if you do, a Google search will bring up this post and it might point you in the right direction.