Kerry Buckley

What’s the simplest thing that could possibly go wrong?

Setting up a new Mac

leave a comment

I decided to do a clean install on my new [work] Mac, rather than just copying everything across from my old one. I thought I’d note down some of the things I installed, in case it comes in handy for anyone else (particularly new Mac users).

I’ve mostly ignored common cross-platform stuff like DropBox, 1Password etc, and setting up things like git and ruby. If you just want to quickly set up a Mac for ruby development, Thoughtbot’s laptop project (or Adam’s fork) might be the way to go.

If you’ve got any other can’t-live-without Mac apps that I might have missed, let me know!

From the app store:

Other apps:

Packages from homebrew (see above)

brew install direnv dos2unix elixir git mysql node phantomjs postgresql qt redis sqlite ssh-copy-id the_silver_searcher tig youtube-dl

Hidden settings

Show ~/Library in Finder:

chflags nohidden ~/Library

Enable copying from quick look preview windows:

defaults write com.apple.finder QLEnableTextSelection -bool TRUE; killall Finder

Written by Kerry

November 8th, 2014 at 10:57 am

Posted in Apple

Quick-and-dirty “edit text in vim” Mac service

one comment

Today I wanted to edit a big chunk of text (the source of a wiki page) in a browser textarea, and remembered that I used to have an app that let me open any text in vim, then paste the result back where it came from. Eventually I found the app I was thinking of – QuickCursor – but it turns out it’s no longer available.

Coincidentally I was looking into how to do something else tonight, and the answer was to create a service that ran an AppleScript, and it occurred to me that I could probably make a basic “edit in vim” service fairly easily. Turns out I was right…

Open the “Automator” app, and create a new service. Add a shell script action, and paste in the following script (this assumes you’ve installed the mvim script that comes with MacVim):

filename=`mktemp ~/.edit-in-vim-XXXX`
cat > $filename
/usr/local/bin/mvim -f $filename
cat $filename
rm $filename

Creating a service in Automator

Select “Output replaces selected text” in Automator, save the service, and you’re done. Just select some text (it doesn’t select-all for you like QuickCursor did), right-click and run your new service. You can also assign a keyboard shortcut in System Preferences:

Setting a keyboard shortcut

Written by Kerry

September 5th, 2013 at 8:11 pm

Posted in Software

Tagged with , , ,

FESuffolk Jasmine lightning talk

leave a comment

At last night’s FESuffolk meeting I gave a lightning talk on getting started with unit-testing Javascript/coffeescript using jasmine-headless-webkit. I made a screen recording, so if you missed it you can still experience my frantic babbling (I had Keynote set to auto-advance, using the Ignite 20 × 15 second slide format, and as usual I tried to cram too much into each slide).

If you don’t want to watch the recording, you can look at the slides and guess what I might have been saying, read the example code (the commits try to build up in small steps), or just play with the utterly pointless app that the example implements.

Written by Kerry

March 26th, 2013 at 8:00 pm

Interesting little rSpec gotcha

one comment

This one had Adam and me stumped for a while. Trying to check that a method is only called on objects that respond to it:

describe "Foo#call_all_the_things" do
  let(:foo_1) { stub :foo_1, bar: "hello" }
  let(:foo_2) { stub :foo_2 }
  subject { Foo.new foo_1, foo_2 }

  it "only calls bar on objects that respond to it" do
    foo_1.should_receive :bar
    foo_2.should_not_receive :bar
    subject.call_all_the_things(:bar)
  end
end

class Foo
  def initialize *things
    @things = things
  end

  def call_all_the_things method
    @things.each do |thing|
      thing.send method if thing.respond_to? method
    end
  end
end


  1) Foo#call_all_the_things only calls bar on objects that respond to it
     Failure/Error: thing.send method if thing.respond_to? method
       (Stub :foo_2).bar(no args)
           expected: 0 times
           received: 1 time

Hmm. Why is it calling bar on the thing that doesn’t respond to it? Perhaps rSpec doubles don’t handle respond_to? properly?

[1] pry(main)> require "rspec/mocks/standalone"
=> true
[2] pry(main)> foo = stub foo: 123
=> #
[3] pry(main)> foo.respond_to? :foo
=> true
[4] pry(main)> foo.respond_to? :bar
=> false

Nope.

FX: lightbulb above head

Of course! To do the should_not_receive check, it needs to stub the method, which means it responds to it!

Two possible solutions: either let the fact that the missing method isn’t called be tested implicitly, or specify that when objects that don’t respond to the method exist, no NoMethodError is raised.

Written by Kerry

March 20th, 2013 at 3:04 pm

Posted in rspec,Ruby,Software

Adastral Park code retreat

leave a comment

Code retreat poster

Ever since attending code retreat at Bletchley Park (the UK leg of Corey Haines’s code retreat tour) last year, I’d been vaguely intending to try and run an internal one at work. I finally got round to it this month, when a few people expressed an interest in a code retreat as a one-off alternative to the usual developer conferences we run a couple of times a year. I thought I’d share my experiences facilitating a code retreat for the first time, in case it’s helpful to anyone thinking of doing the same.

Beforehand

The “official” code retreat site has some good information on things to consider when hosting and facilitating. I suspect organising an internal company event is easier than a public one, because hopefully you’ll have a venue available and you won’t need to worry about sponsorship. The only one of Corey’s rules I ignored was the one about giving people decent food at lunchtime, but we did at least manage to beg enough budget (thanks Mel!) to be able to provide cheap Tesco sandwiches at lunchtime and a few random snacks and coffee during the day. I don’t feel too bad because rather than asking people to “come out at eight in the morning to spend the day coding”, we were giving them an excuse to spend a work day away from their projects, getting paid to practice.

We also planned to start at nine o’clock rather than the traditional eight. I think for public events the early start, usually on a Saturday, is a good way of selecting only people who really want to take part, but running it at work I wanted to swing the balance a bit towards encouraging people to come (I work on a large site with probably a couple of hundred developers scattered amongst the 3000-odd BT people, and part of the motivation for running this kind of event is to try and put developers from different teams in touch with each other).

With a bit of publicity via mailing lists and posters on site, 27 people had signed up. That seemed like about the right number, although I had no idea how many of those would turn up, or whether anyone would turn up without registering. I turned up at the office in plenty of time, and was lucky enough to get some last-minute tips from the master:

tips

Getting started

As it turned out we had a few glitches to deal with before we could get going – firstly the venue was locked and we only managed to get in because Rupert knew how to find the secret back entrance, then Adam and Anna got kicked out of Tesco for a fire alarm while buying the food – so we ended up starting at closer to 9.30.

Coffee machine hack

Potentially the biggest disaster though was that the hopper was missing from the coffee machine (most likely locked away to stop people like us using it with our own beans instead of paying for catering). Fortunately it takes more than a little detail like that to defeat a room full of engineers, and we soon had a workaround in place using a plastic cup with the bottom cut out.

Once the caffeine supply was assured and we were finally ready to start, I gave a quick presentation covering the purpose of the day, the four rules of simple design and Conway’s game of life:

First session

With everyone hopefully having a reasonable idea of what was going on, people paired up and started the first session. There were about 25–30 people (I forgot to count), split something like 70:20:10 between Java, Ruby and Groovy. In the first session, as expected, everyone was mostly just getting a feel for the problem, and realising that it’s not quite as simple as it appears from the list of rules.

Getting started

First session

After 45 minutes, I put on my best exam invigilator voice and got everyone to stop typing, delete their code and gather round. Pairs were split roughly evenly between those that started with a grid and those that started with a cell, and we had a short discussion about how a grid approach using a two-dimensional array fitted in with the “infinite grid” constraint. Some of those who had started from a cell were thinking about approaches for allowing cells to know about their neighbours, but no-one had got that far yet.

Mixing it up

For the second session, I suggested that those that had started with a grid tried starting from a cell instead, and vice versa. There were also quite a few people who wanted a chance to try Ruby, so they tried to grab the rubyists before anyone else did. Little did they know that they would find themselves exposed to Vim (and in some cases Mac OS X) at the same time, but that’s just a bonus.

In general people who tried both seemed to prefer the cell approach, partly because it made it easier to test-drive the rules. With a grid, people found they were having to implement some kind of to_string or render method to allow the transformed grid to be compared with the expected result. The biggest puzzle with the cell approach was how to handle the relationship between multiple cells.

I introduced a list of possible constraints people might want to consider to make things more interesting:

  • No if statements (or other conditionals).
  • No loops.
  • No methods longer than five lines (or four, or three…)
  • No more than three methods per class.
  • No ‘naked’ language primitives.
  • No built-in data structures.

I also attempted to explain TDD as if you meant it, in case anyone fancied giving it a go. Unfortunately my description was rather garbled, not helped by the fact that I’m not sure I fully understand it myself (I’m still annoyed that I picked the wrong track at SC2009 and missed Keith’s original presentation of the exercise, and it turns out Adam made the same mistake at SPA2010).

Pairing

Flying fingers

Session three

By the third session several pairs were starting to experiment with storing cell objects either in a set (with each cell knowing its position) or a hash, keyed by position. Some people used a comma-separated “x,y” string as the key, but with a bit of prodding most ended up replacing it with a point class of some kind.

One pair decided to start with a single cell, then initialise other cells with their neighbours. This initially seemed promising, but soon led to complications with circular references, as well as the need to keep dead cells around to maintain a connection between cells with space between them.

Another pair started developing the rules with a cell class, and initially had an alive? method which performed the life calculation. After some discussion they split the class into two – one for a living cell and another for a dead one. This removed a level of nesting from the calculations (which ended up in a different method), but interestingly left alive? in both classes, hardcoded to true and false respectively. We decided that this would have looked less odd in Java, where the method would have been declared abstractly in a superclass or interface.

One pair had decided to try implementing a solution with no if statements, but resorted to cheating by using a case instead!

In general, a lot of people – particularly those new to TDD – were starting to get frustrated, feeling like they were just getting to the point where they were “about to start the interesting bit” as the 45 minutes were up. I told them that we’d just have two sessions after lunch rather than three, with the second session lengthened to give people a chance of implementing a working game.

The countdown begins

Fourth – and as it turned out final – session

After lunch we started the fourth session, but it soon became clear that several pairs had already decided to abandon TDD and clean code in favour of trying to solve the problem. I completely understand how they felt – I wasted most of the sessions at the code retreat I attended simply pushing for the finish line instead of letting it go and concentrating on perfecting the internal quality of the code – but it seemed like we’d lost sight of the key goal of the day a little. I suspect I’m mostly to blame for that, and I’m sure a more experienced facilitator could have done a better job of gently guiding people in the right direction. Anyway, it seemed sensible to bow to the inevitable, let everyone carry on for another 45 minutes and wrap up a bit earlier than planned.

Basking in the glow

With the extra time, a few pairs did manage to get a working implementation, but the prize for the most impressive (cancelled out by a booby prize for abandoning TDD and neglecting the quality of their code!) goes to Jon and Jia Yan, who produced an SWT app that allowed graphical editing of the grid, variable speed and saving and loading files. They used the “set of live cells” approach, with an effectively infinite grid. Resizing the window on screen allowed you to show or hide cells, but the ones off the edge of the visible area continued to multiply.

Matthew and Rupert decided to look at someone else’s solution in Clojure, to understand how a functional approach using list comprehensions can lead to a very concise solution. They also played around with a Ruby version using the same techniques. Looking at the Clojure code led to a slightly rearranged version of the rules for life at a particular location after a clock tick:

  1. Cells which had three neighbours are alive
  2. Cells which had two neighbours are unchanged
  3. All other cells are dead

Paul, stuck on his own after his pair got dragged off onto a conference call (the perils of being a senior manager!), had a play with TDD as if you meant it. I think by the end of the session he was starting to get somewhere. Further than me, at any rate.

Wrapping it all up

After letting Jon show off a glider gun running in their SWT universe, we gathered round for the closing circle, asking everyone what they’d learned, what had surprised them, and what they’d do differently back at work on Monday as a result. Probably the most common comments revolved around the fact that neither of the two most obvious solutions turned out to be the best, suggesting that it’s worth spiking a few alternative approaches before heading blindly off down what may be a blind alley. Quite a few people were new to pairing, TDD or both, and generally seemed to agree that both provided benefits, although a couple of people felt that TDD was making things take much longer. Reassuringly, of the pairs that had abandoned TDD to concentrate on trying to get a working solution, most had found themselves descending into confusion without the tests to guide them.

Final thoughts

Overall, the event seemed to be a success. For quite a few people this was their first experience of TDD, and the opportunity to get first-hand experience by pairing with someone more experienced seemed to be useful. I was particularly pleased to see someone who’s recently come back into software development after a long time away doing a great job of holding his pair back from rushing into implementation without a failing test.

On a less positive note, I didn’t feel we really concentrated enough on “perfect” code and the four rules of simple design. I know from experience that it’s hard to let go of the natural desire to try and finish solving the problem, and I felt that a lot of people were getting frustrated enough without me peering over their shoulders picking holes in their code (however nicely I tried to do it). Perhaps this is a case of horses for courses, with the concentration on design perfection more suitable for a different mix of participants, or perhaps it just reflects my lack of experience facilitating this kind of event.

Also, I don’t think we really had enough sessions for the code retreat magic to start kicking in properly. We’d planned to have the recommended five or six, but a combination of the late start and a general loss of direction after lunch reduced that to four (with the fourth being double-length).

The important thing, of course, is that people learned something by participating, and I hope that was the case. Maybe we’ll do another one one day, and use the experience of this one to make it even better.

Written by Kerry

November 1st, 2011 at 11:42 am

Build and push to upstream git repository in the background

leave a comment

[Updated 11 Feb 2011 – I’m not sure the original version actually worked properly.]

Quite often I’ll make small changes to a project’s code which I’m 99% sure won’t break the build. Of course thanks to Murphy’s Law that probability falls to about 10% if I decide to risk it push the changes upstream without running the full build locally.

Now even if the build only takes a few minutes to run, that’s a few minutes which I can’t spend doing anything else with the code, because if I save a file I can’t be sure whether the original or changed version will be used in the tests. What I’d like to be able to do is to run the build in the background, with changes automatically pushed upstream if everything’s green, or an alert shown if either the build fails or the push is rejected (because someone else has pushed a change while the build was running).

Fortunately with a distributed version control system like git, this is fairly easy.

Firstly, clone the local working repository and set up a remote pointing to the clone:

[code]
git clone . ../myproject-staging
git remote add staging ../myproject-staging
[/code]

Now set up the cloned copy:

[code]
cd ../myproject-staging

# Allow pushing into the checked-out branch
git config receive.denyCurrentBranch ignore

# Create a remote for the upstream server you want to push to (change the URI!)
git remote add upstream [email protected]/your/repository/path.git
[/code]

Do anything that needs doing to allow the cloned project to build (if you have a test database, create a separate instance to avoid tests interacting with your main workspace), and verify that the build runs.

Now add the following to .git/hooks/post-receive:

[code]
#!/bin/bash

cd ..
export GIT_DIR=.git
git reset –hard
nohup .git/hooks/post-reset 1> build_output 2>&1 &
[/code]

This will reset your checked-out working copy to the pushed master, then run .git/hooks/post-reset in the background. Create that file with the following content:

[code]
#!/bin/bash

function error {
growlnotify -s -t `basename $PWD` -m “$1″
exit 1
}

# Replace this with your build command
bundle install –local && rake || error “Rake failed”

git push upstream master || error “Git push failed”
growlnotify -s -t `basename $PWD` -m “Built and pushed successfully”
[/code]

Make sure both these hooks are executable:

[code]
chmod u+x .git/hooks/post*-receive
[/code]

Now, back in your normal working repository you should just be able to type git push staging (add --force if you’ve rebased your normal master and get an error about the push not being a fast-forward), and the project will build in the staging repo. If everything works, it’ll push upstream and you’ll get a Growl notification (assuming you have Growl and growlnotify, or the equivalents for your OS, installed). If it fails, you’ll also get a Growl notification. I’ve made the alerts sticky, so I don’t miss them – if you don’t want that, just remove the -s flag. The build errors will be in the build_output file.

Written by Kerry

December 20th, 2010 at 5:14 pm

Posted in git,Software

Tagged with , ,

The Wandering Book

one comment

After procrastinating and holding on to the wandering book for much too long, I finally made my entry. You can read it on the site, and here’s the text if you can’t decipher my handwriting:

I work in a large “enterprisey” company where craftsmanship is, unsurprisingly,
not the dominant model of software development. However, a craftsmanship ethos
still exists in some of the remaining pockets of in-house development, and if
anything appears to be strengthening.

For the first few years of my software career, I treated it very much as a day
job. I turned up; I wrote some code (or, for a horrendous proportion of the
time, some documents); I went home. I learned just enough to enable me to
complete whatever I was working on, but no more. I used the tools I was given,
or those I was familiar with, regardless of their suitability for the task at
hand.

Then I started working with someone who was trying cool new stuff. Dependency
injection. Automated testing. Continuous integration. I realised how
out-of-touch I was in my ideas of how software should be developed, and started
trying to put that right. I read about XP and Agile, and started following
blogs and reading articles. I went to XPDay, and turned up at the odd XTC pub
night when I happened be in London on a Tuesday.

More importantly, I started caring about my code. Instead of thousand-line
procedures, I started separating concerns and following OO principles. I wrote
unit tests, and after a while I got the hang of TDD. Month-long manual testing
and bug fixing phases gave way to an automated nightly build and a quick
confidence check, and eventually to something approaching continuous
deployment.

Of course, the more widely I look at what others are doing, the more aware I am
that even when I think I might be doing something reasonably well, there are
always people out there doing it much, much better. I may never approach their
level, but I can learn from their work and thus improve mine. In turn I hope I
can pass something on to others.

So, to finally get to the point, I agree with earlier contributors that
craftsmanship is about caring, learning, practicing and sharing. Underpinning
all this though is community. Not ‘the’ software craftsmanship community, but
the community of other developers you work with; the communities around the
languages and tools you use; the agile, lean and XP communities; the community
of random people you met once at a conference – or maybe have never met at all
– and started following on Twitter.

Without these communities, we become isolated, and may start believing that the
way we currently work is ‘best practice’. We need our fellow professionals to
compare ourselves against and to learn from, otherwise we can lapse into
complacency and let our skills (and our profession) stagnate.

Technorati Tags: , ,

Written by Kerry

July 2nd, 2010 at 8:56 am

Posted in Software

Memoised remote attribute readers for ActiveRecord

leave a comment

I was recently working with an ActiveRecord class that exposed some attributes retrieved from a remote API, rather than from the database. The rules for handling the remote attributes were as follows:

  • If the record is unsaved, return the local value of the attribute, even if it’s nil.
  • If the record is saved and we don’t have a local value, call the remote API and remember and return the value.
  • If the record is saved and we already have a local value, return that.

Here’s the original code (names changed to protect the innocent):

[ruby]
class MyModel < ActiveRecord::Base
attr_writer :foo, :bar

def foo
(new_record? || @foo) ? @foo : remote_object.foo
end

def bar
(new_record? || @bar) ? @bar : remote_object.bar
end

def remote_object
@remote_object ||= RemoteService.remote_object
end
end
[/ruby]

The remote_object method makes a call to the remote service, and memoises the returned object (which contains all the attributes we are interested in).

I didn’t really like the duplication in all these accessor methods – we had more than the two I’ve shown here – so decided to factor it out into a common remote_attr_reader class method. Originally I had the method take a block which returned the remote value, but that made the tests more complicated, so I ended up using convention over configuration and having the accessor for foo call a remote_foo method.

Here’s the new code in the model:

[ruby]
class MyModel < ActiveRecord::Base
remote_attr_reader :foo, :bar

def remote_foo
remote_object.foo
end

def remote_bar
remote_object.bar
end

def remote_object
@remote_object ||= RemoteService.remote_object
end
end
[/ruby]

Here's the RemoteAttrReader module that makes it possible:

[ruby]
module RemoteAttrReader
def remote_attr_reader *names
names.each do |name|
attr_writer name
define_method name do
if new_record? || instance_variable_get(“@#{name}”)
instance_eval “@#{name}”
else
instance_eval “remote_#{name}”
end
end
end
end
end
[/ruby]

To make the module available to all models, I added an initialiser containing this line:

[ruby]
ActiveRecord::Base.send :extend, RemoteAttrReader
[/ruby]

Here’s the spec for the module:

[ruby]
require File.dirname(__FILE__) + ‘/../spec_helper’

class RemoteAttrReaderTestClass
extend RemoteAttrReader
remote_attr_reader :foo

def remote_foo
“remote value”
end
end

describe RemoteAttrReader do
let(:model) { RemoteAttrReaderTestClass.new }

describe “for an unsaved object” do
before do
model.stub(:new_record?).and_return true
end

describe “When the attribute is not set” do
it “returns nil” do
model.foo.should be_nil
end
end

describe “When the attribute is set” do
before do
model.foo = “foo”
end

it “returns the attribute” do
model.foo.should == “foo”
end
end
end

describe “for a saved object” do
before do
model.stub(:new_record?).and_return false
end

describe “When the attribute is set” do
before do
model.foo = “foo”
end

it “returns the attribute” do
model.foo.should == “foo”
end
end

describe “When the attribute is not set” do
it “returns the result of calling remote_” do
model.foo.should == “remote value”
end
end
end
end
[/ruby]

To simplify testing of the model, I created a matcher, which I put into a file in spec/support:

[ruby]
class ExposeRemoteAttribute
def initialize attribute
@attribute = attribute
end

def matches? model
@model = model
return false unless model.send(@attribute).nil?
model.send “#{@attribute}=”, “foo”
return false unless model.send(@attribute) == “foo”
model.stub(:new_record?).and_return false
return false unless model.send(@attribute) == “foo”
model.send “#{@attribute}=”, nil
model.stub(“remote_#{@attribute}”).and_return “bar”
model.send(@attribute) == “bar”
end

def failure_message_for_should
“expected #{@model.class} to expose remote attribute #{@attribute}”
end

def failure_message_for_should_not
“expected #{@model.class} not to expose remote attribute #{@attribute}”
end

def description
“expose remote attribute #{@attribute}”
end
end

def expose_remote_attribute expected
ExposeRemoteAttribute.new expected
end
[/ruby]

Testing the model now becomes a simple case of testing the remote_ methods in isolation, and using the matcher to test the behaviour of the remote_attr_reader call(s).

[ruby]
require File.dirname(__FILE__) + ‘/../spec_helper’

describe MyModel do
it { should expose_remote_attribute(:name) }
it { should expose_remote_attribute(:origin_server) }
it { should expose_remote_attribute(:delivery_domain) }

describe “reading remote foo” do
# test as a normal method
end
end
[/ruby]

Technorati Tags: , , , , , , ,

Written by Kerry

April 27th, 2010 at 11:29 am

Posted in Rails,Ruby

Managing gems in a Rails project

3 comments

Over the years I’ve tried a number of approaches for managing gem dependencies in a Rails project. Here’s a quick round-up of what I’ve tried, and the pros and cons of each.

Just use what’s on the system

This is probably most people’s default approach when first starting with Rails. Just sudo gem install whatever you need, require the appropriate gems (either in environment.rb or in the class that uses them), and you’re away.

This mostly works OK for small projects where you’re the only developer, but you still need to make sure the right gems are installed on the machine you’re deploying the application to.

Worse, though, is what happens when you come back to the project after a while, various gems have been updated, and things mysteriously don’t work any more. Not only do you have to mess around getting the code to work with the latest gem versions, but you probably don’t even know exactly which versions it used to work with.

Freeze (unpack) gems

I think I first came across this technique in Err the Blog’s Vendor Everything post. The idea is to install copies of all your gems into the project’s vendor/gems directory, meaning that wherever the code is running, you can guarantee that it has the correct versions of all its dependencies.

This got much easier in Rails 2.1, which allowed you to specify all your gems using config.gem lines in environment.rb (you can also put gems only needed in specific environments in the appropriate file, eg you might only want to list things like rspec and cucumber in config/enviroments/test.rb). You can then run sudo rake gems:install to install any gems that aren’t on your system, and rake gems:unpack to freeze them into vendor/rails, and be sure that wherever you check out or deploy the code, you’ll be running the same versions of the gems. There’s even a gems:build task to deal with gems that have native code (but more on that later).

Subsequent versions of Rails have improved on the original rake tasks – dependencies are now handled much better, for example – but there are still a few problems. The main one is the handling of gems that are required by rake tasks in your project, rather than just from your application code.

When you call a rake task in your Rails project, this is more-or-less what happens (I may have got some of the details slightly wrong):

  1. The top-level Rakefile is loaded.
  2. This in turn requires config/boot.rb, but not config/environment.rb.
  3. It then requires some standard rake stuff, and finally tasks/rails (which is part of Rails – specifically railties). This finds and requires all the .rake files in your plugins and your project’s lib/rake directory.

The problems start when you have a task depends on the rails environment task, and also requires a gem which is listed in environment.rb. Because the gem-loading magic only happens when the environment is loaded, the rake task will be blissfully unaware of your frozen gems, and will load them from the system instead.

If the system gem is newer than the frozen one, you get errors like this:

can't activate foo (= 1.2.3, runtime) for [], already activated foo-1.2.4 for []

If you work on two projects that use different versions of a gem like this, you end up having to uninstall and reinstall them as you switch from one to the other, which gets tedious fairly quickly.

Specify gems, but don’t freeze

You can get round the wrong-version problem to some extent by specifying version numbers in environment.yml as ‘>=x.z.y’ (or by not specifying them at all). If you’re doing that, though, there’s not really much benefit in unpacking the gems, and you may as well just use rake gems:install to make sure they’re on the system. Of course the downside of this approach is that you can’t be sure that everyone’s running the exact same versions of the gems. Worse still, you can’t be sure that what’s on your production box matches your development and test environments.

GemInstaller

GemInstaller solves most of the problems with the built-in Rails gem management by running as a preinitializer, meaning it gets loaded before the other boot.rb gubbins.

GemInstaller uses the gems installed on the system rather than freezing them into the project, but because it gets to run first it ensures that the correct versions are used, even if there are newer versions installed. By default it checks your project’s gem list and installs anything that’s missing every time it runs (which is whenever you start a server, run the console, execute a rake task etc). You create a YAML file listing the gems you need (dependencies are handled automatically), and other options such as an HTTP proxy if necessary.

Of course on Unix-like systems, which is most of them (although I hear there are still people developing Rails projects on Windows), gems are generally installed as root. GemInstaller can get round this in two ways – either by setting the --sudo option and setting a rule in /etc/sudoers to allow the appropriate user(s) to run the gem commands as root without having to provide a password, or by using the built-in gem behaviour that falls back to installing in ~/.gem.

Personally I like to keep all my gems in one place, accessible to any user, so I went for the sudo approach. The only problem with this is that it uses sudo for all gem commands, rather than just install or update, which means it runs a sudo gem list every time your app starts up. Depending on the way you have Apache and Passenger set up this may mean granting sudo access to what should be a low-privileged user.

I ended up disabling the automatic updating of gems, and just warning when they’re missing instead. In fact later versions of GemInstaller don’t try to handle the update automatically anyway.

I created a separate script to do the update, which can be run manually, on a post-merge git hook, or as part of the Capistrano deployment task.

Because GemInstaller needs to go out to the network to fetch any new or updated gems, things get a bit more painful (as always) if you are unfortunate enough to be stuck behind a corporate HTTP proxy. Actually it’s easy enough to configure if you’re always behind a proxy, but it gets slightly trickier if your web access is sometimes proxied and sometimes direct. Nothing that can’t be solved of course.

Unfortunately you can still end up with version conflicts if a gem is required by one you have specified, then you explicitly require an older version, but these can usually be resolved by shuffling the order of the gems in geminstaller.yml.

Bundler

Bundler is the newest kid on the gem management block, and looks to have solved pretty much all the problems faced by the other approaches. It’s based on the gem management approach from Merb, and can be used in any Ruby project (not just Rails).

Bundler works by unpacking gems into the project (I recommend using a directory other than the default vendor/gems to avoid confusing Rails – this can be configured by setting bundle_path and bin_path in the Gemfile), but the intention is that you only commit the .gem files in the cache directory to source control. Gems are then installed locally within the project, including any platform-specific native code as well as the commands in bin.

Because Bundler resolves all dependencies up-front, you only need to specify the gems you’re using explicitly, and let it handle the rest, which hopefully means an end to version conflicts at last.

Here’s an example Gemfile:

[ruby]
source ‘http://gemcutter.org’
source ‘http://gems.github.com’
bundle_path ‘vendor/bundled_gems’
bin_path ‘vendor/bundled_gems/bin’

gem ‘rails’, ‘2.3.4’
gem ‘bundler’, ‘0.6.0’

gem ‘capistrano’, ‘2.5.8’
gem ‘capistrano-ext’, ‘1.2.1’
gem ‘cucumber’, ‘0.4.3’, :except => :production
# [more gems here]

disable_system_gems
[/ruby]

Note the two additional sources (rubyforge.org is configured by default), the path overrides, and the last line, which removes the system gems from the paths, avoiding any potential confusion.

I’ve put this in config/preinitializer.rb to update from the cached gems on startup (this doesn’t hit the network):

[ruby]
$stderr.puts ‘Updating bundled gems…’
system ‘gem bundle –cached’
require “#{RAILS_ROOT}/vendor/bundled_gems/environment”
[/ruby]

To avoid any startup delays after an upgrade, I also call system 'gem bundle --cached' from the after_update_code hook in the capfile.

Finally, to make sure only the .gem files are checked in, add these lines to .gitignore (you’ll still need to explicitly git add the bundled_gems/cache directory):

vendor/bundled_gems
!vendor/bundled_gems/cache

[Update 3 November] Yehuda Katz just posted an article all about Bundler, including features coming in the imminent 0.7 release.

Technorati Tags: , , , , ,

Written by Kerry

November 2nd, 2009 at 5:24 pm

Posted in Rails,Ruby

Maintaining a Read-Only svn Mirror of a git Repository

7 comments

Our team been using git at work for the past couple of years, but there’s now a corporate push to keep everything in a centrally-managed subversion repository. We lost the battle to get corporate approval for git (apparently we’re happy to employ people to write our code that we wouldn’t trust to be able to use a DVCS), but at least we have agreement that we can continue using git as long as we mirror the code into subversion.

This isn’t entirely trivial to do using git-svn (which is really intended for use with subversion acting as the master), but I found a few sets of instructions on the web. The simplest one was this Nano Rails blog post, which is what the steps below are based on.

The end result should be a subversion trunk which is a mirror of the git repository’s master branch. This is only intended to be a one-way mirror – I wouldn’t recommend also trying to commit into subversion and merge those commits back upstream into git.

You need to have git-svn installed (this comes with the default installation, or you can use the +svn option when installing via MacPorts).

Create subversion repository

Create the subversion repository in the usual way, using svnadmin.

Once you’ve got an empty repository to point to (we’ll imagine it’s at http://svn.example.com/foo), you also need to commit an initial version (I also created a trunk directory in this step, in case we later decide to mirror branches too):

svn co http://svn.example.com/foo
cd myproj
svn mkdir trunk
svn commit -m'Created trunk directory'

Once this is done, you can throw away the directory you checked out of subversion.

Set up the subversion remote

This step, and subsequent ones, need to be performed on whichever git repository you want to mirror from.

In our case, we have a central repository running on a local installation of Gitorious. This is a bare repository, which makes things a little tricker, as git-svn requires a working copy. To get round this, we create a clone, which we’ll use as an intermediate step in the mirroring process. If you’re not mirroring a bare repository, you can omit this step.

The repositories we want to mirror are in ~git/repositories, and we’ve created a directory ~git/repositories/svn-mirror where we’ll put the clones. For this example, we’ll use a repository called foo/mainline.git.

Create the clone:

git clone ~git/repositories/foo/mainline.git ~git/repositories/svn-mirror/foo
cd ~git/repositories/svn-mirror/foo

Now add the following to .git/config (with the correct svn URI, of course):

[svn-remote "svn"]
	url = http://svn.example.com/foo/trunk
	fetch = :refs/remotes/git-svn

Now do an initial fetch of the empty subversion remote, and check it out as a new git branch (called svn):

git svn fetch svn
git checkout -b svn git-svn

You can now merge in all your commits from master, and push them to subversion. You’ll probably want to go and make a coffee or something while the dcommit runs – if you haven’t used subversion for a while you’ve probably forgotten just how much slower it is than git.

git merge master
git svn dcommit

To allow pushing to svn from master, rebase master to the svn branch (which can then be deleted):

git checkout master
git rebase svn
git branch -d svn

At this point you should be able to manually update subversion at any time by running git svn dcommit from the master branch.

Automating subversion updates

In theory it should be possible to set up post-receive hooks to push from the gitorious repository to the clone and from there to subversion, but I decided to separate the two by just periodically polling for changes, as we don’t really care about subversion being right up-to-the-minute. To poll hourly, add something like this to the git user’s crontab:

0 * * * * (cd /usr/local/git/repositories/svn-mirror/foo;/usr/local/bin/git pull origin master;/usr/local/bin/git svn dcommit) >>/usr/local/git/gitorious/log/svn-mirror.log 2>&1

Technorati Tags: , , ,

Written by Kerry

October 6th, 2009 at 10:17 am

Posted in git,Software