Categories
Weeknotes

Weeknotes 2022-02

OK, so it’s week two and I’m posting this four days late (just edited that from the original “a day late”). At this rate, expect the final post of this year some time in 2028.

A couple of weeks ago I took my Strava heatmap and drew round it to show the area enclosed by places I’d run to starting and finishing at home:

Someone on Facebook saw it and mentioned CityStrides, which is what in the early 2000s we would have called a mashup of activity tracker data with Open Streetmap, showing you how many of the streets in your town you’ve run along. I’m always a sucker for this kind of nonsense, so I signed up and synced it with Garmin (Strava didn’t work for some reason – I suspect API rate limiting). It turned out I was in third place for Ipswich, with 461 of 1,412 streets completed (and I know both the people above me in the table). Of course when I mentioned this three other people signed up, pushing me down to 6th, but I did add some convoluted meanderings to a couple of runs this week to pick up another 24 streets.

With the end of the grace period for free corporate use of Docker Desktop fast approaching, I finally got round to switching to MiniKube. I tried the default option of Hyperkit, but it wouldn’t start (maybe something to do with security settings on my work Mac), but it was fine with VirtualBox, and actually seems much quicker than Docker Desktop. I don’t use Docker for anything fancy – just to create dummy instances of other systems that my application can ssh to for integration testing, and to allow me to compile the OTP application for Linux on my Mac – and the only change I had to make was to have the app connect to the containers using the VM’s IP address, whereas the previous setup had automatically mapped the ports to the host address. I feel slightly guilty about being part of the problem of big companies using open source without paying anything back, but apparently someone else is looking at getting a company licence for Docker Desktop, so maybe they’ll get some of our money eventually.

Friday saw the third “agile fest” at work, with a variety of talks (I wasn’t involved this time). All good stuff, although I did get into a minor disagreement in the chat when someone suggested we needed a single mandated structure for Jira, rather than teams doing what works best for them (it would be bad enough if someone told me I had to use Jira at all, let alone also telling me how to use it!)

Categories
Weeknotes

Weeknotes 2022-01

First proper week of the year, and I nearly forgot to do this, so that bodes well for the experiment continuing.

Back to work properly this week (well, apart from Monday), and other people were mostly back too, so meetings and stuff have started happening again. One of those annoying Fridays where something turns out to be more complicated than I thought, so ended up leaving it half-done to pick up on Monday. It was one of those rabbit holes where I started adding a new feature, realised that the related tests didn’t quite cover all the permutations of existing behaviour, then discovered that the untested bits didn’t quite behave as I expected, and had to be fixed too. I seem to remember there being another layer of yak shaving too, but I can’t remember what it was (which doesn’t bode well for the morning).

I got my unicycle out for the first time in ages. I bought it at the beginning of 2020, but stopped trying to learn to ride it when I decided that turning up at A&E having injured myself during lockdown might not be looked on too favourably. This time I’m practicing outside, for the sake of the floor and walls in the hall, where I started before. Still at the “pedal a couple of times while using a wall for stability, then fall off” stage. Hopefully when (if) I finally get the knack, it’ll be a useless skill that never goes away, like riding half a bicycle.

I finally watched Don’t Look Up, which seems to be dividing opinion, but lots of people had recommended it. Not terrible, but at least 45 minutes too long, and felt at times like it wasn’t sure whether it was serious satire or slapstick.

Just short of 40 miles of running this week, culminating in a very muddy, but almost enjoyable, county cross-country championships (which sounds more impressive than it is – you only have to be a member of a club to enter, and not actually be fast or anything).

Categories
Weeknotes

Weeknotes: 2022 week zero

Having gone almost two years without posting here (and I can’t even really blame the pandemic), I thought I’d have a go at writing weeknotes. I’m not sure yet whether I’ll really have anything interesting to include each week (I suspect not), but we’ll see. Maybe it’ll even encourage me to do more stuff so I can write about it.

Apparently week one of 2022 is the one that started today, so this is more of a preface and a trial run. I’d almost forgotten I had this blog too – I might migrate it to something more towards the static end of things at some point. Or, you know …

As usual, running featured quite heavily (47.3 miles), including drawing a reindeer, and the traditional NYE idiocy of cycling to Felixstowe, running on the prom for 45 minutes, getting in the sea (shoulders under and minimum one stroke – them’s the rules), then drying off and heading home via various pubs. Ended up in the Cricketers (’spoons), where they had Abbot Special Reserve for £1.99, and – as usual – Mr Harper was a bad influence.

I’m never usually one for staying up to see in the new year, but this time I excelled myself by going to bed at around 8pm. Got woken up by the fireworks, naturally.

Did a bit of geocaching for the first time in ages. A few years ago I reached the point where I’d either found, or failed to find, all the ones within cycling distance, but there are plenty more now, and I bagged a few while I was out running.

I compiled another cryptic crossword; something else I haven’t done for a few months. And I hit a 365-day streak learning Spanish on Duolingo, although I had cheated slightly by using a “streak freeze” a couple of times – once when I went to do my daily practice and realised it was gone midnight, and again on New Year’s Eve, having run out of hearts (and time) in the morning before quite finishing a lesson, and unsurprisingly forgotten to try again once I arrived home after the Abbot Special Reserve.

I’m still intermittently practicing drums – mostly using Melodics at the moment for a bit more structure, after spending a few months just repeatedly trying to learn and play along to the same few songs.

I even did a couple of days’ work between Christmas and New Year because I was bored (though that doesn’t help with using up the leave I need to take before April). For some reason it seems easier to spend time at this time of year on technical debt type stuff more than new features, although I did a bit of both. Nothing very exciting, although I will say that I still really like Phoenix LiveView (at least for the kind of work I mostly do, which is internal-use web apps with no API, and no mobile apps).

Well that’s more than enough for one week I reckon. I wonder how long I’ll manage to keep this up …

Categories
Food

Marmalade

A few people have asked for my marmalade recipe over the years, and up to now I’ve never bothered writing it down because it’s pretty much just the basic one you’d find in any simple cookbook. Nevertheless, here it is. Partly so I have somewhere to point people, and partly as a handy place to remind myself how much I normally make, so I know I’ll have enough to last the year.

TL;DR

Here’s the basics:

1:2:2 oranges/sugar/water, plus half a lemon for every pound of oranges. Pips & pith in muslin, then boil with water, juice and orange peel for 1½ to 2 hours, until half the water’s gone. Add sugar, heat gently and stir until dissolved, then boil for 15 minutes. Test for set, and boil more if necessary. Pour into heated jars.

Now for the long version …

Ingredients

This is how much I make in one batch, and will produce somewhere around 12–14 jars. Two batches lasts me nearly a year, and three gives me a few jars left over (I eat a lot of marmalade). I guess 2½ would be about right.

Seville oranges generally only appear in the shops for a few weeks around January, at least in the UK, so get them while you can. Quantities aren’t particularly critical, which is why the imperial and metric versions don’t quite line up.

  • 4lb/2kg Seville oranges
  • Two lemons
  • 8lb/4kg preserving sugar (not jam sugar, which has added pectin)
  • 8pt/4l water

You’ll also need a jam pan (or some other massive saucepan – it should be big enough for at least twice the amount of water you’re starting with), some muslin squares (Lakeland sell them), jars, and a nice sharp chef’s knife.

Preparation

Put the muslin in a sieve above a bowl (or the pan), and squeeze the lemon juice and pips into it. Discard the outside of the lemons.

Get rid of the little stalk things from the oranges, and cut them into quarters (the oranges, not the stalks).

Cut the oranges into quarters

Give each quarter a quick squeeze into the muslin, then scrape the pips and pith in as well. You want to boil these bits up to get the pectin that makes the marmalade set, but you don’t want them in the finished product, which is where the muslin comes in.

Squeeze out the juice, and scrape the pips & pith into the muslin

Once everything’s in the muslin square (you might need two), pull the corners together, give it a good squeeze, then tie it up with string. Put the tied-up bag(s) into the pan, along with the orange juice and the water.

Tie up the muslin square

Now for the tedious bit – cutting all those orange peel quarters into thin strips. If you like chunky marmalade you’re in luck, as it won’t take quite as long, but either way you’ll probably want to put the radio or a podcast or something on to relieve the boredom. Keep your wits about you though, as it’s easy to lose concentration and shave a bit off the end of one of your fingers.

I find the easiest technique is to cut each bit of peel into three lengthwise, then finely chop across the three bits.

Finely chop the peel

Add the chopped peel to the pan, and bring to the boil. Simmer uncovered for 1½ to 2 hours, until the volume of water has reduced by about half (and what’s left will mostly be peel). Remove the bag(s) of pips, then scrape any pieces of peel back into the pan, and squeeze as much juice as you can back into the pan too (this is harder than it sounds, because it’ll obviously be far too hot to touch).

Boil the peel and bag of pips

Now add the sugar. This is the point when you realise that marmalade is basically just sugar with some bits of orange peel in it.

At this point, put your jars (without their lids) in the oven at 100ºC (to avoid any risk of cracking when you pour the marmalade in), and put a small plate or saucer in the fridge (you’ll use this to test for the setting point later).

Add the sugar (so much sugar!)

Heat gently, and keep stirring until the sugar has all dissolved. Then turn up the heat and bring to the boil. Boil vigorously for 15 minutes, stirring occasionally to stop it sticking to the bottom of the pan. A word of warning: as you scrape the bottom of the pan with the spoon, it tends to encourage some mildly volcanic eruptions, and you probably don’t want to be splashing boiling syrup on yourself.

If you do end up burning some of the peel, don’t worry too much – it doesn’t seem to particularly affect the final flavour. You’ll end up with a darker finished product though, so maybe chuck in the word “caramelised” and pretend it was deliberate.

Double, double toil and trouble …

After 15 minutes, turn off the heat, take half a teaspoon of marmalade out of the pan and tip it onto the cold plate from earlier. Leave it for a minute, then drag your fingertip across it. If it forms a skin (or has already taken on a jelly-like consistency), then your marmalade is ready. If not, boil for another five minutes and try again. I’ve occasionally had nightmare batches that took ages; probably because I hadn’t boiled away enough water.

Leave to cool slightly (five minutes or so is fine), then pour, ladle or otherwise decant into jars. A jam funnel will greatly reduce the amount of mess generated by this stage, although it won’t be able to help with the inevitable sticky splashes you’ll already have on and around your cooker.

Transfer to jars

And that’s it! Ready to spread on your toast in the morning.

The finished product
Categories
Elixir Software

Playing with property-based tests

I’ve never really tried property-based testing in anger before (just a brief play with QuickCheck during an ill-fated attempt to learn me some Haskell), but I recently came across a practice kata that seemed perfectly suited, so thought I’d give it a go. Unfortunately I haven’t got round to reading the book that I got in a recent PragProg sale, so it was mostly fumbling around and a lot of Googling.

The problem, basically, is this:

Given a list of names, generate a set of randomised “secret Santa” gift tags, such that no-one ends up giving themself a present.

This kind of thing isn’t always easy to test using traditional example-based testing, because of its intrinsic unpredictability, but using a property-based approach instead allows us to define a set of rules that should always hold true, and have the test framework automatically throw a variety of input data at your code and check that the rule isn’t broken.

The rules I came up with were:

  • returns the same number of tags as supplied names
  • includes each name once as a sender
  • includes each name once as a recipient
  • does not tag anyone’s present as from themself

The framework I used was PropCheck, which is an Elixir wrapper around the Erlang proper library. It acts as an extension to ExUnit, so you can mix and match example- and property-based tests.

PropCheck comes with a number of built-in generators, which will produce instances of primitive types like integers, strings, lists etc. Strings behave a bit oddly, thanks to the way Elixir strings are based on Erlang binaries, whereas Erlang natively uses charlists (also supported, but not commonly used, in Elixir). When QuickCheck generates binaries, it tends towards using zeros as the individual codepoint values, rather than a printable character. This means you’ll see values like <<0, 1, 0>>, rather than "ABA" (which is equivalent to <<65, 66, 65>>).

Initially, I picked a simple list of binaries as the input, meaning I could define my first property like this:

property "returns the same number of tags as supplied names" do forall names <- list(binary()) do length(SecretSanta.tags(names)) == length(names) end end
Code language: PHP (php)

For the later properties though, it became clear that not all lists were valid input. Specifically neither empty nor single-element lists really made sense, and neither did zero-character names. This is one of the advantages of this kind of testing – the framework will come up with all sorts of edge cases, forcing you to stop and think whether they need to be handled, or explicitly excluded.

Restricting the input data meant writing a simple custom generator. I actually split it up into several smaller generators, which built on each other. First, let’s have some more realistic names. For simplicity, let’s say a firstname and a surname are each eight characters long (and are mysteriously made up of non-printable characters!):

defp name() do let([first <- binary(8), second <- binary(8)], do: first <> " " <> second) end
Code language: JavaScript (javascript)

The let macro allows us to take some existing generators, and transform their output – in this case joining two strings with a space to make a name.

Now we can create a list of names. My first approach to making a minimum length list didn’t work, but after getting some help via a GitHub issue I ended up with this, which basically says “append two names to a list of names”. That list may be empty, but by appending two values to it we are guaranteed a minimum of two elements:

defp at_least_two_names() do let([first <- name(), second <- name(), many <- list(name())], do: [first, second | many] ) end
Code language: PHP (php)

Again, although it seemed at first that this was enough, the tests revealed another hidden assumption: the names in the list need to be unique. I extended the previous generator to enforce this using such_that. This applies a guard to the generated values, forcing a new one to be produced if the condition is not met. If it can’t find a suitable value after a configurable number of tries, it will give up and fail the test.

defp at_least_two_unique_names do such_that(names <- at_least_two_names(), when: length(Enum.uniq(names)) == length(names)) end
Code language: HTML, XML (xml)

With all those in place, here are the implementations of the properties listed earlier:

property "returns the same number of tags as supplied names" do forall names <- at_least_two_unique_names() do length(SecretSanta.tags(names)) == length(names) end end property "includes each name once as a sender" do forall names <- at_least_two_unique_names() do tags = SecretSanta.tags(names) names -- Enum.map(tags, & &1.from) == [] end end property "includes each name once as a recipient" do forall names <- at_least_two_unique_names() do tags = SecretSanta.tags(names) names -- Enum.map(tags, & &1.to) == [] end end property "does not tag anyone's present as from themself" do forall names <- at_least_two_unique_names() do tags = SecretSanta.tags(names) tags |> Enum.all?(&(&1.from != &1.to)) end end
Code language: JavaScript (javascript)

And for completeness, here’s my implementation of the algorithm. It may not be pretty or efficient, but at least I know it works!

defmodule SecretSanta do def tags(names) do allocate(Enum.shuffle(names), Enum.shuffle(names)) end defp allocate([], []), do: [] # guard against leaving the same last person in both lists defp allocate([from, name], [to, name]) do allocate([from, name], [name, to]) end # if the two heads are the same, shuffle the second list defp allocate(from, to) when hd(from) == hd(to) do allocate(from, Enum.shuffle(to)) end defp allocate([from | from_tail], [to | to_tail]) do [%{from: from, to: to} | allocate(from_tail, to_tail)] end end
Code language: PHP (php)

Categories
Elixir

GenServer: Elixir’s Swiss Army Knife

If you ask someone what the coolest thing about Elixir is, chances are they’ll start with pipelines (the |> operator even made it to the front cover of the PragProg book). But the next thing they mention will probably have something to do with OTP in general, and/or GenServer specifically, and these are arguably more powerful.

I’ve been using Elixir more and more lately, and have started to get my head round how to use GenServer (and friends) for a bunch of different things. I thought I’d write up some of those examples, in case it’s helpful to other Elixir learners (it takes a while to get your head round this different way of structuring code, and I’m definitely not there yet).

You can find the code used in the examples on GitHub, and the commit history should follow the stages in text.

Important caveat

I have no idea what I'm doing

As I said above, I’m quite new to all this, so take everything with a pinch of salt. There are probably better ways to solve all these problems, and better ways to structure GenServer applications. Also, in the interests of preventing an already far too long post getting longer, I’ve completely ignored things like testing, code organisation and so on.

Hello world

First of all, we’ll create a standard mix application:

$ mix new hello_world
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/hello_world.ex
* creating test
* creating test/test_helper.exs
* creating test/hello_world_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd hello_world
    mix test

Run "mix help" for more commands.

Mix gives us a skeleton top level module for the application – in this case HelloWorld. Let’s tweak that slightly to make a function that simply prints a message to the terminal:

lib/hello_world.ex:

defmodule HelloWorld do
  def hello do
    IO.puts "Hello World"
  end
end

Nothing particularly interesting there, but we’ll just check that it works in iex:

$ iex -S mix
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Compiling 1 file (.ex)
Generated hello_world app
Interactive Elixir (1.5.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> HelloWorld.hello
Hello World
:ok

That all looks OK then. From here on I’ll tidy up the output from iex sessions a bit for readability, removing the version info, compiler output etc.

Testing, 1 2 3

OK, on to our first task. We want to implement a counter, so that successive calls to HelloWorld.hello will print “Hello World 1”, “Hello World 2” etc. We’ll split this task into a few steps, starting with simply implementing a counter.

Now, the thing with functional languages is that they don’t have an obvious place to store mutable state between function calls. In Elixir’s case, the answer to this (borrowed from the underlying world of Erlang and OTP) is to maintain state in the event loops of processes. OTP provides a handy abstraction for this pattern in the form of gen_server, and Elixir wraps this up in GenServer, along with a sprinkling of metaprogramming to remove a lot of the tedious admin.

In fact Elixir has a more specific version of GenServerAgent – whose only job is to hold state, but we’ll stick with the vanilla GenServer for the sake of this example.

Here’s our first stab at a counter server:

lib/hello_world/counter.ex

defmodule  HelloWorld.Counter do
  use GenServer

  def handle_call(:next, _from, count) do
    {:reply, count, count + 1}
  end
end

There’s not a huge amount of code there (although admittedly more than there would be in a simple class or struct in an OO language), but let’s run through what there is.

First we use the GenServer module, which triggers some macro/metaprogramming magic to pull default implementations of all the callbacks required by the GenServer behaviour into the HelloWorld.Counter module. This saves a lot of boilerplate compared to the underlying gen_server from Erlang.

Now all we need to do is implement whatever custom behaviour we need – in this case storing an integer, and when asked returning its current value and incrementing the stored value. Our counter is a tiny server running in its own Erlang process, and at its centre is an event loop, which sits waiting until it receives a message, then handles the message appropriately. The event loop is initialised with some initial state (which could be anything, but here it’ll just be a simple number), and every time it handles a message it calls itself again with the new value for the state.

Thanks to GenServer, we don’t need to implement the event loop ourselves; we just need to write callbacks to handle the messages we expect to receive. The three main callbacks are handle_call (when the caller is expecting a response), handle_cast (when it isn’t), and handle_info (for messages that aren’t sent using call or cast). We need to pass back the next counter value when asked, so we’ve implemented the former.

The contract says that handle_call has to take three arguments: the message received (we pattern-match this to :next, which is the only message we care about), information about the sender (which we ignore), and the current state (in our case the counter’s value). There are several valid response tuples, but we’ve used the most common. This consists of the atom :reply (to indicate that we want to send a reply to the sender of the message), the reply itself (for us that’s the counter value), and the new state (the incremented count).

OK, let’s try that out in iex. First we’ll use GenServer.start_link to start a new process for our counter. We pass in our counter module, along with the initial state (the number we want to start counting from). We’ll pattern-match the reply to the expected success tuple, capturing the process ID (pid):

$ iex -S mix
iex(1)> {:ok, pid} = GenServer.start_link HelloWorld.Counter, 1
{:ok, #PID<0.206.0>}

Now let’s check that the counter works as expected, by repeatedly sending the :next message to the pid of the counter we started:

iex(2)> GenServer.call pid, :next
1
iex(3)> GenServer.call pid, :next
2
iex(4)> GenServer.call pid, :next
3

Good, we seem to have a working counter. There’s a small snag though: we have to keep hold of its pid (another piece of state) for as long as we want to use it, so we seem to be back where we started. Fortunately Elixir has us covered. We can specify a name when starting a server, then use that name instead of the pid when sending messages, and have Elixir find the correct process for us:

iex(1)> GenServer.start_link HelloWorld.Counter, 1, name: :hello_counter
{:ok, #PID<0.151.0>}
iex(2)> GenServer.call :hello_counter, :next
1
iex(3)> GenServer.call :hello_counter, :next
2
iex(4)> GenServer.call :hello_counter, :next
3

So far, so good. It seems a bit clunky to have to call GenServer functions directly from any code that wants to use the counter though, so let’s add a couple of wrappers in our own module. We’ll add our own start_link function, which takes a tuple of the counter name and the initial value, and a next function that wraps the Genserver.call:

lib/hello_world/counter.ex

defmodule  HelloWorld.Counter do
  use GenServer

  def start_link({name, start_at}) do
    GenServer.start_link __MODULE__, start_at, name: name
  end

  def next(name) do
    GenServer.call name, :next
  end

  def handle_call(:next, _from, count) do
    {:reply, count, count + 1}
  end
end

Let’s give that a spin, and while we’re at it prove that counters with different names behave independently:

iex(1)> HelloWorld.Counter.start_link {:counter_1, 1}
{:ok, #PID<0.240.0>}
iex(2)> HelloWorld.Counter.start_link {:counter_2, 10}
{:ok, #PID<0.242.0>}

iex(3)> HelloWorld.Counter.next :counter_1
1
iex(4)> HelloWorld.Counter.next :counter_1
2
iex(5)> HelloWorld.Counter.next :counter_2
10
iex(6)> HelloWorld.Counter.next :counter_1
3
iex(7)> HelloWorld.Counter.next :counter_2
11

All looking good so far. All we need to do now is to start a counter with our application, so we can access it from HelloWorld without having to worry about whether it’s already running. For this, we’ll need a Supervisor. Normally you’d generate a supervised application by providing the --sup flag to mix new, which we didn’t do. No problem though: we can just step up a directory and re-run the command, making sure not to let it overwrite our modified HelloWorld module:

$ cd ..
$ mix new --sup hello_world
The directory "hello_world" already exists. Are you sure you want to continue? [Yn] y
* creating README.md
README.md already exists, overwrite? [Yn] y
* creating .gitignore
.gitignore already exists, overwrite? [Yn] y
* creating mix.exs
mix.exs already exists, overwrite? [Yn] y
* creating config
* creating config/config.exs
config/config.exs already exists, overwrite? [Yn] y
* creating lib
* creating lib/hello_world.ex
lib/hello_world.ex already exists, overwrite? [Yn] n
* creating lib/hello_world/application.ex
* creating test
* creating test/test_helper.exs
test/test_helper.exs already exists, overwrite? [Yn] y
* creating test/hello_world_test.exs
test/hello_world_test.exs already exists, overwrite? [Yn] y

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd hello_world
    mix test

Run "mix help" for more commands.
$ cd -

This has created the module HelloWorld.Application (in lib/hello_world/application.ex) to describe our application, and also added an extra line to mix.exs setting this as the main module to run at startup. Here’s the default module as generated:

lib/hello_world/application.ex

defmodule HelloWorld.Application do
  # See https://hexdocs.pm/elixir/Application.html
  # for more information on OTP Applications
  @moduledoc false

  use Application

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Starts a worker by calling: HelloWorld.Worker.start_link(arg)
      # {HelloWorld.Worker, arg},
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: HelloWorld.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

All we need to do is add a worker for our counter. We supply the module name and the argument to pass to start_link:

lib/hello_world/application.ex

  ...
  def start(_type, _args) do
    children = [
      {HelloWorld.Counter, {:hello, 1}},
    ]
  ...

Now we can use the counter in our original hello function, safe in the knowledge that it’ll have been started before our application runs:

lib/hello_world.ex

defmodule HelloWorld do
  def hello do
    IO.puts "Hello World #{HelloWorld.Counter.next :hello}"
  end
end

Look! It works!

iex(1)> HelloWorld.hello
Hello World 1
:ok
iex(2)> HelloWorld.hello
Hello World 2
:ok
iex(3)> HelloWorld.hello
Hello World 3
:ok

Tick, tock

So far, so good. It’s tedious having to keep asking the computer to greet us though – what we really want (bear with me here) is for it to keep doing so, once every second. How can we build a timer to make this dream a reality? The answer, I’m sure you’ll be unsurprised to hear, is with another GenServer. Here’s one way of doing it:

lib/hello_world/timer.ex

defmodule  HelloWorld.Timer do
  use GenServer

  def start_link(args) do
    GenServer.start_link __MODULE__, args
  end

  def init(_args) do
    send self(), :tick
    {:ok, nil}
  end

  def handle_info(:tick, state) do
    IO.puts "TICK"
    HelloWorld.Counter.next(:hello) |> HelloWorld.hello
    Process.send_after self(), :tick, 1_000
    IO.puts "TOCK"
    {:noreply, state}
  end
end

So, what’s going on here? Firstly, we implement start_link again. This time we’ve decided to make the timer specific to this one very important job, so there’ll only ever be one instance of it running. This means we don’t need to keep track of separate pids or names, so we’ll simply name the process after the current module (ie HelloWorld.Timer), which we obtain using the __MODULE__ macro. Our server doesn’t require any arguments, but we’ll dutifully pass whatever we’re given to GenServer.start_link anyway.

In the init callback, which is called on startup, we’ll just send ourselves (the call to self() simply returns the current pid) an initial :tick message, to start things going. With that message dispatched in the background, we return a tuple indicating that all is well, with nil as our initial (and, as it turns out, permanent) state.

The main work happens in handle_info, which is the callback that handles receipt of arbitrary messages. When a :tick message arrives, we grab the next counter value and pipe it into HelloWorld.hello (which we’re about to change to accept a number rather than accessing the counter itself). Once that’s done, we trigger the next clock tick by sending a delayed message, using Process.send_after. This isn’t part of GenServer, which is why we’re using plain messages instead of cast.

We also top and tail things by printing “TICK” and “TOCK”, purely so we can see the order that things are happening (this will become useful shortly).

We can add the timer as another child process:

lib/hello_world/application.ex

  ...
  def start(_type, _args) do
    children = [
      {HelloWorld.Counter, {:hello, 1}},
      {HelloWorld.Timer, {}},
    ]
  ...

Finally here’s our pared-down HelloWorld module:

lib/hello_world.ex

defmodule HelloWorld do
  def hello(n) do
    IO.puts "Hello World #{n}"
  end
end

And this is what happens when we fire up iex again:

$ iex -S mix
TICK
Hello World 1
TOCK
TICK
Hello World 2
TOCK
TICK
Hello World 3
TOCK
[ ... and so on ]

Everything but the kitchen async

This all seems to be working, but what if saying hello was a really slow operation? As things stand, the call to HelloWorld.hello is synchronous, so we’re holding up the timer’s event loop while we wait for it to complete. Let’s push the work into the background, by turning HelloWorld itself into yet another GenServer. This time we’ll make it a single shot fire-and-forget job (normally you’d probably want to do this with a Task – another of Elixir’s GenServer specialisations). We’ll add a random sleep, to simulate something that took some time to complete.

Here’s the code:

lib/hello_world.ex

defmodule HelloWorld do
  use GenServer

  def hello(n) do
    GenServer.start HelloWorld, n
  end

  def init(n) do
    GenServer.cast self(), :hello
    {:ok, n}
  end

  def handle_cast(:hello, n) do
    :rand.uniform * 1900 |> round |> Process.sleep
    IO.puts "Hello World #{n}"
    {:stop, :normal, n}
  end
end

The hello function has been modified to start a server, with the number to append to the message supplied as an initial argument. In init, we send ourselves a :hello message, then return (with the number as the value to be used as the initial state), allowing the calling process to carry on with its business.

In handle_cast (with the message pattern-matched to :hello), we first sleep for up to two seconds, then print our traditional message, and finally instruct GenServer to cleanly terminate the process by returning :stop, with a reason of :normal and a value for the final state.

Let’s see what happens when we run the new asynchronous, sleepy hello program:

$ iex -S mix
TICK
TOCK
Hello World 1
TICK
TOCK
Hello World 2
TICK
TOCK
Hello World 3
TICK
TOCK
TICK
TOCK
Hello World 5
Hello World 4

It all starts out promisingly, with the message appearing after the timer tick finishes, which means the hello job is successfully running in the background. The fourth and fifth calls go a bit wonky though, appearing out of order (clearly the fourth one decided to sleep for over a second longer then the fifth). Depending on what we’re doing, this might not be a problem, but sometimes it is. The real-life application that inspired this blog post involved periodically sending configuration to routers. Most of the time one set of commands finished before the next was sent, but occasionally a particularly large job could overrun, in which case the next one needed to wait its turn.

Form an orderly queue

Let’s assume that we don’t want one hello to start until the current one has finished. To accomplish this, we need some kind of queue. Erlang/OTP does actually have a queue module, but that might be overkill for our needs. At this point we remember that the mechanism that passes the messages into our GenServer event loop is itself a queue – maybe that’s all we need?

Let’s tweak our module slightly, making it run continuously as a single server, rather than firing up a new one every second (this is probably far better practice anyway!):

lib/hello_world.ex

defmodule HelloWorld do
  use GenServer

  def start_link(args) do
    GenServer.start_link __MODULE__, args, name: __MODULE__
  end

  def hello(n) do
    GenServer.cast __MODULE__, {:hello, n}
  end

  def handle_cast({:hello, n}, state) do
    :rand.uniform * 1900 |> round |> Process.sleep
    IO.puts "Hello World #{n}"
    {:noreply, state}
  end
end

Note how hello is no longer responsible for creating a new process, but is now just a wrapper round cast, sending ourselves a message asking for the hello operation to happen. A server can only handle one message at a time, so any new request while it’s sleeping will be queued up in the process’s inbox waiting for the next go round the event loop. This will happen as soon as handle_cast completes (we’ve changed it to return a :noreply tuple instead of :stop, so the loop will continue).

We need to add the server to the child list:

lib/hello_world/application.ex

  ...
  def start(_type, _args) do
    children = [
      {HelloWorld.Counter, {:hello, 1}},
      {HelloWorld.Timer, {}},
      {HelloWorld, {}},
    ]
  ...

Let’s run it one last time:

$ iex -S mix
TICK
TOCK
TICK
TOCK
Hello World 1
TICK
TOCK
Hello World 2
TICK
TOCK
Hello World 3
Hello World 4
TICK
TOCK
TICK
TOCK
Hello World 5
TICK
TOCK
Hello World 6

Well that seems to have done the trick, and is probably more than enough GenServer for one post (sorry, it ended up much longer than I expected – thanks for sticking with it to the end!)

Categories
Software Uncategorized

The value of spiking: a cautionary tale

A while ago I built a little web app using the Strava API, that tells you how far you’ve run so far this year, and how many miles you’re projected to do my the end of the year if you keep up the same average.

As Strava lets you see your friends’ year-to-date mileage on their profile pages, I thought it would be a nice addition to generate a leaderboard to show where you stood against your running friends. The URI to query an athlete’s stats is https://www.strava.com/api/v3/athletes/:id/stats, which I’m already using to get the stats for the person using my app, so all I have to do is query that person’s friends list and map over all the IDs calling the same API with their IDs, right?

First things first, mock up the UI for the new page. After a bit of wrestling with Bootstrap I had some navigation elements I was happy with, so moved onto the core app. It turned out that the Strava library I’m using didn’t have a wrapper for the friends/followers API, but it was pretty easy to add one, so I made the required changes and submitted a pull request. Then a nice test-driven feature from the outside of the app, stubbing the boundary to the function that retrieved the stats for each friend, as that was already there for the current user and didn’t need to be changed. A data structure specifically to hold the friend’s name, profile URL and mileage to display in the table, and a public function to return a list of those structs, and the core was all done.

Finally I put it all together, wired the controller in the web app up to the core function to make the required query, and modified the view to display each friend’s details in a row in the leaderboard. Tests all green! Finally! Let’s try it out!

500 internal server error

Wait, that’s not supposed to happen. Lets look at the logs. Hmm, the call to the Strava API seems to be failing. Unauthorised. That’s odd.

Better take another look at the API documentation.

Totals and stats

Returns recent (last 4 weeks), year to date and all time stats for a given athlete. Only available for the authenticated athlete. This is the recommended endpoint when polling for athlete upload events.

Wait, what’s that bit in the middle again? “Only available for the authenticated athlete”. Oh. Realisation dawns. No leaderboard then, and a couple of days’ work wasted on a branch that’s never going to make it into master.

So what’s the moral of the story? If you’re building a feature that relies on something outside your control, don’t make assumptions about what works and what doesn’t. Build a spike first, to test your assumptions before ploughing your time into building the real thing.

Oh, and read the documentation.

Categories
Software

Phoenix ecosystem for Rails developers

I’ve been tinkering with Elixir and Phoenix a bit recently. I really like the language (it’s the first time since Ruby that a new (to me) language has seemed this much fun) and Phoenix seems like a rock solid framework, but I was interested in what the surrounding ecosystem was like compared to the much more mature Ruby/Rails landscape. This was just a tyre-kicking exercise – I only really took each tool as far as making sure I could install it and do the basics. Incidentally, a great place to find popular Elixir resources is Awesome Elixir.

The app I started creating (which does virtually nothing!) is on github, and here are some brief notes of the stuff I tried out:

BDD executable specifications

White Bread

Ruby equivalent: Cucumber

White Bread (because that’s what you make cucumber sandwiches with) is a Gherkin parser/runner. It works in much the same way as Cucumber.

It took me a bit of fiddling to get it working with Phoenix, but nothing too difficult. According to the docs it should generate a sample context (equivalent to a Cucumber step definitions file) when you first run it, but that failed. Once I’d copied the example from the docs though, it worked fine.

The most obvious difference in the steps is that they operate (as you would expect) in a functional style, passing the context from one step to the next rather than relying on shared state. This is actually quite nice, as it makes it very clear what data is being passed between steps.

Test data creation

Ex Machina

Ruby equivalent: Factory Girl

Completely painless to install and get started with, and seems to work very much like Factory Girl (although I only scraped the surface).

Blacksmith is another option, which I didn’t investigate.

Browser simulation/automation

Hound

Ruby equivalent: Capybara

First I tried TucoTuco (apparently it’s an animal similar to a capybara). The first problem I had was that Webdriver (a dependency) failed to compile under Elixir 2.2, so I had to drop back to Elixir 1.1.1 (I raise an issue against the project, and got a quick reply saying it’ll be fixed soon). I also had to explicitly add ibrowse as a dix dependency, as it seems to only be available from GitHub, not hex. I found TucoTuco’s documentation to be a bit sketchy, and never quite got it working with Phoenix.

Switching to Hound, I found the documentation to be much better. It still uses webdriver, so had the same compatibility issue as TucoTuco, but not ibrowse.

One difference compared with Capybara is that you need to manually start phantomjs (phantomjs --wd), rather than it being started for you. It should be easy enough to wrap it in an Elixir application, but I haven’t tried that yet.

Elixir version management

Exenv

Ruby equivalent: rbenv/RVM

Previously I’d just been using the latest version of Elixir from Homebrew, but having to downgrade because of the webdriver incompatibility set me looking for a version manager. Exenv follows the philosophy of rbenv rather than rvm, is dead simple to install and use, and just keeps out of your way and works flawlessly.

Unit tests/specs

ESpec/ESpec_Phoenix

Ruby equivalent: RSpec/RSpec-Rails

Pretty similar to RSpec. Includes the same kind of matchers, mocking, stubbing etc.

Style checking

Dogma

Ruby equivalent: Rubocop

The docs a bit out of date, but otherwise works more-or-less as expected, failing the build if the rules you specify are broken. I suspect it doesn’t have quite the range of rules or configurability as rubocop (yet).

If you just want gentle style suggestions rather than enforcement, consider Credo instead (which I didn’t try).

Slim templating

Phoenix_slime

Ruby equivalent: Slim

Works as expected. Seems to show up on Mix as phoenix_slim too, which is the one I specified. Might have been renamed to phoenix_slime? Who knows?

Test coverage

Ruby equivalent: Simplecov

Coverage is built into exunit and espec, using Erlang’s cover tool:

mix espec --cover

The output’s not great though, so enter Coverex, which produces output much more like simplecov’s.

Unfortunately both tools seem to mark things like use statements within defmodule blocks as uncovered, which leads to misleadingly low coverage scores. I can’t see any obvious way to have it fail the build on low coverage either. In ruby I normally fail if any line is uncovered (if you’re test-driving the code, that’s never going to happen, right?), so that’s two reasons I can’t do the same thing in Elixir.

Debugging

Ruby equivalent: ByeBug or Pry

This is built into IEx. In your code, do:

require IEx
IEx.pry

Make sure you start the server attached to an IEx session:

iex -S mix phoenix.server

Categories
Uncategorized

TDD example with Rails, Cucumber and RSpec

In the unlikely event that anyone wants to scroll through the whole history of the TDD demo Rails app I did the other day, I wrote a little script to munge it all into one page. And here it is.

Categories
Software

Telling a story with git

This may be the ugliest bash one-liner ever:

git log --pretty=%h | tail -r | while read sha ; do if [[ -z $(git rev-parse --verify $sha^2 2> /dev/null) ]] ; then echo '----' ; git show --summary $sha | tail -n+5 | egrep -v '^ (?:create|delete) mode' ; git --no-pager diff --color -U999 $sha~ $sha | awk '/diff --git/ { sub("^a/", "", $3) ; print "===" $3 } /diff --git/, /@@/ { next } { print }' ; fi ; done | ansifilter -H | sed 's/^----$/<hr \>>/;s/===\(.*\)/<h2>\1<\/h2>/' > history.html

It runs through the git history of a project, printing the full message for each commit, followed by a complete listing of each changed file, with added, removed or changed lines highlighted. It then outputs the result into a big ugly HTML file. Like this.