Building Knod Part 2: Workflow and Testing

This is part 2 in a series about building Knod, a tiny HTTP server designed to help with front-end prototyping.

One of the first challenges I faced in building Knod was establishing a good workflow. Using the REPL as a development tool is one of the great joys of using Ruby and languages in the LISP family. I tend to experiment with different things on the REPL when the domain isn’t well known and I want to spike on a solution – the exact circumstance faced at the outset of this project. However, the REPL falls short when building a server because, while a server will certainly run in a REPL, it provides no useful output. We ultimately need a client to connect, send a request, and ensure it receives an appropriate response. Furthermore, since the server will block while it is waiting for a connection, we cannot run the server and test it in the same REPL. When I started building Knod, my workflow was as follows:

  1. Split the terminal
  2. Start the server in one terminal session
  3. Use cURL or Net::HTTP in the other session to make a request
  4. Use pry-debugger to step through the request/response cycle.
  5. Change the code as necessary
  6. Shut down and restart the server
  7. Repeat steps 1-6 until the desired result is achieved

Repeatedly stopping and starting the server and jumping back and forth between terminal sessions became tedious almost immediately so I started looking for ways to streamline my workflow. There were two problems to solve – reloading the server every time I made a change, then rerunning a series of requests to the server to see if the changes had the desired effect without breaking anything that was already working. I ultimately needed was a faster, more repeatable way to test the server’s behavior.

Ruby’s standard library includes almost everything one needs for behavior-driven development in the form of MiniTest::Spec. Its syntax will look immediately familiar to anyone used to RSpec:

1
2
3
it 'should add numbers' do
  (2 + 2).must_equal 4
end

MiniTest does not provide a way to send HTTP requests out of the box. I initially tried using Net::HTTP to make requests. It looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
describe Knod, 'a tiny HTTP server' do
  describe 'PUT' do
    before do
      path = '/items/81.json'
      @host = '0.0.0.0'
      header = {'Content-Type' => 'application/json'}
      @request = Net::HTTP::Post.new(path, header)
      @request.body = {id: 81, state: 'swell', favorite_animal: 'giant squid'}.to_json
    end

    it 'returns a 200 on success' do
      response = Net::HTTP.new(host, port=4444).start {|http| http.request(request) }
      response.code.must_equal '200'
    end
    # more tests
  end
end

This is awful. Tons of setup for just a few tests. At this rate I would end up spending more time writing tests than on the actual server! I received an excellent piece of advice from my friend Eno Compton – create a wrapper class for Net::HTTP to cut down on boilerplate. Dan Knox provides a great rundown of creating such a wrapper class in this blog post. With a small wrapper class suited to my purposes, tests became much easier to read and write:

1
2
3
4
5
6
7
8
9
10
11
12
13
describe Knod, "a tiny http server" do
  let(:connection) {Connection.new("http://0.0.0.0:4444")} # Net::HTTP wrapper

  describe 'PUT' do
    let(:path) {"index/81.json"}
    let(:data) { {state: 'swell', favorite_animal: 'giant squid'} }

    it 'returns a 200 on success' do
      response = connection.put path, data
      response.code.must_equal '200'
    end

    # considerably more tests...

We could even take this a step further and delegate all of the HTTP verbs to our connection to almost exactly mimic the functionality of rspec-rails or minitest-rails:

1
2
3
4
5
6
7
8
9
10
11
describe Knod, 'a tiny http server' do
  extend Forwardable
  let(:connection) {Connection.new("http://0.0.0.0:4444")}
  def_delegators :connection, :get, :put, :post # ... all the verbs

  describe 'PUT' do
    # after setup:
    it 'writes to the local path' do
      put path, data
      File.file?(path).must_equal true
    end

With the Net::HTTP wrapper class complete, I was halfway to a good BDD setup – I could quickly write useful tests, but still had to stop and start the server between each test run to pickup code changes. I revisited Jesse Storimer’s Working With Ruby Threads for inspiration:

I’ve been saying the GIL prevents parallel execution of Ruby code, but blocking IO is not Ruby code… MRI doesn’t let a thread hog the GIL when it hits blocking IO.

This means we can just start the server in its own thread before the test suite starts running. At the top of our test file:

1
2
3
4
require 'minitest/autorun'
require 'knod'

Thread.new { Knod.start(port: 4444) }

The only issue with this implementation is that we are in trouble if another server is already running on the port specified when we start the server. The solution lies in an interesting propery of Ruby’s socket library: A new instance of TCPServer initialized with a port of 0 will automatically select an open port in the ephemeral range:

1
2
server = TCPServer.new('0.0.0.0', 0)
port   = server.addr[1]

Setting up tests this way took a couple of hours that I could have spent on the server, but I think it was time well spent. Subsequent features became much easier to build with the improved workflow a test suite provided and I gained insight into how DSLs like rspec-rails work in the process.

Next up in this series: A post on higher-level learnings that came out of the gem-building process.