Build System Integration Testing with Chef

At Wealthfront, we’re constantly pushing to use cutting edge technologies to take advantage of new language features, improved compilers, new libraries, and the like. Downtime in our build system would be a big problem with how often we like pushing new code to production, so we want a process that would allow us to upgrade parts of our build system without risking unforeseen compilation errors and library incompatibilities. This post describes how we use integration testing with TestKitchenChef, and Serverspec to solve this problem.

Background

I joined Wealthfront relatively recently, previously having developed software tools for satellite operations (think applications DirecTV would use to make sure Game of Thrones is transmitted properly to your TV), and dev-ops was not something I’d had any experience with. Everyone at Wealthfront encourages cross-functional learning, so the dev-ops team was very helpful in getting me up to speed with what I needed to know and providing feedback on my work on this project. I was eager to find a solution that fit well with our values of automating anything that can be automated and adding safety to our processes, and I was able to learn a lot through this project.

Our build system consists of a handful of machines compiling code and running tests across many different repositories. Whether it is using a new Protobuf compiler or migrating to a new Java version, we often need to upgrade portions of our build system. Because we don’t want to risk rendering the build system unusable for any period of time, we can’t just blindly assume upgrading compiler versions will go smoothly; we need to systematically ensure any changes to our build machine configurations are compatible with their current uses.

As a concrete example, we recently upgraded JDK versions across our codebase. We knew a number of our repositories would have various issues with the upgrade, having code that was not compatible with the new JDK. Because of this, we wanted a way to find these incompatible bits of code in our repository so that we could fix them before performing the upgrade.

One solution might have been to have someone run tests on an isolated machine using the new JDK. This would have been done before deploying anything to our production build systems, identifying incompatibilities before they became a major headache. However, at Wealthfront we don’t believe in tests that can not be automated. Instead, we prefer a solution that continuously tests our full build system.  We decided to leverage our existing Chef/TestKitchen/Serverspec development pipeline to make our lives easier and our processes safer.

Overview

The following diagram gives a high-level overview of what the resulting process looks like for testing changes to our build machine configuration:

rob_blog_post_chef_ci

  1. Jenkins, our continuous integration system, identifies changes on one of our side branches in our git server, and it spawns a new slave to run integration tests on that side branch.
  2. The Jenkins slave clones the relevant repository, which contains the Chef cookbook for putting together one of our build machines.
  3. Using Chef-client, the Jenkins slave puts together all the necessary components of the build machine and brings it up.
  4. TestKitchen runs the Serverspec tests on the newly spun-up build machine.
  5. Results are pushed back to the Jenkins master and the temporary build machine is torn down.

In the sections below, we’ll go over in a little more detail what the Chef and TestKitchen setup looks like for this, which corresponds to steps 3 and 4 above. The example shows a simplified version of the JDK upgrade we performed.

Codebase Setup

For simplicity’s sake, let’s assume we have a very small codebase, consisting only of the following (troubling) Java file:

public class HelloWorld {
  public static <T> T getMessage() {
    return (T) "Hello, world!";
  }
  public static void main(String[] args) {
    System.out.println(getMessage());
  }
}

An old JDK 7 compiler would compile this, albeit with warnings for unsafe operations. Upgrading to a newer JDK 8 compiler, however, would fail to compile, giving us error for multiple matching “println” methods. Obviously, we could have caught this by trying to compile the codebase with the new JDK manually, but relying on someone to do this for all of our repositories every time we want to upgrade a library would be labor intensive and error prone. Instead, we can use Chef to ensure our build system stays happy.

Let’s assume our build and test procedure is as simple as the bash script below, named “integration_test.sh”:

#!/bin/bash
set -e
 
javac HelloWorld.java
if [[ $(java HelloWorld) = $'Hello, world!' ]]; then
    exit 0
else
    exit 1
fi

This simply compiles our Java code, runs it, and validates the output. The script exits with a normal exit code if all is as expected, otherwise it exits with a simple error code.

Chef Configuration

If our JDK is provided by the “java” recipe, we can update our Chef recipe to include the following to install our integration test script:

include_recipe 'java'
cookbook_file '/home/vagrant/HelloWorld.java' do
  source 'HelloWorld.java'
  mode 00644
  owner 'vagrant'
  group 'vagrant'
end
cookbook_file '/home/vagrant/integration_test.sh' do
  source 'integration_test.sh'
  mode 00755
  owner 'vagrant'
  group 'vagrant'
end

This will install anything specified in the java recipe and copy our code file and integration test script onto the machine. From here, the only other addition we need is to create a specfile test to run our integration test:

require 'spec_helper'
describe 'build_integration_test' do
  describe command("/home/vagrant/integration_test.sh") do
    its(:exit_status) { should eq 0 }
  end
end

In English, this has a test named “build_integration_test”, which runs our “integration_test.sh” script, and ensures that the exit code is 0. Assuming the rest of the cookbook is set up properly, a simple “chef exec rake” then runs our integration test to see if the configuration is ready for production. We can update our “jpackage” recipe to install a new Java compiler and be confident any problems in compiling our codebase will be revealed in our integration tests.

Wrapping Up

To further automate everything, we keep our Chef configurations version controlled and have Jenkins setup to automatically poll for changes and run “chef exec rake” on each of our side branches. Merging any changes to our build system is predicated on the success of that Jenkins job. Obviously, the above example is simple, but extending it is a straightforward exercise. We could easily replace HelloWorld.java with a script to clone all of our git repositories, and then our integration test could run steps in our maven lifecycle.

Interested in working on other cool projects like this? Join us.