As Wealthfront grows our company and products, scaling our engineering organization requires robust, well tested infrastructure and tooling provided by the DevOps team. In this blog post, I’ll go over how DevOps is defined at Wealthfront, why we use Go, and a brief overview of how we test Go.
What is DevOps at Wealthfront?
At Wealthfront, we define DevOps as the team that manages the hardware and software infrastructure to support and develop our products. While backend engineering teams mainly rely on Java, the DevOps team builds out infrastructure and automation services using Go. Every Go service we build must be well tested since quality code results in maintainable code and reliable experiences for our clients.
Why Use Go?
Back in early 2015, Wealthfront was modifying its engineering infrastructure and the DevOps team had the opportunity to design the architecture for their next set of tools. Go seemed attractive because it was a high level language that made concurrency simple and didn’t have the overhead of the JVM or an interpreted language. Go also made building and deploying artifacts easy compared to previously using Ruby to manage environments for each artifact. Some of the tools we use are open source and written in Go, so if there are any changes we want to add we’ll be familiar with the language and toolchain.
Today we use Go for any new projects or build tools for the DevOps team. Some examples of our Golang services are services to automate monitoring, database schema changes, and database metrics collection.
Design and Structure of Go Tests
To ensure that people will always test their code and maintain existing tests, the design choices for building our testing suites are rather simple – we use Go’s testing package and an open source logger, logrus. The standard library testing package provides similar syntax and documentation compared to other standard library packages – picking up the testing package would provide less documentation to sort through. As for logging, we chose logrus instead of using Go’s log package to use logs as objects instead of strings. Logrus logs can be treated as objects with specified fields – in our case each of our logs are JSON objects which can be easily parsed and understood if consumed by another service.
Our tests are broken down into unit tests and integration tests. Unit tests usually test the individual functions and components of the package and integration tests will set up a scenario, usually mocking service behavior and testing the expected output of the code. Since the tests usually require configuring log formatting, generating any necessary test fixtures, etc., we have a test runner to setup and configure any necessary logic in order to run the tests. In the package directory where main.go usually is, the test runner sits in a main_test.go file, which defines the TestMain method. The gist below is a basic example of what the test runner could look like.
Integration tests require repetitive steps for setup, which can easily be modularized and reused throughout tests, leading to test helper packages. Some examples of helpers that are versatile to use are functions to get populated structs, generating test databases, and cleaning up output from previously executed tests. Keeping the parameters more generalized will make the helpers easier to use in other tests. Helpers should also behave as they are defined, and are rigorously logged to describe even setup behavior that happens before test execution. Sometimes services will end up using similar or even the same helpers and, instead of pasting code, building an internal testing package that provides the necessary configurations and redundant helper logic becomes very valuable for testing in the long run.
Helper Function Example – Often when database queries are used in tests, we use helper functions to wrap the logic used to initialize database connections. Keep in mind that code snippets used are purposefully vague for sake of example.
Helper Function Example – Always make sure to define helpers that will tear down any fixtures or logic previously used in the testing environment.
Helper Function and Test Fixture Example – when constantly using a populated struct to test behavior, instead of creating different copies use a helper function to create the struct.
Putting It All Together
To summarize this brief overview, here’s an example of a typical Go unit test that relies on helper functions and fixtures. The intent is to keep the test structure simple, and easy to understand and write. To keep things simple I’ve included theoretical wrapper functions that provide necessary input to demonstrate the test.
The example test function name should describe what functionality is tested, and logs should describe the specific scenarios and errors. Any input or repetitive logic is defined in wrapper functions. We follow the typical pattern of comparing expected and actual outcomes for each testing scenario, and end the test by cleaning up the environment and scenario that the test used.
Simple patterns and structure make testing an easy routine rather than a burden, and testing is a necessity and value for the DevOps team because we want to build reliable systems for our engineers and clients. We’re always working on ways to refine our testing methodology, and are open to fresh perspectives and ideas on how to improve. Interested in how we write Go? The DevOps team is hiring!