Joining Wealthfront as a DevOps New Grad

June 24, 2014

I started at Wealthfront two weeks ago, and I can already say with little doubt that it was the best opportunity for me. Having taken Andy’s career advice, I was determined to start my career in the Bay Area and wanted to join a mid-sized company with momentum. I had followed the fintech space for a while, and Wealthfront’s mission of democratizing sophisticated financial advice for the masses really resonated with me.

First impressions

I got my first glimpse of the people and culture driving Wealthfront’s success during my interviews. I could tell very early on that Wealthfront was a heavily engineering-driven company with some of the best talent in the industry. Software engineers had a strong knowledge of the financial markets, and over 90% of employees actively wrote code. This extended all the way to directors and VPs. When it came to creating a strong engineering culture, Wealthfront really walked the walk. The interview process was a fun and unique experience for me, and I joined without hesitation.

Start

At Wealthfront, engineers hit the ground running. The presence of simple documentation and great mentors provides new hires with a rapid ramp-up that is neither stressful nor intimidating. We spend a lot of effort on documentation so that the learning process is successively easier for new hires. On my first day I was already pushing code to production, and within the first week I had fixed bugs, updated server configurations, and most importantly, written tests for all my commits.

The emphasis on testing at Wealthfront was obvious from the first day. Clients have trusted us with over $1 billion of their money, and with that trust comes an incredible level of responsibility. Everything is tested, even the simplest change to the code base. There are no QAs or SETs. Engineers are responsible for developing and maintaining quality software that meets the highest standards. The entire team goes to great lengths to make sure that clients’ trust in Wealthfront is rightly placed.

With automation in mind, we have software and systems that make continuous deployment simple and painless. This allows our software to be developed to high standards and deployed quickly to production, resulting in our ability to rapidly and reliably push enhancements and bug fixes to customers. The combination of a strong infrastructure and test-driven culture is a major reason why Wealthfront is leading the way in this space. Everyone here really is dedicated to building and maintaining Wealthfront as an engineering-driven company.

DevOps

One of my first tasks as a DevOps engineer was to fix an issue that was causing our backup monitor to fail. Backup Monitor is a service that maintains the state information of all backups. After diving into the issue, I found that the problem was an uncaught exception when requesting configured backups. The root cause of the problem was a short loss of network connectivity to the Chef server. Here is the original code:

I explored several different approaches:

  1. Ignore the issue and continue execution.
  2. Throw the exception back to the caller and have it implement the exception handling logic.
  3. Return an empty hash.
  4. Retry x number of times at expanding intervals until reaching a specified retry limit.
  5. Cache the configured backups and return the cache.

Each approach has tradeoffs. Ultimately, I went with (3).

The three criteria used to evaluate different approaches were (a) responsiveness to client, (b) client satisfaction with response, and (c) ease of implementation. Approach (1), ignoring the issue means that the backup monitor will crash again with an exception for a known issue. Approach (2), throwing the exception back to the caller does not result in the caller/client being very happy to have to handle the exception, and does not deal with the problem closest to the point of origin. Approach (4), retrying at expanding intervals will work, but may not return a result to the client for a long time. Failing fast is a more desired solution in this specific case. Approach (5), caching the configured backups and returning the cached version, is the best approach that ensures client’s satisfaction (the client receives the list of configured backups even when the chef server is down). However, implementing a proper key-value store like Redis for caching is a major undertaking that was out of scope for this exercise. I thought returning an empty hash satisfied the three criteria the best:

Lastly, writing tests is a critical part of the process of making changes to the code base. I wrote simple RSpec tests to test both success and failure cases:

Final Thoughts

My first weeks at Wealthfront have been great. Going forward, I’m really excited to dive even deeper into the stack. I am motivated and inspired by the opportunity to take on tough technical challenges and build valuable financial products. Providing high-quality advice and investment management at low cost is a difficult and noble mission, and I’m proud to be a part of the team making it happen.