Menu

Intelligent Metrics Monitoring

August 12, 2016

Here at Wealthfront we have many offline computations running in Spark. In some cases, small changes have caused a job to slow down dramatically, and other times, the size of the input may be growing and causing the job runtime to quickly increase. We normally check pipeline runtimes manually to make sure jobs are running… Read more

Continuous Deployment: API Compatibility Verification

March 19, 2013

We haven’t said much about our continuous deployment system recently. Mostly that’s because there hasn’t been much to say. We invest in systems and infrastructure through a process called proportional investment: we spend time on areas that cause us problems and our deployment infrastructure has performed well, requiring little incremental investment. However, recently we made… Read more

Belt and Suspenders: Safety and Trust in a Distributed System

May 02, 2012

Wealthfront is built on a traditional service-oriented architecture: requests from web browsers are dispatched to a cluster of Rails instances, which will in turn query back-end services. As a single page load might trigger tens of internal remote calls, it quickly becomes difficult to understand why a back-end service was called, who initiated the call,… Read more

IRC bots for Laziness, Impatience, and Continuous Deployment

December 10, 2010

1988 was an epic year. Not only were the movies “Die Hard”, “Coming to America” and “The Naked Gun” released, but according to Wikipedia, IRC was invented. A tried-and-true technology, constantly reincarnated in similar (perhaps inferior?) forms, IRC is a superb communications hub for a company because it affords so many things: Everyone can chat… Read more

Monitoring Long Running MySQL Queries with Nagios

November 15, 2010

We’ve been using Nagios as part of our monitoring infrastructure from some time now and it has been very useful to monitor our databases. To look at long running queries (those whose running time is measured in seconds) we use the show processlist command. Here is what the output looks like on an empty machine…. Read more

Complement TDD with MDA

January 10, 2010

Test Driven Development (aka TDD) is on the rise. Good developers understand that code with no proper testing is dead code. You can’t trust it to do what you want and its hard to change. I’m a strong believer in Dijkstra’s observation that “Program testing can be a very effective way to show the presence… Read more