Tagged In monitoring :
Intelligent Metrics Monitoring
Here at Wealthfront we have many offline computations running in Spark. In some cases, small changes have caused a job to slow down dramatically, and other times, the size of the input may be growing and causing the job runtime to quickly increase. We normally check pipeline runtimes manually to make sure jobs are running… Read more
Continuous Deployment: API Compatibility Verification
We haven’t said much about our continuous deployment system recently. Mostly that’s because there hasn’t been much to say. We invest in systems and infrastructure through a process called proportional investment: we spend time on areas that cause us problems and our deployment infrastructure has performed well, requiring little incremental investment. However, recently we made… Read more
Belt and Suspenders: Safety and Trust in a Distributed System
Wealthfront is built on a traditional service-oriented architecture: requests from web browsers are dispatched to a cluster of Rails instances, which will in turn query back-end services. As a single page load might trigger tens of internal remote calls, it quickly becomes difficult to understand why a back-end service was called, who initiated the call,… Read more
IRC bots for Laziness, Impatience, and Continuous Deployment
1988 was an epic year. Not only were the movies “Die Hard”, “Coming to America” and “The Naked Gun” released, but according to Wikipedia, IRC was invented. A tried-and-true technology, constantly reincarnated in similar (perhaps inferior?) forms, IRC is a superb communications hub for a company because it affords so many things: Everyone can chat… Read more
Monitoring Long Running MySQL Queries with Nagios
We’ve been using Nagios as part of our monitoring infrastructure from some time now and it has been very useful to monitor our databases. To look at long running queries (those whose running time is measured in seconds) we use the show processlist command. Here is what the output looks like on an empty machine…. Read more
Complement TDD with MDA
Test Driven Development (aka TDD) is on the rise. Good developers understand that code with no proper testing is dead code. You can’t trust it to do what you want and its hard to change. I’m a strong believer in Dijkstra’s observation that “Program testing can be a very effective way to show the presence… Read more