Test Driven Development (aka TDD) is on the rise. Good developers understand that code with no proper testing is dead code. You can’t trust it to do what you want and its hard to change.
I’m a strong believer in Dijkstra’s observation that “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.”
Dijkstra’s statement doesn’t contradict TDD. The test is testing a limited state machine. We do hope will cover the bloody battlefield of production confronting live data from users but if when we find the users did something unexpected which broke our software, we add a test emulating the users behavior and fix the problem.
Introducing Monitoring Driven Architecture (aka MDA)!
MDA is a second line of defense for TDD. MDA means that you bake monitoring into your architecture. Once you have MDA and the software is written in a monitorable (it is a word) way, you can have a faster detection of problems and auto roll back of faulty code. On the other hand, it is not uncommon that a small number of users suffer from a problem which manifests itself in some NPE thrown in one of the logs once in a blue moon and the operations team finds about it after a long while.
This is why I’m so excited about John’s new Flexible Log Monitoring with Scribe, Esper, and Nagios deployment. It means that when we do find a problem we’ll of course fix it but in addition make sure we express it in the logs and have our monitoring tools pick it up and send alerts about its existent without counting on anyone to manually look at the logs.