Keeping the trunk stable matters a lot at Wealthfront. We do continuous deployment, and when the build is broken our deployment chain stops. We have a flashing light in the office and a pre-commit hook which prevents any commit until the build is fixed.
The best way to know if a commit would break the build beforehand would be to run all tests on the developers machines before committing to the central repository. Even though we’re working hard to keep our build time below 5 minutes (parallelizing-junit-test-runs, less-io-for-your-java-unit-tests), we think that commiting code often is important and we can’t offered to wait a few minutes to run all tests each time.
That’s why we came up with a reasonable subset of tests that we run on our local machines before committing. The subset is made of global tests which, in the past, have often been the cause of broken builds:
- DependencyTest (JDepend based test)
- ForbiddenCallTests (byte-code analysis to prevent the usage of “bad” third-party methods, such as using a Date without a TimeZone)
- JavaBadCodeSnippetsTest (source code analysis to identify bad code)
- VisibilityTest (improve java access control)
- QueriesAreTestTest (ensures that all queries are tested)
and a dynamic list of tests which is generated by looking at which source files have been modified. Since we use the standard convention that each source file has an equivalent test file with the same name plus a suffix (‘Test’ for java or R, ‘_spec’ for ruby, …), it’s easy to identify which tests might be broken by the current changes using a version control system. The svn and git implementations are very similar – they list of all the changes by calling the version control system, and find the corresponding unit test.
Individual developers (or group of developers working on the same code) can also add their favorite tests to the list by specifying a list of tests or packages to include.
Even though this technique does not prevent every bad commit, we think it is a reasonnable trade-off between the time we have to wait before committing (20-30s) and the risk of breaking the build.