Since the very beginning, kaChing has been trunk stable. In other words, everybody develops on the trunk and the software is stable at every point in time. Our trunk is being continuously built and tested and we cannot deploy a revision to production unless the revision has been successfully built and all the tests are green.

One side effect of being trunk stable is that broken code committed to the trunk is preventing subsequent revisions to be deployed to production. While our team was growing, we started to become unaware of broken builds. Consequently, we were not able to deploy a new revision to production as quickly as we wanted to as we first had to fix whatever was breaking the build.

In order to prevent the build to be broken for too long, we decided to add a pre-commit hook to SVN that was rejecting commits if the build was currently broken unless the commit log contained the tag #buildfix. This hook changed our behavior instantaneously, and we were back to a point where the build was green for the vast majority of the time.

When we migrated our SVN repository to a different machine a few months later, we never took the time to update the pre-commit hook. However, we added a light in the office that flashes when the build is broken. Thanks to this flashing light, we were all aware when the build was broken and were able to quickly react accordingly.

Unfortunately, with close to 15 engineers in the team, the build light is not visible to everyone anymore. Despite the fact that Hudson sends emails when the build breaks, we were back to a point were the build was preventing us to deploy to production 5 minutes after a commit.

As an experiment, I fixed and re-introduced the pre-commit hook we use to have. I am glad to report we are back to a point where we can happily deploy to production about 50 times a day. Initially, one might think that such hook is extremely annoying. However, we do have one-click reverts on one of our dashboards. Consequently, if whoever breaks the build cannot fix it in a few minutes he can easily revert his commit.

If anyone is interested, below is our pre-commit hook. The file to edit is named pre-commit and is in the hooks/ directory of your SVN server.

REPOS="$1"
TXN="$2"

buildfix=$(echo $message | grep -i "#buildfix" | wc -l)
if [[ $buildfix -gt 0 ]]; then
  exit 0
fi

BUILD_PASS=$(
  curl -s -u USER:PASSWORD
    http://HOST/hudson/job/BUILD/lastCompletedBuild/api/json |
  grep -v "FAILURE" | wc -l)
  
if [[ $BUILD_PASS -ne 1 ]]; then
  echo "The build is broken. Add the tag #buildfix to the commit log." 1>&2
  exit 1
fi

exit 0