Link to

Fork me on GitHub

Monday, February 14, 2011

Write Internal Tools: Business in the Front, Party in the Back

If you find yourself fortunate to work for a company that values your time as a commodity too precious for repetitive and boring tasks (see #6 of wealthfront's values), then your team will hopefully dedicate significant time not only on the great features you offer your customers, but also on developing the tools that help you deliver them in the first place. Personally, I think Yishan Wong's post on "Tools Are Top Priority" is a great commentary on the topic (and I recommend the rest of the series as well).

Over and over, the successive waves of new startups with structurally more advanced operations (think Google with the massive compute power of its datacenters, or startups which can build an entire website with just a small team) are enabled due to an advance in tools. Computerized tools, unlike physical tools, can stack their leverage arbitrarily upon each other, compounding their leverage to enormously high levels.

Hence, your operating efficiency, and thus the number of people you need to hire, and therefore your costs, are directly impacted by the ingenuity of your internal tools. This means that your tools teams should not be a back-office, after-thought function staffed with second-string players. Your most talented engineers should be working on your tools, and your culture must reflect this priority. Writing great tools and continuing to improve and replace them is more important than the next shiny feature.

If you find that your company buys expensive enterprise software instead of putting your A-team engineers on making awesome internal tools, then they don't understand what the word 'leverage' actually means, and you my friend, likely have a serious and systemic problem.

Give your engineers time to build
tools, and they'll be making sweet
music in no time!

Alright, so enough philosophy and propaganda, you want to make a difference today. The tip? Add bash complete to things you use constantly. Then, add some reflection to your services' queries so you can auto-complete those too. If your queries have LongMethodNameJavaitis, you'll be writing me a thank-you email.

So, bash complete seems a little cryptic at first, but it's really simple. For your command, you just register another callback command that gets notified when you hit tab while typing arguments. The callback command should print out a list of potential matches, and bash complete will do the rest. For our example, we'll write a python script (auto_complete_me) that auto-completes itself (just in time for valentine's day).

First, tell bash about the auto-complete option. You'll want to eventually add this to your .bash_profile or the system /etc/profile:
complete -C 'auto_complete_me --bash-complete' auto_complete_me

Then our auto_complete_me script looks something like this (don't forget put the script somewhere in your path with execute privilege):
import sys, optparse

completions = {
  'auto_complete_me' : ['nerds', 'coolguys'],
  'nerds' : ['richard', 'linus', 'bill', 'andrew'],
  'coolguys' : ['snake', 'steve', 'elvis']

def complete(the_command, complete_flag, complete_command, current_word, previous_word):
  if previous_word in completions:
    print '\n'.join(filter(lambda c:c.startswith(current_word), completions[previous_word]))

parser = optparse.OptionParser()
parser.add_option("--bash-complete", action="store_true", 
                                  help="print options for bash complete")
(options, args) = parser.parse_args()

if options.bash_complete:

# run the rest of yer command!

First thing to notice is the complete function. We've set it up that this command gets called with the bash complete. In our case, we've set it up that the_command and complete_command will likely be very similar (other than the path), and the complete_flag should always be our "--bash-complete" flag. The most interesting parts to look at are current_word and previous_word. We use the previous_word to determine the list of possible completion items, and then use the current_word to filter those items down to only the relevant ones. In this case, we just look up our completion items via a key-value lookup, but don't be afraid to go for style points.

At wealthfront, we use the kawala query engine which is conveniently accessible on the command line by doing a POST with curl command. We use it so frequently, that we replaced it with a fairly similar script, but with a more convenient syntax for specifying parameters. Services also provide a "Help" query that lists all the operations that the service provides. In our script, we take the output of "Help" and use the results to bash-complete the query for you. Doing the math with even conservative estimates, we've saved hours in typing (hours! and likely days!)

Imagine typing "users getAwesomeUsersNamedJohn" all bash-completed nicely, and piping the json result into jsawk or doing some other sort command line magic. Now, we're talking about productivity.

The moral of the story, spend time on tools that address your pain points as an organization, everyone benefits, and your products will show it.

For more information regarding bash complete and a tutorial in bash (duh) check out this article:

And make sure to read Yishan Wong on Engineering Management: