Link to wealthfront.com

Fork me on GitHub

Wednesday, March 2, 2011

Protobufs backward and forward

Protobufs are designed to support forward and backward compatibility, but that doesn't mean you can't get it wrong. One issue we ran into recently involved adding a new value to an enum. The protobuf code "worked" in the sense that it did not blow up, but our code that used the data failed. Since at Wealthfront we deploy services continually, we can't assume that all our services have an up-to-date view of the protobuf message formats. Eyal has a good write-up of the service-level implications in Continuous Deployment for Data Not Just Code.

Here's an example of code that may not behave as you expect it to:
switch (myMessage.getType()) {
case INDIVIDUAL:
  return handleIndividual(myMessage);
case IRA:
  return handleIra(myMessage);
default:
  logUnknownMessageForLater(myMessage);
  return null;
}
Cuneiform, an early serialization standard
Cuneiform, an early serialization format
Now, we could sit here arguing about what the right way to handle the default case is, but the surprising thing here is how it behaves with unknown enum values. From protocol buffer documentation:
  • bool hasFoo(): Returns true if the field is set.
  • int getFoo(): Returns the current value of the field. If the field is not set, returns the default value.
This means that if I did not set the type or if the type is not a value that was defined at the time the reader's code was generated, I will get back the default value, which is the first enum value if no default is set. I say surprising because most Java APIs would use null to indicate this missing value. Protobufs are consistent in the has/get pairs for all non-repeated fields.

Let's go into more detail with a very simple protobuf. Assume we have two enums so that we can show the difference between required and optional fields.
message Before {
  enum Color {
    WHITE = 0;
  }
  enum Direction {
    NORTH = 0;
  }
  required Color c = 1;
  optional Direction d = 2;
}
We have to set the required field, and can leave the optional unset:
Before before = Before.newBuilder().setC(Before.Color.WHITE).build();
out.println("Has c? " + before.hasC() + ", c " + before.getC());
out.println("Has d? " + before.hasD() + ", d " + before.getD());

try {
  Before.newBuilder().setC(Before.Color.WHITE).setD(null).build();
} catch (NullPointerException e) {
  out.println(e.getClass().getSimpleName());
}
Gives us:
Has c? true, c WHITE
Has d? false, d NORTH

NullPointerException
Although we didn't set D, we can still read back the default value. Now, let's create a new protobuf message to represent the future evolution.
message After {
  enum Color {
    WHITE = 0;
    BLACK = 1;
  }
  enum Direction {
    NORTH = 0;
    SOUTH = 1;
  }
  required Color c = 1;
  optional Direction d = 2;
}
One nice thing about protobufs is that its very easy to deserialize it to a different class, so we can easily create an After, and read it back as a Before:
After after = After.newBuilder().setC(After.Color.WHITE).setD(After.Direction.SOUTH).build();
Before afterAsBefore = Before.parseFrom(after.toByteArray());
out.println("Has c? " + afterAsBefore.hasC() + ", c " + afterAsBefore.getC());
out.println("Has d? " + afterAsBefore.hasD() + ", d " + afterAsBefore.getD());
This gives us exactly the same thing as before:
Has c? true, c WHITE
Has d? false, d NORTH
Watch what happens when we set the new enum values though:
After after2 = After.newBuilder().setC(After.Color.BLACK).setD(After.Direction.SOUTH).build();
out.println("Has c? " + after2.hasC() + ", c " + after2.getC());
out.println("Has d? " + after2.hasD() + ", d " + after2.getD());
    
try {
  Before.parseFrom(after2.toByteArray());
} catch (InvalidProtocolBufferException e) {
  out.println(e.getClass().getSimpleName() + ": " + e.getMessage());
}
As an After, of course, the values are there and everything is fine.
Has c? true, c BLACK
Has d? true, d SOUTH
But when we read it back as a Before, the required field blows up. I initialized the value to something, but that something was outside the allowed range.
InvalidProtocolBufferException: Message missing required fields: c
Like almost everyone, when I start with a new technology, I read enough of the documentation to get started, and then only go back to it when I run into a problem. In this case, I was lucky enough to end up going to the documentation for what happens when there's a value outside the range. I could see if I had done things slightly differently, I could have ended up obliviously treating an unknown value as the first enum value.

If you use protobufs extensively enough that everyone can internalize their quirks, one good practice would be to always set a default value of "UNKNOWN" and accept that sometimes that means "explicitly set as UNKNOWN" and sometimes it means "new value that you aren't aware of", which should be the same thing. Otherwise, you can use a pattern like this to ensure you do the right thing:
if (!myMessage.hasType()) {
  return logUnknownMessageForLater(myMessage);
}
switch (myMessage.getType()) {
case INDIVIDUAL:
  return handleIndividual(myMessage);
case IRA:
  return handleIra(myMessage);
default:
  return logUnknownMessageForLater(myMessage);
}
This solves the problems at the protobuf level, but of course you can still forget to handle a value you should be handling as with any enum. In a future post I'll go into how to take this one step further to ensure that all your code gets updated using the visitor pattern.