Link to wealthfront.com

Fork me on GitHub

Wednesday, June 12, 2013

ActiveJsonEntity: Bridging the gap between Rails and Java


We have a service-based architecture here at Wealthfront. Instead of our Rails layer talking to a database, it makes RPC calls over HTTP to a collection of services that send and receive JSON. Without a database we can't use ActiveRecord, and all the niceties it provides.

In their first incarnation, our models were plain Ruby objects that took in a JSON hash. We defined methods by hand.

class User
    def initialize(json)
        @json = json
    end

    def first_name
        @json["firstName"]
    end

    def last_name
        @json["lastName"]
    end

    def email
        @json["lastName"]
    end

    def joined
        Time.parse @json["joined"]
    end

    def too_legit_to_quit?
        !!@json["tooLegitToQuit"]
    end
end

We could have generated methods on the fly using the JSON response as our schema, but what if a certain key wasn't present in our JSON? It would throw a NoMethodError, but it might be an entirely valid (optional) field! So the JSON itself was not enough.

This hand-written approach was hardly optimal, nor did it feel like the "Rails way." Most importantly, it didn't measure up to the high bar we set for ourselves on the Wealthfront Engineering team.

The beauty of ActiveRecord models is that you don't have to re-define every attribute of your model. ActiveRecord generates its models by inspecting the schema of your tables. From this it can glean information like column names and data types (boolean, string, dates). But at Wealthfront, our models aren't defined in SQL, they're defined in Java. How can we get the same benefits?

Java is the Schema, Reflection is our friend

The User class you see above would be the Ruby representation of this Java class at Wealthfront:

@Entity
public class User {
    @Value String firstName;
    @Value String lastName;
    @Value String email;
    @Value DateTime joined;
    @Value Boolean tooLegitToQuit;
    //etc
}

Those "@annotations" you see are how we tell our JSON serializer about our class. @Entity says "this class is an entity that can be serialized to JSON". @Value says "this field should be included when you're serializing the object".

This isn't just a Java class definition, it represents our model's schema the same way a SQL definition does with ActiveRecord. To get the same benefit ActiveRecord provides in our system, we need to examine the class the same way ActiveRecord examines an SQL table. Enter Java reflection.

Java's reflection API is incredibly powerful. You can examine every detail of a class, including things like type information, field definitions, and annotations. With it, we have all the information we need to examine our "schema", and then some. Because we use JRuby, we have access to this reflection API in the Rails layer itself!

Enter ActiveJsonEntity

Today Wealthfront employs ActiveJsonEntity. The handwritten Ruby class we saw at the beginning is reduced to this:

class User
    include ActiveJsonEntity
    with_java_entity com.wealthfront.user.User
end

Those 4 lines define a class with all the methods you saw above. Now let's look at how we did it.

Step 1, getting the field names

The first step is to get the field names in our Java class, this is analogous to ActiveRecord using a "columns" query to get the fields that should be present in your ActiveRecord model.

All ActiveJsonEntity asks you to do by default is provide with_java_entity an @Entity annotated Java class. From there we can use reflection to grab the fields. "The declaredFields" method of a Java class will give us an array of all the fields:

irb(main):001:0> ourUserClass = com.wealthfront.user.User.java_class
=> class com.wealthfront.user.User

irb(main):002:0> ourUserClass.declared_fields   
=> [
  java.lang.String com.wealthfront.user.User.firstName,
  java.lang.String com.wealthfront.user.User.lastName,
  java.lang.String com.wealthfront.user.User.email,
  org.joda.time.DateTime  com.wealthfront.user.User.joined,
  java.lang.Boolean com.wealthfront.user.User.tooLegitToQuit
]

Well that was easy, wasn't it?

Next we need to know which fields are annotated with @Value, because those are the ones our Ruby models will need to represent. Let's take a look at the annotation on our first field, firstName.

irb(main):003:0> firstNameField = ourUserClass.declared_fields[0]
=> java.lang.String com.wealthfront.user.User.firstName

irb(main):004:0> anno = firstNameField.annotation(com.twolattes.json.Value.java_class)
=> #<Java::Default::$Proxy35:0x75a351ca>

If our annotation was nil, we'd know this field wasn't annotated with @Value and we wouldn't include it in our list of fields.

With our list of fields we can generate methods for our class that match each @Value field. Our @Value annotation can takes parameters, like optional=true, so we can check those too, allowing us to provide basic required field validation for free.

We've taken care of the basics, but we can do more. Because Java is a strongly typed language, there's a treasure trove of information about our model we can take advantage of.

Step 2, Leveraging type information

There are two things our original, handwritten Ruby class did that we're not doing yet. The first is our #joined method — because our DateTime gets serialized to a string, we were calling Time.parse to turn it into a Ruby Date.

Second, our boolean field tooLegitToQuit is represented idiomatically as a ? suffixed predicate method, "#too_legit_to_quit?".

The good news is we can generate any kind of method we want with Ruby's define_method, but to accomplish our two tasks automagically we need to know what sort of data we're dealing with.

As I mentioned before, we can do this with reflection too. Let's examine our DateTime and Boolean fields and see what we can glean.

irb(main):005:0> dtField = ourUserClass.declared_fields[3]
=> org.joda.time.DateTime com.wealthfront.user.User.joined

irb(main):006:0> dtField.type
=> class org.joda.time.DateTime

irb(main):007:0> dtField.type == org.joda.time.DateTime.java_class
=> true

irb(main):008:0> boolField = ourUserClass.declared_fields[4]
=> java.lang.Boolean com.wealthfront.user.User.tooLegitToQuit

irb(main):009:0> boolField.type
=> class java.lang.Boolean

irb(main):010:0> boolField.type == java.lang.Boolean.java_class
=> true

Just like that we have a test to check if our field is a boolean or DateTime. That's all we need to generate our special methods that parse time and have the ? suffix for boolean fields!

ActiveJsonEntity does a whole lot more

What we've seen so far is just the tip of the iceberg. Our Entities in Java can be nested, placed in collections, or organized in maps (Hashes). Furthermore, Java methods can be annotated with @Value too and values can be optional or have name overrides in the annotations. Finally, our JSON entities in Java support Abstract classes, and we have facilities to deal with the serialization of classes that extend them.

We handle all these things in ActiveJsonEntity, and the end result is pretty cool. Stay tuned for another article in which we dive deeper into ActiveJsonEntity, detecting collections, maps, nested entities, and building support for the auto-discovery of other ActiveJsonEntities.