Converting dynamic SVG to PNG with node.js, d3 and Imagemagick

Visitors to Wealthfront might notice we're using SVG to render the risk meter in our questionnaire, as well as your performance projection and other charts. We build our visualizations using d3.js, a fantastic library that provides just the right amount of abstraction on top of SVG to allow us to develop robust visualizations quickly. SVG is attractive as a vector format, and animations are simple.

However, as most are aware, SVG is not supported in some older browsers - notably IE 8 and below. One option to provide backwards compatibility is to rasterize SVG to a static image. This saves you from creating a pixel-perfect alternate implementation. While you lose animations, you're still able to convey the important information.

SVG to PNG
As a proof of concept I was interested in any method of SVG to PNG conversion. As you might expect, Imagemagick's convert proved to be a good option. Unfortunately Imagemagick's default SVG support is rather poor. You can solve this by installing librsvg before Imagemagick. Imagemagick will defer to librsvg when doing SVG conversion from that point forward (run convert -list configure | grep DELEGATES and check librsvg is in the list to confirm Imagemagick is using the library). With librsvg support, convert produced PNGs of our SVG source that perfectly matched Chrome's rendering.

We'll get to why we're using node.js in a moment. In the meantime, if we want to use convert from within node.js we can spawn a child process using convert's stdin/stdout argument syntax to pipe data in and out. This saves us having to write to disk and deal with temporary files.

The example below sends the SVG source for a red square to convert's stdin and writes the PNG to the node process's stdout.



But our SVG is dynamic...
Using convert is all well and good when you already have the SVG source, but our visualizations are dynamic - built in the browser with d3.js. This is the more interesting problem, since it means we need a DOM and the ability to execute d3.js on the server-side. For this, jsdom - a DOM implementation in Javascript - is an excellent solution. We can execute our d3 visualization code and it will affect the DOM accordingly. Ultimately, this allows us to extract the generated SVG on-the-fly and pipe it to convert.

Here's an example of jsdom using our "insertPie" function to build the SVG for a pie chart. We extract the SVG with innerHTML.



Altogether now
Now that we can generate SVG, pipe SVG to convert, and get a hold of the PNG output we can chain it all together. Below is a simple example that starts up an HTTP server in node.js and serves our visualization as a PNG image. You can see that it generates a pie chart with values from the query string and pipes the SVG generated in jsdom to convert. Once we receive output from convert we write it to the response stream. The end result is a small HTTP server that will generate rasterized versions of our SVG pie chart.



A step further
At Wealthfront we're nearly all Java on the backend. Should we choose this approach, it would be nice to leverage Java bindings with librsvg for rasterization and a Java-based DOM implementation with JS support to handle generation of the visualizations with d3. Having fewer moving parts in the process reduces complexity, even if we do have to sacrifice the sexiness of node.

The full source for the pie chart visualization and the HTTP server is available in this gist: https://gist.github.com/1509145. Enjoy!

Our CEO Andy Rachleff talks about VC, entrepreneurship and Wealthfront at Dec 8th 7pm in SF Stanford Alum Club

Check out Andy Rachleff, our awesome CEO, who will speak at the Stanford SF Alum Club tomorrow night. He's going to talk about entrepreneurship from three perspectives, developed through his career as a long-term venture capitalist at Benchmark Capital funding technology start-ups, as a lecturer teaching entrepreneurship at Stanford Business School, and as the CEO of a next-generation online financial advisor company. It's a rare opportunity to hear someone speak with as much breadth of experience in the industry, and we think you'll like it. This event is open to the public and walk-ins are welcome. Doesn't that sound awesome? Check it out.

Time: Dec 8th Thursday 7pm
Address: University Club 3rd Floor, 800 Powell St., San Francisco

See details at:
http://alumni.stanford.edu/eventSetup/viewAnnounce?event_id=8475&pgAid=0293&cturl=close

Beyond Java's Access Control: the Visibility Test

It is generally a good practice to use the most restrictive access level that makes sense. However, Java's access control is not always as expressive as one would want. In this blog post, I am going to give two examples where an access level must be artificially relaxed. Then, I'll describe Wealthfront's recently open sourced solution to this problem.

First, let's have a look at Guava's Lists class. Some of its factory methods rely on the internal static method computeArrayListCapacity(int), which calculates the capacity of an ArrayList. The perfect access level modifier for this method is private, as the method is not meant to be used by other classes.



Unfortunately, declaring this internal method as private would hinder its testability. Indeed, one would need to test the computeArrayListCapacity(int) method indirectly by invoking one of the factory methods in Lists. The resulting test would be hard to understand (as the tested method is not clearly visible in the test) and fragile (as the test is now tied to the implementation of the invoked method.)

Consequently, the visibility of the computeArrayListCapacity(int) method must be artificially relaxed to package-private. In order to document this decision, the method is annotated with Guava's @VisibleForTesting annotation. Hopefully, this annotation will discourage external users to rely on the method, which is what the compiler would do for a private method.



Another example of the lack of expressivity in Java's access control occurs when using a dependency injection framework such as Guice. Injected dependencies should never be manually assigned outside tests. For instance, the following class



should never be used like this in production code



On the other hand, this class is meant to be used exactly like this in tests. Hence, the processor and transactionLog fields should behave as private members in production code, but as package-private members in tests. As with the @VisibleForTesting annotation, the @Inject annotation acts as a cue for external users not to rely on the annotated fields in production code.

In both examples, annotations are used as a mean to extend Java's access control (among other things, obviously.) Unfortunately, the compiler can't help us control access to the annotated members. That being said, we can write a test that does exactly that, and that's what we did. We call it the Visibility Test, and we open sourced it.



As you can see in this example, the Visibility Test is configured by specifying the intent of an annotation. Here, we want everything in bin/ annotated with @VisibleForTesting and @Inject to behave as private. The VisibilityTestRunner uses ASM to perform the following two passes on our class files:

1. Find all classes, fields, constructors and methods annotated with the specified annotations.
   
2. Find all instructions referring to these classes, fields, constructors and methods.
   

During the second pass, the runner records illegal accesses and reports them.



You can start using this test today by installing the kawala-testing sub-project available on GitHub.