Last year, Wealthfront changed the way we do business in a huge way. In order to better serve our clients by taking on more brokerage responsibilities ourselves, we transitioned from being a fully-disclosed brokerage to an omnibus brokerage. In addition to paving the way for new functionality such as our Portfolio Line of Credit, our clients have enjoyed more subtle improvements such as: deposits invested more quickly, account transfers processed more quickly and more visibly, and simpler monthly statements. For us, this means Wealthfront Brokerage is now wholly responsible for maintaining the official record of our clients’ accounts.
One consequence of this is: every day, our system of record needs to process tens of thousands of transactions. Our infrastructure can comfortably handle transactions at this scale, but we occasionally need to process them on a larger order of magnitude. This has the potential to delay downstream processes. For example, since we store (even small amounts of) client cash within a money market fund which pays a monthly dividend, hundreds of thousands, or even millions, of client dividend transactions must be entered into the system at the end of each month.
Finding the Bottleneck
During one of our recent money market dividend processing instances, I decided to see for myself if there were any obvious changes we could make to speed up the job which seemed to run much slower than one would expect. It was eventually going to become a problem. By adding some simple logging, I could easily see which steps in the process were taking the longest.
When I looked at the logs, I was surprised by what I learned. I had expected the system would be spending more time polling the transactions from the queue, or inserting the transactions into the system-of-record (something database bound), but instead I saw that it was spending a large amount of time just converting the queued transaction payload into a form that could be inserted directly into the system-of-record. Wow, I expected this to be a very efficient operation! Populating the fields in one Java object with the fields from another Java object should take no time at all compared to the time it takes to retrieve the thousands of transactions from the database, or inserting thousands of rows into the system of record.
To get more detail, I decided to use a simple but useful tool called jstack. Jstack can be used to find the call-stack of every currently-running thread in a JVM process. I inspected the stack of the process while it was entering transactions and discovered something surprising. It was spending the bulk of its time retrieving the date for each transaction. To set the proper transaction date, we must look at the system’s current processing date, which is stored in the database.
I recognize that this may seem strange. Why not just use the current date? Well, we store it in the database because we run a process to explicitly advance the date, which marks trades as settled, pays out dividends, etc. Anticipating inefficiency, we have caching in place to avoid code hitting the database every time it is needed, but the code did not appear to be using the cached value – ever!
Understanding the problem
Our backend services use Google’s Guice library for dependency injection. Guice allows us to separate the creation of objects from their use, which lets us focus on implementing the business logic rather than wiring together a bunch of dependencies every time we write a new class. To retrieve the system date, we use a Guice provider, so we can just inject a LocalDate (actually in this case a Provider<LocalDate>) with a custom @SystemDate annotation.
The custom provider uses an ExpiringMemoizingSupplier from Google’s Guava library to avoid hitting the database more than once per minute, returning the most recently retrieved value if it’s not expired.
The supplier caches the system date for 1 minute, and automatically retrieves the date from the database any time it’s needed past its expiration. If the supplier is caching the date, it shouldn’t be hitting the database every time, right? With TransactionConverter (the class injecting the Provider) being a singleton, one would assume that Guice would be instantiating a single SystemDateProvider and assigning it to the SystemDateProvider#systemDateProvider injected field.
Surprisingly, it turns out that Guice creates a new instance of the SystemDateProvider every time the get() method is called! The Provider that is injected is actually an anonymous class that keeps track of the context and source of the injection. This allows Guice to support circular dependencies by creating proxy objects, as well as including additional information in the exception that is thrown when Guice encounters a runtime error. As a result of this behaviour, we were adding an extra database call for every transaction we entered, drastically slowing down the whole process.
The fix for this was simple: Adding a @Singleton annotation to the provider would enforce reuse of the same SystemDateProvider object, allowing the result to be cached, and our code to do the conversion in-memory without hitting the database.
This 10-character change drastically sped up our processing rate. Before this change, our transaction entry process would top out at about 5,000 transactions per minute. Now that we’re properly caching the processing date, we can enter nearly 20,000 transactions per minute, buying us a lot of time before we have to continue optimizing this process. With our increased throughput, we were able to set a new record recently: we entered over 1.2 million transactions into our system of record in a single day with no issues or manual intervention. By measuring the performance of the code, I was able to find the bottleneck, understand the issue, and implement a simple fix that more than tripled the throughput of one of our most critical processes. And, I learned something for next time!
Disclosures:
This blog is powered by Wealthfront Corporation (formerly known as Wealthfront Inc.). The information contained in this blog is provided for general informational purposes, and should not be construed as investment advice. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront Corporation (“Wealthfront”) endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.
Wealthfront may from time to time publish content in this blog and/or on this site that has been created by affiliated or unaffiliated contributors. These contributors may include Wealthfront employees, other financial advisors, third-party authors who are paid a fee by Wealthfront, or other parties. Unless otherwise noted, the content of such posts does not necessarily represent the actual views or opinions of Wealthfront or any of its officers, directors, or employees. The opinions expressed by guest bloggers and/or blog interviewees are strictly their own and do not necessarily represent those of Wealthfront or its affiliates.
Brokerage products and services are offered by Wealthfront Brokerage LLC (formerly known as Wealthfront Brokerage Corporation), member FINRA / SIPC, and a wholly-owned subsidiary of Wealthfront Corporation. Please see our Full Disclosure for important details. Nothing in this communication should be construed as an offer, recommendation, or solicitation to buy or sell any security. Wealthfront and its affiliates do not provide tax advice and investors are encouraged to consult with their personal tax advisors.
All investing involves risk, including the possible loss of money you invest, and past performance does not guarantee future performance. Margin lending can add to these risks, and investors should carefully review those risks as part of their overall financial strategy. PLOC eligibility is subject to a minimum account balance which is subject to change. Diversification does not guarantee a profit or protection against loss.
© 2018 Wealthfront Corporation. All rights reserved.