Onboarding Client Cash and Assets
Here at Wealthfront we attempt to make onboarding onto our platform as easy as possible. We provide a personalized asset allocation to any prospective client free-of-charge, but things only really get interesting for our brokerage platform during the next step: funding an account to make our recommendations a reality. Accounts are funded through two primary channels: ACH bank transfers, which run on the same system used for payroll direct deposits and other direct-to-bank-account payment applications, and ACATS account transfers, which are the industry standard method for moving client assets between different brokerage firms [1]. Inbound cash and assets are the lifeblood of any growing financial advisor; managing the lifecycles of such transfers for our clients and providing visibility into their status are some of the most delicate and crucial responsibilities required to establish trust when a client is first getting started with Wealthfront.
When I first joined Wealthfront last year, the brokerage platform team was in the midst of a massive effort to rebuild the foundation of our brokerage platform, in order to bring more of our critical operations in-house and provide a solid footing for future automation enhancements. One major early part of this effort involved building a new banking platform to manage client bank account links and ACH transfers. From my initial projects, I had exposure to the broken abstractions and the high latency from client deposit intent to investment that largely defined the bank transfer pipeline on our old platform, and knew that we could do better taking a fresh approach. To accomplish this goal we partnered with an outside banking partner to facilitate interacting with the Automated Clearing House (ACH) system run by NACHA [2], which provided a low-level interface upon which we could build the new higher-level systems that would actually orchestrate money movement functions on behalf of our clients. During this integration, I learned several important lessons that would serve me well when it came time to interface with the more complicated ACATS system.
The first design pattern we applied was the use of persistent Java entities that functioned as state machines, where transitions between different steps in the lifecycle of that entity are governed by external input from our banking partner, and are idempotent to repeated processing. For instance, the following diagram represents a simplified version of the ACH transfer state machine diagram in our system:
Figure 1: Simplified ACH Transfer State Machine
This design pattern has several desirable properties. The state machine structure makes it easy to reason about the lifecycle of a transfer and rely on invariants about that transfer’s properties based on its current state, such as whether the transfer is currently cancelable. Idempotency implies that reprocessing the same external input will not trigger duplicate external events in our system, such as re-notifying the client that their transfer has completed. Finally, state transition timing information is also persisted, which makes it easy to reconstruct how and when an entity moved between different states.
A second design pattern involves the use of different abstraction layers to achieve the design goal of separation of concerns, reducing the overall complexity of a system and facilitating the use of modular pieces that can be swapped in and out with relative ease. On the banking platform, this primarily involved separating the client intent, a Deposit, from the actual mechanism by which that deposit takes place, an ACH transfer. Each entity has its own unique state machine and only stores the information relevant to that entity, and state transitions at the lower-level entity largely drive transitions and actions at the higher abstraction level. Code is reused between deposits with different underlying transfer mechanisms, and external systems are only required to concern themselves with properties exposed by the higher-level entity.
ACATS System Overview
Our new banking platform was one of the first parts of our new brokerage platform to go live in production as part of the launch of our 529 college savings account product in June 2016. With the experience of helping to build this system behind me, I was ready to apply the accrued lessons to tackle a new and more daunting challenge: completing our integration with the ACATS system. ACATS (Automated Customer Account Transfer Service) is a batch processing system run by the NSCC (the National Securities Clearing Corporation) and is used by nearly all major brokerages to facilitate the movement of client accounts between different firms. The main goal of the system is to quickly and reliably fulfill a client’s intent to transfer their brokerage account, by standardizing the required information to initiate a transfer and ensure both parties agree on the timing and quantity of assets being transferred. A full account ACATS transfer typically takes 4-5 business days from initiation to settlement; a transfer begins when the receiving firm submits a transfer initiation (TI) record to the ACATS system, containing information about the client’s accounts at the receiving and delivering firms. This TI record is distributed by ACATS to both the delivering and the receiving brokerage firms; the delivering party then must respond to the request by either rejecting the request [3] (if they deem it to be invalid), or submitting asset transfer (AT) records to confirm the transfer, with each AT record specifying an asset and quantity to transfer. The transfer then settles 2-3 days after it has been confirmed by the delivering firm, at which point the client’s assets have been successfully transferred. Partial account ACATS transfers follow a similar lifecycle, with the primary difference being that the receiving firm also must submit AT records along with the initial TI record in order to request only specific assets from the client’s account at the contra firm. The following diagram illustrates a simplified version of this process for a full transfer:
Figure 2: Full ACATS Transfer Record Exchange and Settlement
Interfacing with this system poses several challenges. First and foremost is the sheer complexity of the system itself: there are eight different types of ACATS transfers, each with different rules and lifecycles from the full and partial transfers outlined above; four different record types, each with dozens of fields with specific rules about when they should or should not be populated; five different transaction types for the different records, to enable advanced functionality like the modification of asset information for in-progress transfers, or the repair of records submitted with errors. Additionally, the ACATS system operates with very specific rules around the timing of transfers; during each day, there are five distinct batch input and output “cycles,” and during each cycle only certain flavors of records can be submitted, depending on the type and current state of the transfers in question. Failing to meet the ACATS SLAs can result in the offending brokerage firm being reported to FINRA, the regulatory body that oversees brokerage firms, as members are obligated to honor client account transfer requests in a timely manner. The ACATS User Guide, containing all the “basic” information about these transfers, is a 150 page PDF wall of text and diagrams, guaranteed [5] to cure insomnia in all but the most passionate of brokerage engineers. Thankfully, applying some of the design patterns outlined above makes this integration a more tractable problem and allowed us to break the implementation down into more digestible chunks.
ACATS State Machines and Abstraction Layers
Given the amount of information required to initiate and fulfill an ACATS transfer, and the requirements of our other backend systems that are concerned with the creation and status of these transfers, breaking down the system into multiple abstraction layers proved essential. Our lowest abstraction level tracks exactly the information distributed by ACATS, and the entities at this layer are updated based on the content of the multiple intraday output files and the end-of-day status file we receive from ACATS. The transfer entity at this level keeps track of information like the ACATS Control Number (a unique identifier assigned to each transfer by the system), the settlement date of the transfer assigned by ACATS, the exact rejection code from the contra firm (if relevant), and all the output records that we have associated with that transfer.
Updates to transfer status at the lowest level then propagate upwards to our next abstraction level, which tracks Wealthfront-specific information required to initiate a transfer and submit the required TI and AT records to ACATS, such as information about the Wealthfront client account receiving the transfer and the contra-party account we are attempting to transfer from. This abstraction level tracks all of the input transactions we submit to ACATS for a given transfer, as well as the various notifications that the transfer generates for our brokerage operations team to examine and act on through our in-house back office web application.
Finally, we have a high-level entity that tracks the original client intent to transfer an external account into Wealthfront. This entity serves as the interface for external systems querying about the status of transfers, and hides the fact that there are actually several different (albeit less common) mechanisms of account transfer that we use besides ACATS. Additionally, our initial attempt to perform an ACATS transfer is sometimes rejected by the contra firm and must be retried; this entity links multiple ACATS submissions back to the original client intent, to make it easier to resubmit the transfer and track its progress through these multiple submissions. The existence of this layer also makes it easy to make changes to our underlying systems without impacting external consumers of information about account transfers.
Another crucial pattern applied to our ACATS integration involved the use of state machines with idempotent transitions. The ACATS system already uses the state machine pattern to track the lifecycle of transfers; there are 17 different ACATS “statuses” for a transfer outlined in the ACATS user guide, with well-defined transition criteria between them, triggered by either the passing of time or an action initiated by one of the participating parties. We replicate this logic on our system and are thus able to easily detect and alert on any deviation from the expected lifecycle of a transfer, or determine when a transfer has the potential to transition into a status that we want to avoid.
At our middle abstraction level, we are able to simplify these 17 states down to just six primary states, relying on the fact that certain sets of ACATS states all possess the same key properties from the perspective of our transfer processing system. Although occasionally the underlying specific ACATS status becomes relevant, we can largely use this higher-level state abstraction to determine important facts like whether the transfer can be canceled, or whether the transfer has already been confirmed by the contra-party. These six key states, transitions, and their meanings are outlined below, as well as the associated detailed NSCC ACATS statuses that map to them:
Figure 3: ACATS State Machine and Associated NSCC ACATS Statuses
Finally, at the highest level abstraction, we are able to use only four key, intuitive states to describe the state of the transfer to other system systems within Wealthfront and to the client: REVIEWING, IN PROGRESS, COMPLETED, and FAILED. No knowledge of the underlying transfer mechanism (ACATS) is required to understand the meaning or implication of these states, which is exactly our goal with the abstraction.
Monitoring and Alerting
Given the tight SLAs and the adverse consequences of failure to process ACATS transfers in a timely manner, adequate monitoring and alerting are critical to alerting our on-call engineers to investigate and possibly intervene before a minor issue becomes a major headache. To alleviate these risks, we employ what is sometimes called the “belt-and-suspenders” approach [6], by both alerting on immediate failure cases detected during transfer processing as well as monitoring on desired end-state invariants of our system after transfer processing is completed.
During each ACATS output cycle, we parse the ACATS output file into a number of multicycle records, each of which represents one record that was submitted to ACATS by us or a contra-party during this cycle; if we fail to parse a record from the file for any reason, we raise an alert for an on-call engineer to deal with (the “belt” in this case). In addition, we have separate monitoring to ensure the multicycle output file for the current cycle was marked as processed by our system before the expected cutoff, which is only possible if all the records are parsed successfully (the “suspenders”). At the next step in transfer processing, we group all records for a given ACATS transfer together, and use this record group to update the state of the transfer entities in our system; here again, we alert on any immediate failures to process these record groups (belt), and additionally monitor that all records have been marked as processed (suspenders). Finally, after the state of the transfers has been updated using these records, we process the updated transfers to generate any required notifications for our operations team to review and enqueue any input transactions that we want to submit to ACATS (e.g. a rejection request for an account that doesn’t exist on our books), alerting for immediate transfer processing failures as well as monitoring that the expected notifications and input records have been created for all transfers in a given state.
Automation and Tradeoffs
Before the transition to our new brokerage platform, account transfers were largely a black box: we submitted paperwork by email to initiate a transfer, and then at some point later (if the transfer completed), we received records of assets arriving in a client account. The intermediate statuses and the exact causes of transfer failures were largely hidden from us, often leading to poor client experiences and manual inquiry processes. Now that we are interfacing more directly with the ACATS system, we have the complete set of information required to identify why we are seeing transfer failures and fix them, to inform clients about the exact state of their transfer, and to make the appropriate investments to automate processes that still require manual intervention by our operations team. As one example, in some situations we rely on an external vendor to supply the contra-party account number we use to initiate a transfer; occasionally, these externally-provided account numbers are truncated or otherwise incorrect for certain institutions. This led our operations team to informally keep track of such institutions and remediate when they saw transfer requests from those firms. With the full set of data about why transfers from these institutions are failing, we can automatically detect and highlight potentially problematic transfer requests for operations to investigate, while expediting the approval of transfers that we are confident will succeed. Such ongoing iterative and data-driven improvement will allow us to scale our systems as we see higher transfer volumes without just resorting to increasing the size of our operations team.
Although not without challenges, working on our ACATS integration solidified my understanding of some of the key software engineering paradigms outlined above. Since going live, the system has processed thousands of client transfer requests and transferred hundreds of millions of dollars worth of our clients’ assets, and the initial foundation is now being improved upon by other teams at Wealthfront that are working to improve our client experience and end-to-end automation. If these kind of challenges sound interesting to you, consider working at Wealthfront!
Notes and References
[1] We also support wire deposits and check rollovers, but these are largely managed by our clients or by their existing bank or brokerage firm and pushed onto our system.
[2] https://www.nacha.org/ach-network is a good starting spot for the curious
[3] Transfers can only be rejected using one of the 15 rejection codes provided by ACATS, such as “Invalid Account Number” or “Account in Distribution or Transfer”
[4] Not mapped to any note above, just a test to see if you are paying attention and got this far.
[5] See disclosures, I don’t think I am allowed to actually guarantee anything in this blog post.
[6] The originator of this phrase is unclear, but I imagine their clothes did not fit very well.
Disclosure
Nothing in this communication should be construed as an offer, recommendation, or solicitation to buy or sell any security. Wealthfront’s financial advisory and planning services, provided to investors who become clients pursuant to a written agreement, are designed to aid our clients in preparing for their financial futures and allow them to personalize their assumptions for their portfolios. Additionally, Wealthfront and its affiliates do not provide tax advice and investors are encouraged to consult with their personal tax advisors.
All investing involves risk, including the possible loss of money you invest, and past performance does not guarantee future performance. Wealthfront and its affiliates rely on information from various sources believed to be reliable, including clients and third parties, but cannot guarantee the accuracy and completeness of that information.