At Wealthfront, testing is core to the culture of our engineering organization and business. In fact, on the iOS team, testing is woven into our development process. We host and manage our own continuous integration (CI) pipeline on Jenkins which runs our unit and UI test suites. As most iOS developers know, UI tests take an order of magnitude longer to run than unit tests since they simulate a user interacting with the app. Recently, our UI test suite reached an untenable size, with 90 tests taking over an hour and a half to execute. As we scale our codebase and team (FYI we’re hiring), the need to minimize this execution time becomes paramount. To mitigate the cost of these tests, we decided to parallelize our UI test job across our CI build machines with the help of Jenkins Pipelines.
Our Jenkins architecture
Our CI environment contained 16 Mac Minis serving as Jenkins agents. Each merge to the git main branch kicked off three jobs: UI tests with mock data, UI tests hitting an isolated backend instance, and a master build, which itself parallelized the execution of our three unit test suites (Release, Beta, and Dev). This meant if no other jobs were running, such as a side branch build or an App Store deployment, only five of our 16 agents would be in use. These unused agents presented an opportunity to distribute the workload of our tests.
The plan to parallelize
Parallelizing UI tests involved partitioning our UI test suite into a given number of divisions, running each division on an agent, and collecting the results once every division had finished. This seemed simple in theory but came with two big challenges during implementation: how do we construct the n-divisions of tests and how can we actually run each division in parallel?
Dividing our tests
Partitioning an array of tests is easy enough, but constructing the array in the first place posed a challenge. We knew our array should contain all test classes in our UI test scheme, since the testing tool we use in CI, Fastlane’s scan, can consume a list containing a subset of test cases to be run, of the form Test Bundle[/Test Suite[/Test Case]], …
¹. This can be accomplished using the only_testing
parameter, and in our case an example list might look like: [WealthfrontUITests/CollegeGoalUITest, WealthfrontUITests/AccountTransferInitiationUITest, …]
².
A snippet of our Fastfile
But how might we get the full list of test cases to partition? As it stands, neither xcodebuild
nor Fastlane provides a way of querying the tests in a given test bundle, so there’s no help there. Facebook’s xctool
does enable this through the -listTestsOnly
argument to run-tests
, however incorporating another third party library and rolling it out to all of our agents seemed unnecessary for such a small task³. Since all of our UI test files were located in the same folder we decided it would be simpler to parse and collect the class name from each one at test time. This way, new test files would be implicitly included in the job, so long as each file was in the proper directory. Below is the function we developed to construct the full array:
With the list of all test cases, we could then partition the list into n buckets of equal size (excluding the remainder if n did not divide the number of test cases), and move on to the execution of the tests.
Running in parallel
Our UI test job was configured through the Jenkins classic UI and consisted mainly of a Build shell script which executed our Fastlane Lane. We were limited in that our job could not take advantage of Jenkins’ parallelization capabilities. The only way to accomplish this was to move to a Jenkins Declarative Pipeline, in which the building, testing, and deployment of our code all resided in a single Jenkinsfile
. In addition to supporting parallel execution of stages, Declarative Pipelines have the added benefit of defining a job in Groovy code. Our Jenkins job was defined through the web interface which, though technically versioned, cannot be reviewed in the same manner as our source code. With Declarative pipelines, the process of building and testing our code lives side-by-side with our source code.
The first iteration of our Jenkinsfile
consisted of two stages, the first of which constructed a partition of test cases using the logic described previously, while the second farmed out the divisions of the partition to run on three agents and reported the results. The following is an abbreviated version of our Jenkinsfile
.
Note: As you can see in the runTestsInParallel
method, we dynamically construct the branches, each of which corresponds to a parallel execution of tests. Alternatively, we could have defined a fixed number of stages in place of the ‘Parallel Execution’ stage, but instead we now have the flexibility to update the target number of parallel agents with a single line change of our environment variable.
Measuring the results
With our UI test job fully defined, we could now see the impact of our improvements. The old job regularly ran longer than 90 minutes, while the updated job ran for only 40 minutes on average⁴. Using only two more of our available agents, we gained more than a 50% improvement.
But is this the best we can do? You may have noticed that our partitioning logic simply slices our list of test cases, ordered by the output of find
(which makes no ordering guarantee), into equal divisions weighted by number of test cases. Our divisions happened to be similar in execution time, but it’s plausible that in the future, one arbitrary division might contain test cases with many long-running test methods. This would lead to a branch that would run much longer than the rest, extending the overall job and defeating the purpose of parallelization. Thus, if our goal was to minimize total execution time, we should aim to minimize the range of execution time between our branches.
One last improvement
Enter the Parallel Test Executor Plugin. This helpful Jenkins plugin analyzes the previous run of the job and partitions the job’s tests weighted by execution time, meaning we can delegate the work of balancing our tests. The only work on our end then becomes tacking on any newly added tests (as they weren’t part of the previous run) and formatting the plugin’s output for consumption by Fastlane. Below is the abbreviated, second iteration of our Jenkinsfile
.
We actually observed little difference in total execution time with this new plugin, but we can be confident moving forward that our test job will automatically adjust and adapt as our test base scales.
Learnings & Future Directions
In summary, we found how to leverage CI infrastructure to lower the cost of expensive jobs and realized the benefits of describing jobs in code. In the future, we will likely add one more level of parallelization by running UI tests concurrently on a single agent, a process Apple made quite simple in Xcode 10.
If this work interests you or if you have thoughts on how we might make it even better, check out our careers page!
Resources
- Faster Pipelines with the Parallel Test Executor Plugin
- Pipeline Syntax
- Pipeline as Code with Jenkins
- Technical Note TN2339: Building from the Command Line with Xcode FAQ
Notes
- A note on testing nomenclature: Apple uses the
Test Target > Test Class > Test Method
naming convention to refer to the structure of tests in Xcode, while Fastlane usesTest Bundle > Test Suite > Test Case
to refer to the same hierarchy. Hereafter, when we mention our UI test suite, we mean the collection of all UI test methods in our UI test bundle. - We could have partitioned our tests at the granularity of test method, however it would introduce the overhead of calling the class methods
XCTestCase.setup()
andXCTestCase.tearDown()
once for each agent the test case was distributed on, instead of exactly once. xctool
is also losing support as an CI build tool.- Since there is a fixed overhead of building the application for testing on each agent that runs a subset of tests, we cannot expect the execution time to be cut by two-thirds to 30 minutes.
Disclosure
This communication has been prepared solely for informational purposes only. Nothing in this communication should be construed as an offer, recommendation, or solicitation to buy or sell any security or a financial product. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront or its affiliates endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.
Wealthfront offers a free software-based financial advice engine that delivers automated financial planning tools to help users achieve better outcomes. Investment management and advisory services are provided by Wealthfront Advisers LLC, an SEC registered investment adviser, and brokerage related products are provided by Wealthfront Brokerage LLC, a member of FINRA / SIPC .
Wealthfront, Wealthfront Advisers and Wealthfront Brokerage are wholly owned subsidiaries of Wealthfront Corporation.
© 2021 Wealthfront Corporation. All rights reserved.