Writing tests is a fundamental part of writing good software. They help us to catch bugs, be confident in our changes and serve as documentation. At Wealthfront, our Android team relies heavily on our automated test suite to maintain productivity and code quality — until we hit a snag. With our codebase maturing and our test suite growing, the time taken to exhaustively test our code grew, hindering our ability to quickly merge our changes. This challenge prompted a pivotal question: ‘Do we need to run all these tests?’. Outright deletion wasn’t an option but what if we could intelligently execute only necessary tests?
In this blog post, we delve into our Android team’s journey of creating a custom Gradle plugin designed to filter tests based on changed files, a solution that slashed our test times in half. In this first blog post, we will focus on the process of filtering unit tests, offering insights into our methodology and the transformative impact it had on our workflow. Our goal is to ensure that our testing process scales proportionally with the size of our pull requests as opposed to the size of our codebase.
Test Filtering Strategy
When to Filter tests
We’ve integrated test filtering on PRs targeting our master branch. Given that our CI pipeline is already set up to run our test suite before merging, it was seamless to add test filtering to our workflow. By applying test filtering at this stage, we run relevant tests based on the pull request’s changes while being efficient with resources and time. We continue to run the entire test suite on master before an app release ensuring comprehensive test coverage before new features reach our clients.
How to filter tests
Test filtering itself is a capability provided by Gradle. Gradle enables filtering by file names, package names, and test names. For instance, in a multi-module app like ours with a module named “cash”, filtering all “cash” tests can be achieved with a command like
gradle test --tests 'com.wf.cash.*'
By leveraging the wildcard character “*”, we can seamlessly run all tests within the “cash” module. Explore Gradle’s documentation to learn more.
Project Setup
Our app follows a modular architecture, dividing functionality into feature modules. This also allows tests to be logically grouped which helps facilitate test filtering. For instance, if a pull request only affects the “cash” module while leaving the “investing” module untouched, we can focus solely on running tests related to the “cash” functionality. Code within a module can have tight dependencies on each other and casting a small enough net that only covers the affected dependencies proved to be difficult. We decided that to thoroughly test any changes to a module, we need to run all its unit tests. We ignored testing inter-module dependencies for now since they are covered by our end-to-end tests.
Relying on Gradle
We considered various approaches for implementing our test filtering logic and ultimately decided to develop a Gradle Plugin to take advantage of Gradle’s built-in test filtering functionality and the fact that our project is already split into Gradle modules. Moreover, since our CI pipeline already utilizes Gradle commands for test execution, leveraging a Gradle Plugin seamlessly integrates with our existing workflow. By incorporating a toggle flag into our Gradle test command, we gain the flexibility to enable or disable filtering as needed.
Methodology
At a high level, these are the steps that we need to take to run only the relevant tests on our PRs.
- Use a git diff to find out which files were changed
- Parse the list of changed files to determine which modules they belong to
- Create a test filter to run all tests for the affected modules
Gradle Plugin Creation
Our goal is to extend Gradle’s existing functionality for running tests. Our final command will look something like this so let’s break it down.
./gradlew testReleaseUnitTest -Pfilter-unit-tests=true
testReleaseUnitTest
The command starts with “test” meaning that we are running tests.
testReleaseUnitTest
Next is the build variant that we are running the tests on.
testReleaseUnitTest
Then we have the type of test that we want to run. We only have unit tests in our modules so that’s what we will use.
-Pfilter-unit-tests=true
Finally, the part that we are going to add. A property called “filter-unit-tests” which when turned on, will apply our custom Gradle Plugin to filter tests. Learn more about running tests from command line.
Implementation
First, we are going to update our app level build.gradle
file to respect this new property. It will selectively apply our new Gradle plugin.
Gradle plugin code must be placed in a buildSrc
folder in your project’s root directory. Create a buildSrc
directory if it doesn’t exist and update its build.gradle
file to declare the FilterUnitTests
plugin. The plugin will be using git so we also have a dependency on jgit
which is a Java implementation of git.
Next, we will create a plugin class FilterUnitTestsPlugin
which will call a helper class FilterUnitTests
to do the filtering for us and then apply those filters on any Gradle tasks that are of the type AndroidUnitTest
. We also set isFailOnNoMatchingTests = false
in case our filter returns with no tests.
Next, we create our FilterUnitTests
class. This class will be responsible for transforming the list of changed modules into a filter that can be understood by Gradle. modulesChanged
will be a list containing the file paths of the changed modules. We need to format the file paths into Java packages. The last segment in the file path will be the module’s name and we already know that the top-level package for each module is in the format “com.wf.moduleName”. By appending the wildcard to the end we can run all the tests for the specified module.
Next, we have to implement the FilesChangedComponent
class which will perform a git diff to figure out which files changed. We want to diff the changes between the head of our current branch and the merge-base between the current branch and master. We can then filter the types of files we care about. In our case, we are only keeping Java and Kotlin files because any changes in functionality will be in those files. Finally, we return the file paths from the diff.
Now that we know which files changed, we can infer the modules that were changed because the file paths will always start with the module’s name. Treating the module’s name as a prefix to the file path, we can convert the list of changed files into a list of the modules they belong to. We also only want to include modules which are leaf nodes. This is because if we have multiple levels of nesting in our modules, choosing a parent module and its child module will run a test twice. Since all leaf nodes should have a build.gradle
file we can differentiate the modules we care about by filtering on if this file exists or not.
At this point, we have implemented everything necessary to run a subset of our tests based on changed files. You can now run ./gradlew testReleaseUnitTest -Pfilter-unit-tests=true
on your branch to test it out.
How does this affect our test times
Now that we’ve implemented unit test filtering in our PR workflow, let’s revisit our initial challenge: lengthy CI test times. The premise is simple. Running fewer tests should speed up our CI right? Analyzing the average CI unit test times in the weeks leading up to and following the rollout of test filtering reveals a significant improvement, with times dropping from approximately 18 minutes to 11 minutes. It’s important to note that there’s a fixed cost associated with running tests, primarily the time taken for app compilation, which averages about 5 minutes. Focusing solely on the variable cost — the time taken to execute tests — we’ve achieved a reduction of over 50%!
Conclusion
With the implementation of our Gradle plugin for automated unit test filtering on PRs, we’ve effectively halved the time spent running unit tests prior to merging into master. Since unit tests are small in scope and usually easy to write, we have an abundance of them which necessitates filtering. Our success in optimizing our workflow led to extending our approach to our end-to-end tests as well. Stay tuned for our next blog post where we delve deeper into the unique challenges we faced when filtering our end-to-end tests, how we overcame them, and of course the time savings that came along with it.
Disclosures
The information contained in this communication is provided for general informational purposes only, and should not be construed as investment or tax advice. Nothing in this communication should be construed as a solicitation or offer, or recommendation, to buy or sell any security. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront Advisers or its affiliates endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.
Cash Account is offered by Wealthfront Brokerage LLC (“Wealthfront Brokerage”), a Member of FINRA/SIPC. Neither Wealthfront Brokerage nor any of its affiliates are a bank, and Cash Account is not a checking or savings account. We convey funds to partner banks who accept and maintain deposits, provide the interest rate, and provide FDIC insurance. Investment management and advisory services–which are not FDIC insured–are provided by Wealthfront Advisers LLC (“Wealthfront Advisers”), an SEC-registered investment adviser, and financial planning tools are provided by Wealthfront Software LLC (“Wealthfront”).
All investing involves risk, including the possible loss of money you invest, and past performance does not guarantee future performance. Please see our Full Disclosure for important details.
Wealthfront Advisers, Wealthfront Brokerage and Wealthfront are wholly owned subsidiaries of Wealthfront Corporation.
Copyright 2024 Wealthfront Corporation. All rights reserved.