When to Filter tests

We’ve integrated test filtering on PRs targeting our master branch. Given that our CI pipeline is already set up to run our test suite before merging, it was seamless to add test filtering to our workflow. By applying test filtering at this stage, we run relevant tests based on the pull request’s changes while being efficient with resources and time. We continue to run the entire test suite on master before an app release ensuring comprehensive test coverage before new features reach our clients.

How to filter tests

Test filtering itself is a capability provided by Gradle. Gradle enables filtering by file names, package names, and test names. For instance, in a multi-module app like ours with a module named “cash”, filtering all “cash” tests can be achieved with a command like

gradle test --tests 'com.wf.cash.*'

By leveraging the wildcard character “*”, we can seamlessly run all tests within the “cash” module. Explore Gradle’s documentation to learn more.

Project Setup

Our app follows a modular architecture, dividing functionality into feature modules. This also allows tests to be logically grouped which helps facilitate test filtering. For instance, if a pull request only affects the “cash” module while leaving the “investing” module untouched, we can focus solely on running tests related to the “cash” functionality. Code within a module can have tight dependencies on each other and casting a small enough net that only covers the affected dependencies proved to be difficult. We decided that to thoroughly test any changes to a module, we need to run all its unit tests. We ignored testing inter-module dependencies for now since they are covered by our end-to-end tests.

Relying on Gradle

We considered various approaches for implementing our test filtering logic and ultimately decided to develop a Gradle Plugin to take advantage of Gradle’s built-in test filtering functionality and the fact that our project is already split into Gradle modules. Moreover, since our CI pipeline already utilizes Gradle commands for test execution, leveraging a Gradle Plugin seamlessly integrates with our existing workflow. By incorporating a toggle flag into our Gradle test command, we gain the flexibility to enable or disable filtering as needed.

At a high level, these are the steps that we need to take to run only the relevant tests on our PRs.

1. Use a git diff to find out which files were changed
2. Parse the list of changed files to determine which modules they belong to
3. Create a test filter to run all tests for the affected modules

Gradle Plugin Creation

Our goal is to extend Gradle’s existing functionality for running tests. Our final command will look something like this so let’s break it down.

./gradlew testReleaseUnitTest -Pfilter-unit-tests=true

testReleaseUnitTest

The command starts with “test” meaning that we are running tests.

testReleaseUnitTest

Next is the build variant that we are running the tests on.

testReleaseUnitTest

Then we have the type of test that we want to run. We only have unit tests in our modules so that’s what we will use.

-Pfilter-unit-tests=true

Finally, the part that we are going to add. A property called “filter-unit-tests” which when turned on, will apply our custom Gradle Plugin to filter tests. Learn more about running tests from command line.

Implementation

First, we are going to update our app level build.gradle file to respect this new property. It will selectively apply our new Gradle plugin.

Gradle plugin code must be placed in a buildSrc folder in your project’s root directory. Create a buildSrc directory if it doesn’t exist and update its build.gradle file to declare the FilterUnitTests plugin. The plugin will be using git so we also have a dependency on jgit which is a Java implementation of git.

Next, we will create a plugin class FilterUnitTestsPlugin which will call a helper class FilterUnitTests to do the filtering for us and then apply those filters on any Gradle tasks that are of the type AndroidUnitTest. We also set isFailOnNoMatchingTests = false in case our filter returns with no tests. 

Next, we create our FilterUnitTests class. This class will be responsible for transforming the list of changed modules into a filter that can be understood by Gradle. modulesChanged will be a list containing the file paths of the changed modules. We need to format the file paths into Java packages. The last segment in the file path will be the module’s name and we already know that the top-level package for each module is in the format “com.wf.moduleName”. By appending the wildcard to the end we can run all the tests for the specified module.

Next, we have to implement the FilesChangedComponent class which will perform a git diff to figure out which files changed. We want to diff the changes between the head of our current branch and the merge-base between the current branch and master. We can then filter the types of files we care about. In our case, we are only keeping Java and Kotlin files because any changes in functionality will be in those files. Finally, we return the file paths from the diff.

Now that we know which files changed, we can infer the modules that were changed because the file paths will always start with the module’s name. Treating the module’s name as a prefix to the file path, we can convert the list of changed files into a list of the modules they belong to. We also only want to include modules which are leaf nodes. This is because if we have multiple levels of nesting in our modules, choosing a parent module and its child module will run a test twice. Since all leaf nodes should have a build.gradle file we can differentiate the modules we care about by filtering on if this file exists or not.

At this point, we have implemented everything necessary to run a subset of our tests based on changed files. You can now run ./gradlew testReleaseUnitTest -Pfilter-unit-tests=true on your branch to test it out.