Less IO for your Unit Tests with a Java SecurityManager

Our quest for a truly test-driven engineering team has enabled us to confidently ship our software every few minutes. Automated testing is the keystone of continuous deployment, and as a result our unit tests and smoke tests are thorough. Perhaps, a bit too thorough. (Our last unit testing extravaganza in the form of a cyclomatic-complexity-driven Fix-It day introduced so many new tests that the average time for our test suite increased by about 7%.) At Wealthfront Engineering (we're hiring), we go to great lengths to ensure (a) we test as much of our code as possible while (b) spending precious resources sparingly and (c) increasing awareness about troublesome parts of our engineering culture.

Less IO for your Java Unit Tests with a SecurityManager.

In this article, I will explain our goals and implementation of the open-sourced Apache-licensed Kawala LessIOSecurityManager, a Java SecurityManager implementation that spotlights I/O operations during your tests, while allowing fine-grained control of allowable operations via concise annotations.

Perils of hidden I/O operations.

In our way of executing every possible execution path in our unit tests, we may inadvertently call methods that perform expensive I/O operations, either on the local disk, or across the network. A surprisingly expensive, yet omnipresent, example of a surprise-expensive operation that fails the obviousness test is the implementation of java.net.URL.equals(Object): comparing two instances of URL requires resolving any hostnames. Hostname resolution is a multi-millisecond operation that reaches across the network. Imagine performing equality-related operations to a large List with thousands of URL instances: each new hostname (thanks to Java's automatic DNS caching) would require a new domain name resolution requests. Many unit tests may trigger expensive and time-consuming operations to external resources with harmful side-effects: unit tests that slow down your iterations, or even unit tests that mutate their environment, infecting future tests with their toxic byproducts.

Design Goals of LessIOSecurityManager & Assurances Offered.

Our goal with LessIOSecurityManager is to spotlight such I/O operations to the developer and provide assurances that a conscious decision will have to be made by the developer before any I/O takes place during unit tests. In the example of the URL class, a @AllowDNSResolution on the JUnit class containing the DNS-hungry unit test, would suffice to mend the CantDoItException that be thrown otherwise.

Fine-Grained Annotation-Based Configuration.

Our fine-grained annotations, @AllowDNSResolution, @AllowExternalProcess, @AllowLocalFileAccess, @AllowNetworkAccess, and @AllowNetworkMulticast, allow the developer to cast a perfectly-customizable I/O-awareness net with limited support for wild-cards and on-the-fly matching. The embedded JavaDocs accurately describe the possibilities for each parametrizable annotation, and our meta-tests, our tests for our testing infrastructure, serve as living documentation.

Implementation.

Various methods in the core Java libraries that interact with the underlying system outside the JVM are hard-wired to check with the SecurityManager, if one is installed, before performing potentially hazardous operations. The SecurityManager contains a variety of check[...] methods that correspond to a variety of permission requests. The API is long and cumbersome, and the SecurityManager operates at such a low-level that erroneous states may be caused easily, rendering the JVM unable to load new classes. In our SecurityManager we use the following:

• All calls to interesting check[...] methods are routed to methods that correspond to an annotation (for example, checkConnect(String host, int port) → checkNetworkEndpoint(String host, int port, String description)).
• The per-annotation method fetches the current execution stack in terms of classes,
• and checks for any white-listed classes (such as the built-in ClassLoader). (Note that this design allows you to easily subclass LessIOSecurityManager and provide your own list of white-listed classes.)
• Assuming no white-listed class is involved, we identify the [...]Test class that contains your JUnit @Test-marked methods.
• We make a call to checkClassContextPermissions(...) to which we pass the current execution stack and a Predicate that checks for the correct annotations. We use the toString() method to provide a description of what each Predicate is looking for, enabling meaningful permission-specific feedback to the developer via logging.
• Extensive and precise logging, with configurable verbosity levels, guides you, should you perform any disallowed operations on the precise nature of the operation and exact annotation that would allow such an operation.

Installing the LessIOSecurityManager.

Installing the LessIOSecurityManager is as easy as setting the java.security.manager system property. You can do that either by passing -Djava.security.manager=com.kaching.platform.testing.LessIOSecurityManager as a command-line argument to your java invocation. If you're using Ant, you may declare the java.security.manager system property in the element of your build.xml file. You must set the fork property to ensure a new JVM, with LessIOSecurityManager as the SecurityManager is utilized. (Take a look at the LessIOSecurityManager JavaDocs for an Ant instrumentation example.) Setting up the LessIOSecurityManager with IDEs or Maven is trivial.

Our Experience.

During the instrumentation of our multi-thousand unit tests, we discovered a number of suspicious I/O operations and adjusted unit tests. The performance gains were insignificant compared to the extensive awareness around the side-effects of many commonly used operations that the LessIOSecurityManager brought to our team. We truly believe that the LessIOSecurityManager can help your engineering organization ensure that its tests perform no sneaky I/O operations and never mutate their environment. Feel free to leave comments here with questions, suggestions for improvements, or any bugs you may encounter.