Our quest for a truly test-driven engineering team has enabled us to confidently ship our software every few minutes
. Automated testing is the keystone of continuous deployment, and as a result our unit tests and smoke tests are thorough
. Perhaps, a bit too
thorough. (Our last unit testing extravaganza in the form of a cyclomatic-complexity-driven Fix-It day
introduced so many new tests that the average time for our test suite increased by about 7%.) At Wealthfront Engineering (we’re hiring
), we go to great lengths to ensure (a) we test as much of our code as possible while (b) spending precious resources sparingly and (c) increasing awareness about troublesome parts of our engineering culture.
Less IO for your Java Unit Tests with a SecurityManager.
Perils of hidden I/O operations.
In our way of executing every possible execution path in our unit tests, we may inadvertently call methods that perform expensive I/O operations, either on the local disk, or across the network. A surprisingly expensive, yet omnipresent, example of a surprise-expensive operation that fails the obviousness test is the implementation of java.net.URL.equals(Object)
: comparing two instances of URL
requires resolving any hostnames. Hostname resolution is a multi-millisecond operation that reaches across the network. Imagine performing equality-related operations to a large List
with thousands of URL
instances: each new hostname (thanks to Java’s automatic DNS caching
) would require a new domain name resolution requests. Many unit tests may trigger expensive and time-consuming operations to external resources with harmful side-effects: unit tests that slow down your iterations, or even unit tests that mutate their environment, infecting future tests with their toxic byproducts.
Design Goals of LessIOSecurityManager & Assurances Offered.
Our goal with LessIOSecurityManager
is to spotlight such I/O operations to the developer and provide assurances that a conscious decision will have to be made by the developer before any I/O takes place during unit tests. In the example of the URL
class, a @AllowDNSResolution
on the JUnit class containing the DNS-hungry unit test
, would suffice to mend the CantDoItException
that be thrown otherwise.
Fine-Grained Annotation-Based Configuration.
Various methods in the core Java libraries that interact with the underlying system outside the JVM are hard-wired to check with the SecurityManager, if one is installed, before performing potentially hazardous operations. The SecurityManager contains a variety of check[…] methods that correspond to a variety of permission requests. The API is long and cumbersome, and the SecurityManager operates at such a low-level that erroneous states may be caused easily, rendering the JVM unable to load new classes. In our SecurityManager we use the following:
- All calls to interesting check[…] methods are routed to methods that correspond to an annotation (for example, checkConnect(String host, int port) → checkNetworkEndpoint(String host, int port, String description)).
- The per-annotation method fetches the current execution stack in terms of classes,
- and checks for any white-listed classes (such as the built-in ClassLoader). (Note that this design allows you to easily subclass LessIOSecurityManager and provide your own list of white-listed classes.)
- Assuming no white-listed class is involved, we identify the […]Test class that contains your JUnit @Test-marked methods.
- We make a call to checkClassContextPermissions(…) to which we pass the current execution stack and a Predicate that checks for the correct annotations. We use the toString() method to provide a description of what each Predicate is looking for, enabling meaningful permission-specific feedback to the developer via logging.
- Extensive and precise logging, with configurable verbosity levels, guides you, should you perform any disallowed operations on the precise nature of the operation and exact annotation that would allow such an operation.
Installing the LessIOSecurityManager.
Installing the LessIOSecurityManager
is as easy as setting the java.security.manager
system property. You can do that either by passing -Djava.security.manager=com.kaching.platform.testing.LessIOSecurityManager
as a command-line argument to your java invocation. If you’re using Ant, you may declare the java.security.manager
system property in the element of your build.xml
file. You must
set the fork
property to ensure a new JVM, with LessIOSecurityManager
as the SecurityManager
is utilized. (Take a look at the LessIOSecurityManager JavaDocs for an Ant instrumentation example
.) Setting up the LessIOSecurityManager
with IDEs or Maven is trivial.
During the instrumentation of our multi-thousand unit tests, we discovered a number of suspicious I/O operations and adjusted unit tests. The performance gains were insignificant compared to the extensive awareness around the side-effects of many commonly used operations that the LessIOSecurityManager brought to our team. We truly believe that the LessIOSecurityManager can help your engineering organization ensure that its tests perform no sneaky I/O operations and never mutate their environment. Feel free to leave comments here with questions, suggestions for improvements, or any bugs you may encounter.