Our quest for a truly test-driven engineering team has enabled us to confidently ship our software every few minutes. Automated testing is the keystone of continuous deployment, and as a result our unit tests and smoke tests are thorough. Perhaps, a bit too thorough. (Our last unit testing extravaganza in the form of a cyclomatic-complexity-driven Fix-It day introduced so many new tests that the average time for our test suite increased by about 7%.) At Wealthfront Engineering (we're hiring), we go to great lengths to ensure (a) we test as much of our code as possible while (b) spending precious resources sparingly and (c) increasing awareness about troublesome parts of our engineering culture.
Less IO for your Java Unit Tests with a SecurityManager.
In this article, I will explain our goals and implementation of the open-sourced Apache-licensed Kawala LessIOSecurityManager, a Java SecurityManager implementation that spotlights I/O operations during your tests, while allowing fine-grained control of allowable operations via concise annotations.
Perils of hidden I/O operations.
In our way of executing every possible execution path in our unit tests, we may inadvertently call methods that perform expensive I/O operations, either on the local disk, or across the network. A surprisingly expensive, yet omnipresent, example of a surprise-expensive operation that fails the obviousness test is the implementation of java.net.URL.equals(Object): comparing two instances of URL requires resolving any hostnames. Hostname resolution is a multi-millisecond operation that reaches across the network. Imagine performing equality-related operations to a large List
with thousands of URL instances: each new hostname (thanks to Java's automatic DNS caching) would require a new domain name resolution requests. Many unit tests may trigger expensive and time-consuming operations to external resources with harmful side-effects: unit tests that slow down your iterations, or even unit tests that mutate their environment, infecting future tests with their toxic byproducts.
Design Goals of LessIOSecurityManager & Assurances Offered.
Fine-Grained Annotation-Based Configuration.
- All calls to interesting check[...] methods are routed to methods that correspond to an annotation (for example, checkConnect(String host, int port) → checkNetworkEndpoint(String host, int port, String description)).
- The per-annotation method fetches the current execution stack in terms of classes,
- and checks for any white-listed classes (such as the built-in ClassLoader). (Note that this design allows you to easily subclass LessIOSecurityManager and provide your own list of white-listed classes.)
- Assuming no white-listed class is involved, we identify the [...]Test class that contains your JUnit @Test-marked methods.
- We make a call to checkClassContextPermissions(...) to which we pass the current execution stack and a Predicate that checks for the correct annotations. We use the toString() method to provide a description of what each Predicate is looking for, enabling meaningful permission-specific feedback to the developer via logging.
- Extensive and precise logging, with configurable verbosity levels, guides you, should you perform any disallowed operations on the precise nature of the operation and exact annotation that would allow such an operation.