Extend JUnit Part 1: JUnit 4

JUnit is the most used testing framework in Java world. You will likely encounter a lot of companions like Assertj but the backbone is generally JUnit. The only real concurrent is TestNG and except for some specific usages like Test Compatibility Kit of the specifications, it never got a lot of attraction for developers. One of the reasons is probably a better integration in the whole tooling suite for JUnit (IDE are deeply integration with it compared to TestNG integrations which are good but not that advanced in general.

In any case the fact is you will likely have to work with JUnit tests. A JUnit test is often based on this kind of lifecycle:

start class
    execute class before hooks (@BeforeClass)
        execute test before hooks (@Before)
            execute test (@Test)
        execute test after hooks (@After)
    execute class after hooks (@AfterClass)
end class

This high level lifecycle works for the default JUnit runner but is generally respected by most runners (except cucumber-jvm one maybe but this is for good reasons).

The important things to note are:

1. The lifecycle is symmetric (a before hook has a symmetric after hook). This is very important to be able to create/destroy resources.
2. You have hooks to execute code once per class and once per method.
3. All these hooks are inside an "execution" for the class (first and last lines)

This is the part known by most developers and this is where it becomes interesting. You can rephrase the last two statements this way:

1. You can decorate the class execution (respectively, you can decorate the test execution) with custom logic.
2. JUnit orchestrate the test class execution.

This is the two concepts we will dig into with JUnit 4 in this post.

Before going deeper into the implementation, don't forget you need to add this dependency into your project to be able to test this post examples:

<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>4.12</version>
</dependency>

JUnit Runners

The orchestration of the test is done by what JUnit 4 calls a Runner. A Runner is responsible to prepare the execution of the JUnit test class. It is literally the one converting the class to something runnable. The default - and implicit - runner, called JUnit4 or BlockJUnit4ClassRunner,  does the lifecycle we saw previoulsy (with a small difference we will see in next part). Most of custom runners will inherit from this behavior.

However note that you can also implement it in a custom and completely different way. For instance cucumber-jvm runner doesn't use that and just defines its own lifecycle since scenarii and features don't fit well this default lifecycle. Also note that there is an abstract runner you can extend which shares most of the previous lifecycle without going that deep, the ParentRunner defines a hierarchic execution: the class represents the root and has a list of child you can execute. Finally if you really want a custom lifecycle you just extend Runner class.

To set a runner for a particular class you add on your test class the annotation @RunWith and pass the runner to use as parameter.

Here is an example using the default runner (here it is just expliciting the default):

@RunWith(JUnit4.class)
public class SomeServiceTest {
    @Test
    public void run() {
        // ...
    }
}

Implement your first runner

If you want to implement a custom lifecycle you will extend Runner and need to implement two methods:

public class MyFirstRunner extends Runner {
    private final Class<?> clazz;

    public MyFirstRunner(final Class<?> testClass) {
        this.clazz = testClass;
    }

    @Override
    public Description getDescription() {
        return null; // todo
    }

    @Override
    public void run(final RunNotifier notifier) {
        // todo
    }
}

The description is the one for the test and the run the hook where you implement you own lifecycle. Also note you MUST have a public constructor taking the test class as parameter.

There are static utilities to create the description. Since our runner represents a suite we can just call the relative utility:

return Description.createSuiteDescription(clazz);

And the run is what you need for this runner, for instance:

@Override
public void run(final RunNotifier notifier) {
    System.out.println("Running " + clazz);
}

Now if you set it on a class with the @RunWith annotation and execute it, you will see the corresponding output.

This is a good start but not yet very useful. You probably noticed the run method takes a notifier as parameter. This is the one which makes you able to notify JUnit which part of the lifecycle you execute. All is based on description, for each test, and you can notify if you start a test, if the test suceeded/failed, if the test is skipped etc...

Here is how the notifier is often used:

@Override
public void run(final RunNotifier notifier) {
    getTestDescriptions().forEach(description -> {
        notifier.fireTestStarted(description);
        try {
            doExecuteTest(description);
        } catch (final AssumptionViolatedException e) {
            notifier.fireTestAssumptionFailed(new Failure(description, e));
        } catch (final Throwable e) {
            notifier.fireTestFailure(new Failure(description, e));
        } finally {
            notifier.fireTestFinished(description);
        }
    });
}

This snippet assumes you have a getTestDescriptions() to find the tests to execute and a doExecuteTest() to actually run a test from its description. Then all the rest of the code is the lifecycle handling of the test and the communication with the notifier which can be seen as a JUnit bus here.

Note: the actual lifecycle of the default runner is a bit more complicated because it handles @Ignore and MultipleFailureException. You can reuse EachTestNotifier to get an abstraction to simplify this lifecycle handling.

Implement your hierarchic runner

Going that high in the runner hierarchy is a bit rude, often you extend the default runner or just the ParentRunner. The default runner extension is straight forward and you just plug some custom logic in the child methods so we'll dig into the other solution which is more intesting: the parent runner.

The parent runner defines a set of children to execute. Therefore the API enforces you to define how to find your children, how to describe them and how to execute them:

public class MyHierarchicRunner extends ParentRunner<Class<?>> {
    protected MyHierarchicRunner(final Class<?> testClass) throws InitializationError {
        super(testClass);
    }

    @Override
    protected List<Class<?>> getChildren() {
        return null;
    }

    @Override
    protected Description describeChild(Class<?> child) {
        return null;
    }

    @Override
    protected void runChild(Class<?> child, RunNotifier notifier) {

    }
}

If you implement this runner it means you have a custom API. To illustrate it, we will assume we have a test class which is just a container and we actually want to execute all the nested classes which are Runnable. It can look like:

@RunWith(MyHierarchicRunner.class)
public class MyHierarchicTest {
    public static class MyTest1 implements Runnable {
        @Override
        public void run() {
            Assert.assertTrue(true);
        }
    }

    public static class MyTest2 implements Runnable {
        @Override
        public void run() { // fails for the demo
            Assert.assertTrue(false);
        }
    }
}

This is indeed just for the demo but give an idea of what we want to achieve: each static nested class will get instantiated and executed (run()) as a test.

Here is how we can implement that in our runner:

public class MyHierarchicRunner extends ParentRunner<Class<?>> {
    public MyHierarchicRunner(final Class<?> testClass) throws InitializationError {
        super(testClass);
    }

    @Override
    protected List<Class<?>> getChildren() {
        return Stream.of(getTestClass().getJavaClass().getClasses())
                .filter(c -> {
                    final int modifiers = c.getModifiers();
                    return Modifier.isStatic(modifiers) &&
                            Modifier.isPublic(modifiers) &&
                            Runnable.class.isAssignableFrom(c);
                })
                .collect(toList());
    }

    @Override
    protected Description describeChild(final Class<?> child) {
        return Description.createTestDescription(child, child.getSimpleName());
    }

    @Override
    protected void runChild(final Class<?> child, final RunNotifier notifier) {
        final EachTestNotifier eachNotifier = new EachTestNotifier(notifier, describeChild(child));
        eachNotifier.fireTestStarted();
        try {
            Runnable.class.cast(child.getConstructor().newInstance()).run();
        } catch (final AssumptionViolatedException e) {
            eachNotifier.addFailedAssumption(e);
        } catch (final Throwable e) {
            eachNotifier.addFailure(e);
        } finally {
            eachNotifier.fireTestFinished();
        }
    }
}

Our find children method just filters nested classes to find the public static ones implementing Runnable, the description of each test is built based on the class name and the execution reuses the previous notifier integration and the execution is just the call to the run() method of the Runnable. Simple no?

Implementing custom runners is quite straight forward but for simple integrations and common needs it is too deep into the JUnit stack. It also has a very strong limitation: you can have a single runner per class, no way to compose them without recoding a custom runner which can be very tricky. This is why next part will show you how to extend JUnit 4 more easily to be more efficient and get an easier maintenance.

JUnit rules: the natural extension point

Writing a custom runner opens you a lot of doors and capabilities but is also quite rigid and strict, in particular when it comes to the composition of runners.

To make it simpler to extend JUnit, the framework added to the default runner what it calls Rules. If you are familiar with JavaEE - including Spring ;) - programming model, it is really what is commonly called interceptors or aspects. The main difference with these other implementations of this pattern is the lack of context which goes through the rule itself in JUnit.

Before digging into what rules are, it is important to understand the concept of JUnit Statement. A Statement is basically an execution in the generic way. It can be a method, a method with its before/after hooks, a class execution etc...

Seen like that you already see where the rule idea is coming from: the decoration of statements.

JUnit statement is just a class which defines an evaluate() method which does its execution. It is really a Runnable from a design perspective.

What we commonly call a Rule is actually an implementation of TestRule interface:

public interface TestRule {
    Statement apply(Statement base, Description description);
}

As you can see, it takes a Statement and a Description corresponding to the decorated object and it returns a new Statement which replaces the first one. As mentionned earlier: just the plain old decorator pattern.

JUnit provides some built-in rules like:

• ErrorCollector: allows to continues the execution of a test even after the first failure
• ExternalResource: converts the decorator pattern in two hooks: before() and after()
• TemporaryFolder: creates a temporary folder per execution and gives control to creates temporary files
• Timeout: allows to set a time limit for the test execution
• ExpectedException: allows to test the first execption thrown by a test
• DisableOnDebug: allows to disable a decorated rule if you pass debug the JUnit JVM

You can already see that even built-in rules can be designed to be used as decorators.

What is interesting is to understand that the most common pattern when implementing a rule is this one:

public class MyFirstTestRule implements TestRule {
    @Override
    public Statement apply(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                final MyRuleState state = doStart(description);
                try {
                    base.evaluate();
                } finally {
                    state.stop();
                }
            }
        };
    }

    private State start(final Description description) {
        // ...
    }
}

The rule creates a new Statement which delegates the main execution to the original one (base). It also adds some execution before delegating the execution to the original statement and execute some code after. Indeed the ExternalResource abstraction simplifies it but the fact it doesn't handle a state assumes you pass it through the rule instance which prevents to handle it per thread and run concurrently the tests. Can be an issue or not depending your setup. The previous skeleton doesn't have this issue using an internal state to avoid to store the state in the rule instance.

Now you have a rule, to use it you have a choice to do:

• decorate a single test execution
• decorate all the tests execution

In other words: do you bind your rule to the before/after hooks or beforeclass/afterclass ones.

Note: even if at the same level from a design point of view, the rule are nested in terms of decoration if you compare them to the native hooks (@Before/@After and @BeforeClass/@AfterClass).

Once you picked the decorated execution you just have to declare your rule as required by JUnit in your test class:

• class rule: use a public static field decorated with @ClassRule
• test rule: use a public field decorated with @Rule

As an example is worth a thousand words, here is what it can look like:

public class RuleTest {
    @ClassRule
    public static final TestRule GLOBAL = new MyFirstLoggingTestRule("Around all tests");

    @Rule
    public final TestRule perTest = new MyFirstLoggingTestRule("Around each test");

    @Test
    public void test1() {
        System.out.println("Test 1");
    }

    @Test
    public void test2() {
        System.out.println("Test 2");
    }
}

And if you execute it, you get as expected this output:

Start > Around all tests
Start > Around each test
Test 1
Stop > Around each test
Start > Around each test
Test 2
Stop > Around each test
Stop > Around all tests

Tip: if you need a custom API access, you can use the description to read annotations. For instance if you have a @Debug annotation you can put on the corresponding description (class for class rules and method for rules) which will activate or not some logging statement you can read it this way:

private State start(final Description description) {
    final Debug debug = description.getAnnotation(Debug.class);
    if (debug == null) {
        return new State(); // no-op
    }
    System.out.println("Start > " + message);
    return new State() {
        @Override
        public void stop() {
            System.out.println("Stop > " + message);
        }
    };
}

The Description is the abstraction of the decorated object. You can access the class using the getTestClass() method of the description but it can return null since the runner can use something else to create the description and fill the annotations with another way - including a fully configured and not programmative way.

Rules are very useful, you can check out my rule-them-all project to see some examples of them to set system properties in the scope of a test, start a ftp or sftp server, use spring as a rule instead of a runner to be able to compose it with other rules (and start spring context after the sftp server for instance) etc...

And this is the last thing we need to check out to be complete in the context of this post: the composition. Because if you define this test class you can get some surprises:

public class RuleTest {
    @Rule
    public final TestRule perTest1 = new MyFirstLoggingTestRule("Around each test #1");

    @Rule
    public final TestRule perTest2 = new MyFirstLoggingTestRule("Around each test #2");

    @Test
    public void test() {
        System.out.println("Test");
    }
}

Java doesn't guarantee anymore the order of the fields in a class so using the reflection as does JUnit you can't guarantee perTest1 is launched before perTest2. This can be an issue if the first rule starts a server and set the port in a system property to let the second rule use this system property as a placeholder. Rules are often coupled this way, in particular the last rule which starts the context/container.

To solve that, JUnit introduces the RuleChain rule which is a rule defining an order between other rules:

public class RuleTest {
    @Rule
    public final TestRule perTest2 = RuleChain.emptyRuleChain()
            .around(new MyFirstLoggingTestRule("Around each test #1"))
            .around(new MyFirstLoggingTestRule("Around each test #2"));

    @Test
    public void test() {
        System.out.println("Test");
    }
}

This way you can guarantee the order of your rules and that you are fully deterministic and functional.

Last tip: you can need some rule instance. Typically a rule starting/stopping a server will expose a getPort() method. This is fully compatible with this chaining since you can use plain references in the chain and keep the instance in the class:

public class RuleTest {
    private final MyFirstLoggingTestRule first = new MyFirstLoggingTestRule("Around each test #1");
    private final MyFirstLoggingTestRule second = new MyFirstLoggingTestRule("Around each test #2");

    @Rule
    public final TestRule perTest2 = RuleChain.emptyRuleChain()
            .around(first)
            .around(second);

    @Test
    public void test() {
        System.out.println("Test/" + first.getMessage() + "/" + second.getMessage());
    }
}

With this simple refactoring which doesn't change anything to the lifecycle of the test execution, you can now use the rule instances and access their state if any.

Happy testing!