Marc Philipp

STF Milestone 10: Documentation improvements

Sun, 25 Jan 2026 00:01:00 +0000

JUnit has long been publishing its User Guide and API documentation for every release on junit.org. However, there was no way to easily navigate between versions and no warnings when browsing outdated or preview releases. In this milestone, I finally had the chance to resolve these long-standing issues.

Prior to this milestone, the only way of accessing a specific version was by manually changing the URL. For example, the User Guide for version 5.13.0 is available at https://docs.junit.org/5.13.0/user-guide/. Manually changing 5.13.0 in the URL to 5.14.0 takes one to the corresponding version. However, doing so requires knowledge about which versions are available and is inconvenient and error-prone.

Moreover, users should view the documentation specific to the version they’re using. Looking at documentation that is older or more recent can be confusing: it can point out features that have better replacements or don’t exist yet in a stable version. Therefore, we have decided to adopt Antora which – among other things – adds a version selector to each documentation page. In addition, since it’s aware of all versions being deployed as part of the documentation site, it provides the ability to render a banner on outdated or preview releases. Rather than having a single HTML page for each version, Antora encourages splitting the documentation into separate pages that are focussed on a single topic.

I started working on this milestone after we had released version 6.0.0. I really wanted 6.0.0 to be part of the new documentation site, though. Similarly, since we will continue to support 5.14.x for some time, at least the latest 5.14.x release should also be available on the Antora site. Therefore, I decided to write a script that would migrate our Asciidoctor-based documentation to an Antora component so that I could run it on different Git branches for different releases. This strategy worked out nicely in the end!

Another challenge was adopting Antora’s support for the latest release and prerelease versions. We wanted to ensure that the those versions continued to be available at https://docs.junit.org/current/ and https://docs.junit.org/snapshot/, respectively. Before adopting Antora, we had achieved that by publishing the latest release in two places: /current and /6.0.2, for example. Instead of duplicating pages, Antora relies on redirects. However, GitHub Pages – where the site had been hosted until now – does not support redirects. After trying a few alternatives, we decided to migrate to statichost.eu for hosting instead. It was very simple to set up and connect to GitHub. It being privacy-focused and hosted in the EU was a nice bonus.

Overall, I’m very happy how it turned out! Antora provides a solid foundation to build on. I’m sure the few limitations we encountered (e.g. its search functionality) will be improved over time. But don’t take my word for it and have a look at docs.junit.org for yourself!

STF Milestone 8: Improved parallel test execution

Sun, 18 Jan 2026 00:01:00 +0000

Since its introduction in JUnit 5.3, parallel test execution has become a popular feature of the JUnit Platform. When enabled, it can significantly speed up test execution for many projects. However, there were still some rough edges and long-standing issues that needed to be addressed. Therefore, this milestone focused on improving the parallel test execution capabilities of JUnit.

Vintage engine

JUnit Platform’s parallel execution support works for all test engines extending HierarchicalTestEngine, such as JUnit Jupiter, Spock, and Cucumber. Notably missing from that list is the JUnit Vintage engine, which runs JUnit 3 and JUnit 4 tests. While the Vintage engine is only intended as a temporary migration aid, there are projects that will probably never migrate all their tests to JUnit Jupiter. To help projects that use both the Vintage engine and Jupiter (or another testing framework based on the JUnit Platform), the Vintage engine has been enhanced to support parallel test execution as well, starting with version 5.12.0. As documented in the User Guide, the behavior can be enabled and configured via the following configuration parameters:

junit.vintage.execution.parallel.enabled=true
junit.vintage.execution.parallel.classes=true
junit.vintage.execution.parallel.methods=true
junit.vintage.execution.parallel.pool-size=4

Thanks to contributor Yongjun Hong for implementing this feature!

Resource locks

Resource locks are a declarative mechanism to control which tests may run in parallel. For example, the test methods in the following example class use resource locks to prevent conflicting concurrent access to system properties:

@Execution(CONCURRENT)
class StaticSharedResourcesDemo {

    @Test
    @ResourceLock(value = SYSTEM_PROPERTIES, mode = READ)
    void customPropertyIsNotSetByDefault() {
        assertNull(System.getProperty("my.prop"));
    }

    @Test
    @ResourceLock(value = SYSTEM_PROPERTIES, mode = READ)
    void anotherCustomPropertyIsNotSetByDefault() {
        assertNull(System.getProperty("my.other.prop"));
    }

    @Test
    @ResourceLock(value = SYSTEM_PROPERTIES, mode = READ_WRITE)
    void canSetCustomPropertyToApple() {
        System.setProperty("my.prop", "apple");
        assertEquals("apple", System.getProperty("my.prop"));
    }
}

While customPropertyIsNotSetByDefault() cannot run concurrently with canSetCustomPropertyToApple(), it may run in parallel with anotherCustomPropertyIsNotSetByDefault(). In order to express that, prior to JUnit 5.12, every test method in this test class needs to be annotated with a @ResourceLock annotation. To reduce this boilerplate, a new target attribute has been added to the @ResourceLock annotation. It allows specifying that the lock applies to all test methods in the annotated class. A test method such as canSetCustomPropertyToApple() may define a lock with the same value and a stricter mode, though.

@Execution(CONCURRENT)
@ResourceLock(value = SYSTEM_PROPERTIES, mode = READ, target = CHILDREN)
class StaticSharedResourcesDemo {

    @Test
    void customPropertyIsNotSetByDefault() {
        assertNull(System.getProperty("my.prop"));
    }

    @Test
    void anotherCustomPropertyIsNotSetByDefault() {
        assertNull(System.getProperty("my.other.prop"));
    }

    @Test
    @ResourceLock(value = SYSTEM_PROPERTIES, mode = READ_WRITE)
    void canSetCustomPropertyToApple() {
        System.setProperty("my.prop", "apple");
        assertEquals("apple", System.getProperty("my.prop"));
    }
}

Another improvement was made for test engines such as Cucumber where parent containers have no execution behavior of their own but solely exist to group child tests. Similar to Jupiter’s @Isolated annotation, Cucumber scenarios can be annotated with @isolated to indicate that they should not run in parallel with other tests/scenarios. However, the JUnit Platform always pulled up that lock to the test class/feature level since a test class in Jupiter or Spock can have its own execution behavior (for example, a Jupiter @AfterAll method). This made it impossible to run Cucumber features in parallel if one of them contained an isolated scenario. The Cucumber engine will be able to configure this behavior once the new API is released in JUnit 6.1.

Open heart surgery Reimplementing parallel test execution

The initial implementation of parallel test execution in the JUnit Platform is based on Java’s ForkJoinPool. Given the hierarchical nature of the test tree, this seemed like a natural fit. For example, a test class would perform its setup, then submit its test methods to the pool for execution (fork), wait for them to finish (join), and finally perform its teardown.

However, over time, it became clear that this implementation decision is causing issues in certain scenarios. For example, when the code under test also uses ForkJoinPool for its own parallelism, the pool used to run tests in parallel would suddenly spawn additional threads. This could lead to more tests running in parallel than configured, causing resource exhaustion and out-of-memory errors. Another problem was unwanted work-stealing in the presence of resource locks. We had to implement special handling to prevent tests holding a set of resource locks from stealing work from other tests with conflicting locks.

There have been several suggestions over the years on how to deal with these issues. Most recently, a contributor proposed a solution that used a regular thread pool in addition to the existing ForkJoinPool. This milestone finally gave me a chance to explore this idea. However, I soon realized that using two thread pools would not be a good solution. Not only would it double the number of threads and require context switching between the two pools, it would also add significant complexity to the implementation. Therefore, I set out to reimplement parallel test execution using a regular thread pool with “simple” work-stealing support. As you can imagine, that was quite an undertaking, and difficult to test properly. Luckily, I had help! Both, Leonard Brünings from Spock and Rien Korstanje from Cucumber helped challenge, test, and improve the new implementation. A huge thanks to both of them!

The new implementation will be an opt-in feature in JUnit 6.1. You can already give it a try today by using version 6.1.0-M1 (see User Guide for details). If we receive promising feedback in 6.1, we will switch the default to the new implementation in 6.2. Looking forward to your feedback!

STF Milestone 7: Safe cancellation

Sun, 28 Dec 2025 00:01:00 +0000

Prior to this milestone, the JUnit Platform provided no safe way to cancel test execution early, e.g. after the first test failed. The only option was to forcibly terminate the JVM running the tests. However, that also caused cleanup operations, such as deleting temporary files or stopping Docker containers used in tests, to be skipped. Letting all tests run is often wasteful in terms of resources such as CPU and causes longer feedback cycles for developers. Therefore, this milestone was all about introducing a safe cancellation mechanism to the JUnit Platform.

The Launcher API is the main entry point to the JUnit Platform for IDEs and build tools. The new CancellationToken API allows clients to request cancellation of a running test execution. The Launcher interface already contained two methods for executing tests; one taking a LauncherDiscoveryRequest and another taking a TestPlan. Rather than adding two additional overloads that take a CancellationToken, a new LauncherExecutionRequest class was introduced that encapsulates all parameters required for test execution, including an optional CancellationToken. This design allows for future extensions of the test execution parameters without breaking existing clients. A LauncherExecutionRequest can be created from a LauncherDiscoveryRequest, a TestPlan, or via LauncherDiscoveryRequestBuilder.forExecution().

Using these new concepts, clients can implement “fail fast” behavior as follows (see User Guide for details):

CancellationToken cancellationToken = CancellationToken.create();

TestExecutionListener failFastListener = new TestExecutionListener() {
    @Override
    public void executionFinished(TestIdentifier identifier, TestExecutionResult result) {
        if (result.getStatus() == FAILED) {
            cancellationToken.cancel();
        }
    }
};

LauncherExecutionRequest executionRequest = LauncherDiscoveryRequestBuilder.request()
        .selectors(selectClass(MyTestClass.class))
        .forExecution()
        .cancellationToken(cancellationToken)
        .listeners(failFastListener)
        .build();

try (LauncherSession session = LauncherFactory.openSession()) {
    session.getLauncher().execute(executionRequest);
}

Cancelling tests relies on test engines checking and responding to the CancellationToken. As a stop-gap solution, the Launcher also checks the token and cancels test execution when multiple test engines are present at runtime.

At the time of writing, the following test engines support cancellation:

JUnit Jupiter
JUnit Vintage
JUnit Platform Suite
TestNG
All HierarchicalTestEngine implementations such as Spock and Cucumber

In addition to the changes in JUnit, I submitted a pull request to the Maven Surefire project. It uses the new cancellation mechanism to implement support for Surefire’s skipAfterFailureCount feature. It was merged and released in Maven Surefire 3.5.4. For Gradle, there’s an open issue to implement similar support across all testing frameworks.

To try out the new safe cancellation mechanism, you need to use JUnit 6.0.0 or later. I am looking forward to your feedback!

STF Milestone 6: Improved Kotlin support

Thu, 21 Aug 2025 00:01:00 +0000

Kotlin is a highly popular language for JVM applications (and beyond). Not only is it the default language for Android apps, it has also become increasingly popular for backend applications. JUnit Jupiter has always supported writing tests in Kotlin and provides Kotlin-specific assertions. However, there were some long-standing and highly-voted issues for improving the usability when writing tests and extensions in Kotlin which have now been addressed in this STF milestone.

The improvements made in this milestone fall into the following categories:

Kotlin contracts
JSpecify nullness annotations
Kotlin suspending functions
Kotlin Sequence support

Kotlin contracts

Kotlin is a statically typed language and the Kotlin compiler has powerful static analysis capabilities. One of them and an often cited difference between Java and Kotlin is its null safety.

Given a Person class, a Kotlin variable can be of type Person or Person? where the former does not allow assigning null while the latter does. When using a variable of a nullable type such as Person?, the compiler requires code to deal with the possibility of the value being null. For example, to access the firstName property of Person class on person: Person?, one can’t just use person.firstName. Instead, one usually either writes person!!.firstName (which will throw a NullPointerException in case person is null) or person?.firstName (which will evaluate to null if person is null and person.firstName otherwise).

To make dealing with nullable types easier, Kotlin has a feature called “smart casts”. If the Kotlin compiler can deduce that a value can no longer be null because it has already been checked for nullness, it allows using it directly without the !!. or ?. operators. To support the compiler, Kotlin functions can define contracts that describe their behavior.

As part of this milestone, JUnit’s Kotlin-specific assertion functions have been enhanced to define such contracts. For example, the assertNotNull function now defines the following contract:

fun assertNotNull(actual: Any?) {
    contract {
        returns() implies (actual != null)
    }
    // ...
}

The contract states that if the function returns normally, meaning without throwing an assertion error, that its actual argument has been checked for nullness and that subsequent code can assume it’s not null. The following example demonstrates why this can be useful:

val person: Person? = findPerson("John", "Doe")
assertNotNull(person)
assertEquals("John", person.firstName)
assertEquals("Doe", person.lastName)

In both calls to assertEquals, person is dereferenced without using !!. or ?. since the compiler knows it can no longer be null due to the prior call of assertNotNull.

JSpecify nullness annotations

In addition to contracts for Kotlin code, JUnit 6.0.0-RC1 started using JSpecify to indicate nullness throughout its Java APIs. The Kotlin compiler supports JSpecify’s annotations which makes writing Kotlin code using JUnit’s APIs more safe and idiomatic.

Of course, there are also benefits when writing code in Java. JSpecify is supported in IDEs such as IntelliJ IDEA so you may notice new warnings when programming against JUnit’s APIs. To check nullability during your build, please check out Error Prone and NullAway.

Kotlin suspending functions

Kotlin has first-class language support for concurrency and asynchronous programming via coroutines. One such language feature is the suspend keyword that can be applied to function definitions. Such suspending functions allow writing asynchronous code in regular sequential style but can pause and resume without blocking a thread.

Prior to JUnit 6.0.0-RC1, writing tests for suspending functions with JUnit Jupiter was not straightforward. Since suspending functions can only be called from other suspending functions, it felt natural to simply add the suspend keyword to the @Test method as well. However, that didn’t work because @Test methods with the suspend modifier were silently ignored by JUnit due to the way the Kotlin compiler generates JVM bytecode for them. For example, the @Test suspend fun test() { /*...*/ } function will be compiled to the equivalent of the following Java method:

@Test
public final java.lang.Object test(kotlin.coroutines.Continuation<? super kotlin.Unit> c) {
	/*...*/
}

As discussed in the previous post, @Test methods must have a void return type. Consequently, JUnit 5.13.0 started reporting discovery issues of severity “warning” in such cases. However, wouldn’t it be better if it “just worked”?

Kotlin core libraries provide different ways of executing coroutines, meaning code that calls suspending functions. One of the simplest is runBlocking (docs) from kotlinx.coroutines. It runs the supplied code and blocks the current thread until its completion. Starting with JUnit 6.0.0-RC1, @Test functions written in Kotlin may now carry the suspend modifier. Behind the scenes, JUnit Jupiter will wrap the function body in a call to runBlocking. For example, the following two @Test methods are equivalent.

class CoroutineTests {
    
    @Test
    suspend fun test1() {
        someSuspendingFunction()
    }
    
    @Test
    fun test2(): Unit = runBlocking {
        someSuspendingFunction()
    }
    
    suspend fun someSuspendingFunction(): String {
        // ...
    }
}

If you’d rather use one of the alternatives to runBlocking, such as runTest (docs), you can do so explicitly as follows.

class CustomCoroutineTests {
    @Test
    fun test(): Unit = runTest {
        someSuspendingFunction()
    }
}

Kotlin `Sequence` support

Kotlin’s Sequence (docs) abstraction is similar to Java’s Iterable. The main difference lies in how it’s used in Kotlin’s standard library. Methods operating on Iterable are typically eager whereas those using Sequence are lazy. For example, the Iterable<T>.map { ... } extension function applies the supplied function to all elements of the Iterable and returns a List<R>. The corresponding Sequence<T>.map { ... } extension function returns another Sequence<R> and does not apply the supplied function until a terminal operation like toList() is called. So, from a Java perspective, it’s more similar to Stream.

JUnit supports conversion of a variety of types to Stream for use with dynamic or parameterized tests. Prior to JUnit 5.13.0, this included different types of Stream, Collection, Iterable, Iterator, and all array types. This list has been extended to support all types defining an Iterator iterator() method, including Kotlin’s Sequence. For example, Kotlin sequences can now be used with @MethodSource or @FieldSource.

object SequenceTests {

    @JvmStatic
    val data = sequenceOf(
        arguments(1, Month.JANUARY),
        arguments(3, Month.MARCH),
        arguments(8, Month.AUGUST),
        arguments(5, Month.MAY),
        arguments(12, Month.DECEMBER)
    )

    @ParameterizedTest
    @FieldSource("data")
    fun test(value: Int, month: Month) {
        assertEquals(value, month.value)
    }
}

Wrapping up

Thanks to these improvements writing JUnit Jupiter tests in Kotlin has just gotten more convenient and idiomatic. Please give JUnit 6.0.0-RC1 a try and let us know what you think and provide feedback on GitHub.

STF Milestone 5: Discovery issues

Sat, 16 Aug 2025 00:01:00 +0000

This milestone was all about finally resolving issue #242 by introducing an error handling mechanism for test discovery. As you can tell by the low issue number (we’re currently at #4840) it has been a long time coming. Since introducing the discovery phase in an early version of JUnit 5, we had seen the need for being able to report “discovery issues” such as errors and warnings but also info-level messages to users.

Have you ever written a test and noticed at some later point that it wasn’t actually being executed? I know, it sounds like a real nightmare. But, can you spot what’s wrong with these test methods?

@Test // Java
int test() {
    return 42;
}

@Test // Kotlin
fun test(): Nothing = fail()

They look like regular tests, don’t they? Well, actually, @Test methods must be declared having a void return type (Unit in Kotlin)! During the test discovery phase, the JUnit Jupiter test engine looks for methods that are annotated with @Test and have a void return type. Before version 5.13, if it found a malformed declaration, such as the ones above, it silently discarded them – even if they were clearly intended to be test methods.

To be fair, IDEs such as IntelliJ often provide inspections to help avoid running into such issues. However, such IDE-only warnings are easy to overlook, especially in a large codebase.

Therefore, rather than keeping you up at night, JUnit 5.13 introduced the concept of “discovery issues” and a two-part mechanism for reporting them. First, test engines can report issues such as malformed declarations during test discovery. Second, the JUnit Platform takes care of reporting them to users. It logs all non-critical issues and, if there are any critical issues, test execution will fail for that test engine. The severity that’s considered “critical” is configurable via the junit.platform.discovery.issue.severity.critical configuration parameter. Currently, the default value is ERROR, but that may be changed in a future release.

All test engines provided by JUnit (Jupiter, Vintage, and Suite) already make use of this feature to report a variety of issues. For example, the Jupiter engine will now report a warning for malformed test and lifecycle method declarations such as the ones shown above. For the full list, please consult issue #242 on GitHub.

We’ve introduced this mechanism at a stage when the JUnit Platform and its test engines were already widely used. Therefore, it was probably inevitable that the initial 5.13.0 release reported a few false-positives. For example, in 5.13.0 abstract test methods were overeagerly reported as being problematic, but we’ve since learned about use cases for them in the wild. Those and other checks have since been refined and improved in 5.13.x patch releases.

On the other hand, we’ve already seen a bunch of projects that had actual problems that needed addressing reported to them after they updated. For example, one project had lots of inner classes that contained test methods but those inner classes were not annotated with @Nested so those tests were never executed. That’s exactly the kind of problem we wanted to address by introducing this mechanism!

If you haven’t yet updated to JUnit 5.13 or later, please do and look out for any reported discovery issues. To surface all discovery issues in your project, we recommend setting the junit.platform.discovery.issue.severity.critical configuration parameter to INFO and then running all your tests.

STF Milestone 4: Parameterized test classes

Sat, 07 Jun 2025 00:01:00 +0000

In version 5.0, JUnit Jupiter introduced support for parameterizing test methods. In JUnit 4, only test classes could be parameterized. Therefore, being able to do so on the method level provided more flexibility. However, there are cases where a set of tests should be executed against the same sets of arguments. The recent 5.13 release introduced support for @ParameterizedClass thereby finally resolving this highly-voted feature request.

To write a parameterized test method, the @ParameterizedTest annotation is used instead of @Test. In addition, at least one @...Source annotation is required to specify the source of argument sets that the method will be invoked with.

class SomeTests {
    @ParameterizedTest
    @ValueSource(strings = {"foo", "bar"})
    void shouldNotBeNull(String value) {
        assertNotNull(value);
    }
    @ParameterizedTest
    @ValueSource(strings = {"foo", "bar"})
    void lengthShouldBeThree() {
        assertEquals(3, value.length());
    }
}

In the above example, both test methods are parameterized with an identical @ValueSource annotation. Each will be invoked twice, once with "foo" and once with "bar" as its String parameter.

Since both test methods use the same parameters, parameterizing the test class allows to remove the duplication. To do so, the test class is annotated with the new @ParameterizedClass annotation and the @...Source annotation. The test methods are changed to use the @Test annotation.

@ParameterizedClass
@ValueSource(strings = {"foo", "bar"})
class SomeTests {
    @Parameter String value;
    
    @Test
    void shouldNotBeNull() {
        assertNotNull(value);
    }
    @Test
    void lengthShouldBeThree() {
        assertEquals(3, value.length());
    }
}

Instead of declaring the parameter on the methods, JUnit injects it into the @Parameter-annotated field. Alternatively, JUnit also supports constructor injection.

@ParameterizedClass
@ValueSource(strings = {"foo", "bar"})
record SomeTests(String value) {
    @Test
    void shouldNotBeNull() {
        assertNotNull(value);
    }
    @Test
    void lengthShouldBeThree() {
        assertEquals(3, value.length());
    }
}

In the example above, a Java record is used and the class-level parameters are specified as record components. This allows reducing the boilerplate of declaring a constructor and assigning its parameters to fields. Regular classes may be used as well, though.

A common use case

A common use case for parameterizing a test class is to run its tests against different implementations of an interface. Prior to JUnit 5.13, one would typically achieve that by writing an abstract base test class and creating a subclass for each concrete implementation. For example, the following test class implements two tests against Java’s List interface that are executed against three implementations of List that are part of the JDK.

class ListTests {

    private static abstract class AbstractListTests {

        List<String> list;

        @BeforeEach
        void initializeList() {
            list = createList();
        }

        protected abstract <T> List<T> createList();

        @Test
        void newListIsEmpty() {
            assertTrue(list.isEmpty());
        }

        @Test
        void itemCanBeAdded() {
            var added = list.add("value");

            assertTrue(added);
            assertTrue(list.contains("value"));
            assertEquals("value", list.getFirst());
        }
    }

    @Nested
    @DisplayName("ArrayList")
    class ArrayListTests extends AbstractListTests {
        @Override
        protected <T> List<T> createList() {
            return new ArrayList<>();
        }
    }

    @Nested
    @DisplayName("LinkedList")
    class LinkedListTests extends AbstractListTests {
        @Override
        protected <T> List<T> createList() {
            return new LinkedList<>();
        }
    }

    @Nested
    @DisplayName("Vector")
    class VectorTests extends AbstractListTests {
        @Override
        protected <T> List<T> createList() {
            return new Vector<>();
        }
    }
}

This test can be rewritten using @ParameterizedClass and a @MethodSource as follows.

@ParameterizedClass
@MethodSource("listImplementations")
class ParameterizedListTests {

    static Stream<?> listImplementations() {
        return Stream.of(
                argumentSet("ArrayList", new ArrayList<>()),
                argumentSet("LinkedList", new LinkedList<>()),
                argumentSet("Vector", new Vector<>())
        );
    }

    @Parameter
    List<String> list;

    @Test
    void newListIsEmpty() {
        assertTrue(list.isEmpty());
    }

    @Test
    void itemCanBeAdded() {
        var added = list.add("value");

        assertTrue(added);
        assertTrue(list.contains("value"));
        assertEquals("value", list.getFirst());
    }

    @AfterEach
    void clearList() {
        // Necessary since the `List` parameter is mutable!
        list.clear();
    }
}

This allows to get rid of the abstract base class and subclasses resulting in a much simpler structure.

Converters

The same use case can also be implemented using @ValueSource (and constructor injection) instead of @MethodSource as follows.

@ParameterizedClass(name = "[{index}] {0}")
@ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
class ParameterizedWithValueSourceListTests {

    List<String> list;

    ParameterizedWithValueSourceListTests(Class<? extends List<String>> listType) {
        this.list = ReflectionSupport.newInstance(listType);
    }
    
    @Test
    void newListIsEmpty() {
        // same as above...
    }

    @Test
    void itemCanBeAdded() {
        // same as above...
    }
}

Since @ValueSource only supports class literals, not concrete instances, the constructor relies on JUnit’s ReflectionSupport class to instantiate the List via the implementations’ default constructors. Whether it’s ok to use reflection here is, of course, debatable. Personally, I think it’s acceptable in this case since test code is executed with every build and would fail right away.

If you find yourself using such a conversion in multiple places, it might make sense to extract it into a custom ArgumentConverter and annotate the parameter with @ConvertWith instead.

@ParameterizedClass(name = "[{index}] {0}")
@ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
class ParameterizedWithValueSourceAndConverterListTests {

    @Parameter
    @ConvertWith(ClassToInstanceConverter.class)
    List<String> list;

    @Test
    void newListIsEmpty() {
        // same as above...
    }

    @Test
    void itemCanBeAdded() {
        // same as above...
    }

    @AfterEach
    void clearList() {
        // Necessary since the `List` parameter is mutable!
        list.clear();
    }
}

The custom ArgumentConverter implementation is called ClassToInstanceConverter in this example and contains the code that was part of the constructor in the previous example.

class ClassToInstanceConverter extends SimpleArgumentConverter {
    @Override
    protected Object convert(Object source, Class<?> targetType) {
        return ReflectionSupport.newInstance((Class<?>) source);
    }
}

You can even make this shorter by utilizing JUnit’s support for composed annotations.

@Retention(RUNTIME)
@Target({PARAMETER, FIELD})
@ConvertWith(ClassToInstanceConverter.class)
@interface Instantiate {
}

Of course, that also works with constructor injections, for example, using a Java record.

@ParameterizedClass(name = "[{index}] {0}")
@ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
record ParameterizedWithValueSourceAndConverterRecordListTests(
        @Instantiate List<String> list) {

    @Test
    void newListIsEmpty() {
        assertTrue(list.isEmpty());
    }

    @Test
    void itemCanBeAdded() {
        var added = list.add("value");

        assertTrue(added);
        assertTrue(list.contains("value"));
        assertEquals("value", list.getFirst());
    }

    @AfterEach
    void clearList() {
        // Necessary since the `List` parameter is mutable!
        list.clear();
    }
}

Mutable vs. immutable arguments

When using mutable data as parameters, one has to be mindful of tests that change state. In the above example, if itemCanBeAdded() were to run before newListIsEmpty() without the @AfterEach lifecycle method, newListIsEmpty() would fail because the list would no longer be empty. Therefore, it’s usually a better idea to use immutable data as parameters. In this case, this can be achieved by using a Supplier to create a list in the test class constructor. Since each test method uses a separate instance of the test class (unless @TestInstance(PER_CLASS) is used), this will prevent test methods from influencing each other. This also removes the need to reset the state of the parameters in an @AfterEach lifecycle method.

@ParameterizedClass
@MethodSource("listImplementations")
class ParameterizedListWithSuppliersTests {

    static Stream<?> listImplementations() {
        return Stream.of(
                argumentSet("ArrayList", (Supplier<?>) ArrayList::new),
                argumentSet("LinkedList", (Supplier<?>) LinkedList::new),
                argumentSet("Vector", (Supplier<?>) Vector::new)
        );
    }

    final List<String> list;

    ParameterizedListWithSuppliersTests(Supplier<List<String>> listSupplier) {
        this.list = listSupplier.get();
    }

    @Test
    void newListIsEmpty() {
        // same as above...
    }

    @Test
    void itemCanBeAdded() {
        // same as above...
    }
}

Argument sources

In addition to @ValueSource and @MethodSource, JUnit provides the following annotations for specifying the sources of arguments:

@EnumSource, @NullSource, and @EmptySource for simple values
@CsvSource, @CsvFileSource for CSV content
@FieldSource for custom code
@ArgumentsSource(MyProvider.class) for custom providers

Please refer to JUnit’s User Guide for details.

Lifecycle methods

Parameterized classes may declare @BeforeParameterizedClassInvocation and @AfterParameterizedClassInvocation lifecycle methods which are called once before/after each invocation of the parameterized class with a set of arguments. This may be used, for example, to initialize an argument as demonstrated in the following example.

@ParameterizedClass(name = "{0}")
@ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
record CustomInitializationListTests(
        @Instantiate List<String> list) {

    @BeforeParameterizedClassInvocation
    static void before(List<String> list) {
        if (list instanceof ArrayList<?> arrayList) {
            arrayList.ensureCapacity(100);
        }
    }

    @Test
    void newListIsEmpty() {
        assertTrue(list.isEmpty());
    }

    @Test
    void itemCanBeAdded() {
        var added = list.add("value");

        assertTrue(added);
        assertTrue(list.contains("value"));
        assertEquals("value", list.getFirst());
    }

    @AfterEach
    void clearList() {
        list.clear();
    }
}

Cartesian products

The @ParameterizedClass annotation may be combined with @Nested, also within an enclosing @ParameterizedClass-annotated test class. Moreover, a @ParameterizedClass may contain @ParameterizedTest methods. Both can be used to test all combinations or, mathematically speaking, the Cartesian product of two parameter lists.

The following example demonstrates that by testing all three concrete list implementations from the previous examples against each other in the @Nested Interoperability test class. The @Nested class additionally contains a @ParameterizedTest method to test against different String values.

@ParameterizedClass(name = "[{index}] {0}")
@ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
class ParameterizedWithNestedListTests {

    @Parameter
    @Instantiate
    List<String> list;

    @Test
    void newListIsEmpty() {
        assertTrue(list.isEmpty());
    }

    @Test
    void itemCanBeAdded() {
        var added = list.add("value");

        assertTrue(added);
        assertTrue(list.contains("value"));
        assertEquals("value", list.getFirst());
    }

    @AfterEach
    void clearList() {
        list.clear();
    }

    @Nested
    @ParameterizedClass(name = "[{index}] {0}")
    @ValueSource(classes = {ArrayList.class, LinkedList.class, Vector.class})
    class Interoperability {

        @Parameter
        @Instantiate
        List<String> secondList;

        @Test
        void twoListsWithSameItemsAreEqual() {
            list.add("foo");
            list.add("bar");

            secondList.addAll(list);

            assertEquals(secondList, list);
        }

        @ParameterizedTest
        @ValueSource(strings = {"baz", "qux"})
        void removeAllItemsInPassedList(String extraItem) {
            list.add("foo");
            list.add("bar");

            secondList.add("foo");
            secondList.add(extraItem);

            list.removeAll(secondList);

            assertEquals(1, list.size());
            assertEquals("bar", list.getFirst());
        }

        @AfterEach
        void clearList() {
            secondList.clear();
        }
    }
}

A word of warning: the number of combinations can quickly become very large! Therefore, you should take that into consideration when deciding whether to use this feature.

Summary

Parameterized test classes are a powerful testing tool that has long been missing from JUnit Jupiter. I’m super happy that I’ve finally had the chance to resolve this long-standing and highly-voted issue thanks to the Sovereign Tech Fund.

Edit: Thanks to my fellow JUnit 5 co-maintainer Sam Brannen for his feedback which I applied after initially publishing this post.

STF Milestone 3: Release verification and automation

Tue, 01 Apr 2025 00:01:00 +0000

A major goal of the Sovereign Tech Fund’s investment is to help projects become more sustainable and decrease their truck factor. For JUnit, one activity in desperate need of improvement in this area was performing a release. Prior to this milestone, all JUnit releases of the past years had been performed from my local computer.

Artifacts were uploaded to Maven Central via Sonatype’s OSSRH infrastructure from my machine. Documentation and sample projects were also updated by a local build running on my machine. In total, the release checklist had 24 steps that I would perform manually for every release. Whenever I told people this, I was afraid they would gasp. However, releases of JUnit were not that frequent and it never felt important enough to spend the little time I had on release automation.

While that may sound reasonable, doing releases this way was also a bad idea from a security perspective. A malicious actor would only have to compromise my machine (or me) in order to pull of a supply chain attack. A manipulated JUnit jar would soon find its way to a large portion of the Java ecosystem. While usually not part of any production system, JUnit runs during the build and could be used to manipulate production code. Therefore, I jumped at the opportunity to rectify this deficiency and proposed a milestone to the STF for verifying the release and automating the majority of it.

One additional reason for doing releases locally was that Maven Central requires artifacts to be signed with a PGP key. Doing the release on a CI system therefore would mean trusting that system with my private key. In my book, that’s a bad idea as well. Even if the CI system might only store the private key temporarily, it could have been compromised, have some kind of data leak etc. Thus, I decided to keep that step local. However, since JUnit’s artifacts are completely reproducible, I would keep building and signing them locally, but rebuild them on a CI system. Thereby, the release process would verify the integrity of the binaries and rule out that they had been tampered with.

Thus, I decided to keep the first part of the release process local:

Create a release branch
Change to release version
Change release date in README and release notes
Build and deploy release artifacts to Sonatype staging repository
Create a tag for the release
Change back to snapshot version
Push to GitHub

The remainder of the release is now performed by triggering a GitHub Actions workflow:

The most important job in the above workflow is “Verify reproducibility”. It checks out the Git tag, rebuilds all binaries, and verifies that they are identical to the ones uploaded to the staging repository. This step verifies not only their reproducibility but also rules out that they have been compromised by my local build.

The “Verify consumability” job updates all sample projects to consume the artifacts from the staging repository and builds them. Once the first phase is done, the staging repository is released. While waiting for the release artifacts to be synced to Maven Central, the documentation is published. Once the artifacts are indeed available on Maven Central, the sample projects are updated again but without the previous configuration that caused them to consume artifacts from the staging repository. Finally, a GitHub release is created.

Since I introduced this workflow in January, it has already been run 7 times taking between 15 and 26 minutes. It has definitely made releasing less stressful and I’m really happy how it turned out.

STF Milestone 2: Open Test Reporting

Fri, 21 Mar 2025 00:01:00 +0000

When JUnit was first released, it did not provide a feature to write test reports in machine-readable format. Build tools like Ant filled that gap and introduced an XML-based report. It was later adopted and sometimes extended by other Java-based build tools such as Maven and Gradle. CI tools like Jenkins introduced support for parsing these XML reports in order to display test results for each CI run. That led to other ecosystems also adopting the format to benefit from existing tool support. However, given that the format was initially defined for JUnit 3, it’s focused on test classes and methods and does not support nested structures. Some tools defined their own extensions of the schema that added such support or additional attributes. However, there are various partially conflicting schema definitions and subtle differences what gets written to which XML attributes since a proper “standard” was never agreed upon.

Open Test Reporting to the rescue

To address this situation, the JUnit team originally introduced the Open Test Reporting format in 2022 along with JUnit 5.9. It aims to replace the legacy XML format in a way that supports all features of JUnit 5. Moreover, it intends to support general testing concepts rather being tied to test classes and methods in order to appeal to other ecosystems as well. Additionally, it supports extension schemas for adding framework-specific information.

Establishing a new format will, of course, take some time. Testing frameworks need to be adjusted in order to write the new format which they will only do if there is significant interest in their user communities. Similarly, tools like CI servers will only add support for reading test results written in the new format once there is a considerable number of testing frameworks supporting it. Therefore, milestone 2 of my STF-supported work focused on making the new format production-ready and increasing its appeal by providing an API and command-line tool for converting from XML into a user-friendly, standalone HTML report.

Developing a new HTML report

It’s been a long time since I’ve been involved in setting up a frontend project from scratch. Therefore, I had mixed feelings before starting on the development of the HTML report. However, it actually turned out to be a lot of fun to work on once the initial setup was in place. Compared to Java, the development loop of making a change and seeing it in action was insanely fast. For reference, I decided to use Vue.js with Typescript and Tailwind CSS. In addition, there’s a little Java code that injects the test results into the template that is produced by Vite during the build process.

I introduced a CLI command that takes a single or multiple XML files as input and produces a self-contained HTML file. The test tree is rendered on the left with a details view on the right. The color and icon of the top bar changes depending on the overall status: it’s red if there was at least one failure and green otherwise.

The screenshot also shows one other major new features that was added in this milestone: file attachments. In case of images they are rendered inline. This is useful for integration and UI tests. The screenshot shows a browser test that is part of Open Test Reporting’s own test suite and verifies the HTML report gets rendered correctly. The TestReporter API in JUnit (along with a similar method in ExtensionContext for extensions) allows attaching files during test execution.

Of course, there’s also a dark theme:

The format is extensible via an SPI. For example, the “JUnit metadata” section from the screenshot above is contributed by a JUnit-specific extension. This allows other testing frameworks to render HTML sections if they write additional information to the XML reports.

Seeing it in action

If you want to see it in action, please have a look at JUnit’s CI builds which archive the reports. You can find them at the end of the summary page of every GitHub Actions workflow run. Unfortunately, GitHub does not (yet?) support browsing HTML files directly, so you’ll have to download the zip file and extract it locally. It contains one HTML report per subproject.

Even better, you can (and should!) try it out in your own project. JUnit‘s User Guide documents how to enable writing the XML report. You can then use the Open Test Reporting CLI to convert it to HTML.

java -jar open-test-reporting-cli-0.2.2.jar html-report \
	--output open-test-report.html \
	open-test-reporting.xml

Where to go from here

Once I was more or less done with the implementation part of this milestone, I reached out to testing framework maintainers in other ecosystems. Some were hesitant or lacked the time to investigate but others were quite enthusiastic. For example, PHPUnit right away implemented a proof-of-concept and drafted a concrete plan for adopting the Open Test Reporting format. Moreover, they plan to deprecate the “legacy” XML format in their next major version!

All in all, I am really happy with the outcome of this milestone! If you have any questions about any of the above, please raise an issue in the Open Test Reporting repository (if it’s about the format, CLI, etc.) or the JUnit 5 repo (if it’s JUnit-specific).

Happy testing!

STF Milestone 9: Jupiter extension improvements

Mon, 27 Jan 2025 00:01:00 +0000

As you can tell from the high number, this milestone was originally planned to be done later. However, a contributor (@JojOatXGME) was eager to work on it so I reached out to the STF and asked to change the order. They agreed and we changed the plan accordingly.

The issue addressed by this milestone was a crucial piece of missing functionality in the Jupiter extension API that kept popping up and was re-discussed every few weeks/months: there was no way to access the method-level extension context for TestInstancePostProcessor (and other) extensions which is important to create and destroy resources on the test class instance level. The hope was that resolving this issue would reduce friction for users and free up time for maintainers to work on other duties.

After the initial PR from Johannes that introduced a new annotation to opt-in to the new behavior, I iterated quite a bit on it based on feedback from the rest of the JUnit team. We decided to introduce a new interface with a method instead because it allows custom logic to determine which scope should be used (for example, checking for a configuration parameter):

public interface TestInstantiationAwareExtension extends Extension {
    default ExtensionContextScope getTestInstantiationExtensionContextScope(ExtensionContext ctx) {
       return ExtensionContextScope.DEFAULT;
    }
    enum ExtensionContextScope {  
       DEFAULT, TEST_METHOD
    }
}

I then added a configuration parameter to change the default behavior so it could easily be tested by consuming projects without editing any code. Moreover, I updated the User Guide and modified all core extensions to opt-in right away. This change introduced constructor injection support for @TempDir and now allows writing a test class that requires a temporary directory like this (using a record for brevity):

record TempDirTests(@TempDir Path tempDir) {
    @Test
    void shouldExists() {
       assertTrue(Files.exists(tempDir));
    }
    @Test
    void shouldAllowCreatingFiles() throws IOException {
       Files.createFile(tempDir.resolve("test.txt"));
    }
}

Summing up, I was grateful that the STF allowed us the flexibility to change the original plan. Even more importantly, I was really happy that this long-standing issue was finally resolved!

STF Milestone 1: Adopt Renovate

Sun, 19 Jan 2025 00:01:00 +0000

When drafting the project description for the STF, it was important to choose something as a first milestone that I could get done in under a month and that would have a direct benefit to the project right away and improve its maintainability. That’s why I eventually decided on migrating to Renovate for automating updates to dependencies and build tools.

We had already come a long way as a team when it comes to managing dependency updates. When we started to work on JUnit 5 in 2015, dependencies were few and we updated them manually. Over time, we adopted more tools, in particular in the build. We used the Gradle Versions Plugin for a while but it had its limitations. Eventually, we adopted Dependabot and switched to Gradle’s version catalogs which made updating our main dependencies much easier.

However, the amount of builds we needed to manage increased over time. Standalone projects such as the TestNG engine were introduced and samples for more build tools were added. Dependabot only supported a subset of those. It also did not support keeping the build tools themselves up to date. So we had a separate repository that used the Wrapper Upgrade Gradle Plugin for updating our Gradle and Maven (for samples) builds.

Adopting Renovate promised to enable us to remove this incomplete patchwork of tools with a single solution. So far, it has lived up to this promise. It not only supports Gradle and Maven, but also Bazel, sbt, and npm. We now also use it for pinning our dependencies in GitHub Action workflows. It allowed us to remove the separate repo for updating wrappers and gives us peace of mind that everything is shipshape regarding our dependencies.

The flip side of the coin are many PRs that can be quite noisy at times. While Renovate can be configured to work without PRs, we decided against that after discussing the topic in the JUnit team. We also decided to enable its auto-merge feature but require its PRs to be approved prior to merging to reduce the risk of supply chain attacks. Overall, I think we’re happy with the switch and would do it again.