Develocity Test Distribution User Manual | Develocity Documentation

Develocity Test Distribution takes your existing test suites and distributes them across remote agents to execute them faster.

See this explainer video for a quick introduction to the concept and its benefits including a product demo.

How it works

Historical test execution data provided by Develocity is used to create balanced partitions of similar expected execution times in order to optimize distribution across remote agents. The tests and their supporting files are transferred to each agent and executed, with their logging and results streamed back to the build in real time. If there are no remote agents available, tests can still be executed locally on the build machine.

This manual covers the build configuration aspect of using Test Distribution. Please refer to the Test Distribution Agent User Manual for information on how to deploy and scale agents.

Develocity provides an interface for administering remote agents, and viewing the current and historical system usage. Please see the Develocity Admin Manual for more information.

Test compatibility

Frameworks and languages

Tests must be run via JUnit Platform, which is part of JUnit 5. Many popular test frameworks are compatible with JUnit Platform and are supported by Test Distribution, such as JUnit (including JUnit 3 and JUnit 4), Spock, Kotest (version 5.6.0 and later), TestNG, jqwik, ArchUnit and more. Cucumber tests are also supported, but they do require some extra configuration via the Cucumber-Companion plugin. Please see Gradle Testing in Java & JVM projects guide or Maven Surefire’s documentation for guidance on configuring your build to use JUnit Platform. When in doubt, please contact Develocity support to assess whether your setup is compatible with Test Distribution.

Tests and sources authored in popular JVM languages are supported, including Java, Kotlin, Groovy, and Scala.

Some JUnit Platform test engines, such as Spek, are currently not supported. Please contact Develocity support if you are interested in using this or other incompatible test engines with Test Distribution.

Gradle

Develocity Test Distribution is compatible with Gradle 5.4 and later, and works with Gradle’s built-in Test task and its subclasses. This includes Android unit test tasks, but does not include most Android device tests.

Tests must be run with Java 8 or later.

Maven

Develocity Test Distribution is compatible with the Maven Surefire plugin 2.22.2 and later, and Maven Failsafe plugin 2.22.2 and later.

Tests must be run with Java 8 or later.

External test resources

Consideration must be given to any supporting resources required by tests.

Test Distribution takes your tests and executes them on remote agents. The test runtime classpath and any additionally declared file inputs are transferred to the agents. If your tests are expecting any other resources to be locally available, such as a running database, the test agents must also provide the resource or the tests will fail.

There are two possible approaches to using such resources in your tests:

Have the tests themselves provision necessary resources (e.g. using Docker containers)
Deploy Test Distribution agents in environments that provide all required resources

The first approach is recommended where possible as it is more portable and minimizes the need to change agent environments when test resources change. The Testcontainers project provides an easy way to utilize Docker containers within tests.

Depending on how you deploy Test Distribution agents, additional configuration may be required in order to make Docker accessible to the tests running on those agents. Please refer to the Test Distribution Agent User Manual for more information on this topic.

When using the second approach of having the agent environment provide resources, it is important to declare the requirements of tests so that appropriate agents that provide the capability can be assigned.

Prerequisites

Test Distribution requires a Develocity 2020.2/2020.5 (for Gradle/Maven) or later installation with connected Test Distribution agents. For information on configuring a Develocity installation to enable Test Distribution, please see the Develocity Admin Manual.

Your build must authenticate with Develocity via an access key and the user be granted the “Test Distribution” access control permission. Unauthenticated usage of Test Distribution is prohibited due to the inherent security implications of such a remote code execution service. See below for build-tool specific authentication instructions.

The build and Test Distribution agents connect to the Develocity server using a WebSocket connection. In case of connection issues, please ensure all load balancers and proxies that are used between both ends support WebSocket connections.

Build configuration

Gradle

(Develocity 2020.2+)

The configuration examples in this section assume you are using Develocity Gradle plugin 3.17 or later. For older versions, please refer to the (Legacy) Gradle Enterprise Gradle Plugin User Manual.

Applying the Gradle plugin

Your build must use Gradle 5.4 or later, and use the Develocity Gradle plugin to connect to the Develocity server.

Develocity 2022.3 and later

Starting with Develocity 2022.3, the Develocity Gradle plugin is the only Gradle plugin that needs to be applied to your build as a prerequisite of using Test Distribution. Please refer to the Develocity Gradle Plugin User Manual for detailed information on how to do this or see below for the short version.

Gradle 6.0 and above

The plugin must be applied in the settings file of the build.

settings.gradle.kts

plugins {
    id("com.gradle.develocity") version "3.18.2"
}

develocity {
    server.set("https://develocity.mycompany.com")
}

settings.gradle

plugins {
    id('com.gradle.develocity') version '3.18.2'
}

develocity {
    server = 'https://develocity.mycompany.com'
}

Gradle 5.4 to 5.6.x

The plugin must be applied to the root project of the build.

build.gradle.kts

plugins {
    id("com.gradle.develocity") version "3.18.2"
}

develocity {
    server.set("https://develocity.mycompany.com")
}

build.gradle

plugins {
    id('com.gradle.develocity') version '3.18.2'
}

develocity {
    server = 'https://develocity.mycompany.com'
}

Develocity 2022.2.x and earlier

Earlier versions of Develocity require applying the Develocity Test Distribution Gradle plugin in addition to the Develocity Gradle plugin as a prerequisite of using Test Distribution. Its plugin ID is com.gradle.enterprise.test-distribution (last version: 2.3.5).

Gradle 6.0 and above

The plugin must be applied in the settings file of the build in addition to the com.gradle.enterprise plugin.

settings.gradle.kts

plugins {
    id("com.gradle.enterprise") version "3.10.3"
    id("com.gradle.enterprise.test-distribution") version "2.3.5"
}

gradleEnterprise {
    server = "https://develocity.mycompany.com"
}

settings.gradle

plugins {
    id('com.gradle.enterprise') version '3.10.3'
    id('com.gradle.enterprise.test-distribution') version '2.3.5'
}

gradleEnterprise {
    server = 'https://develocity.mycompany.com'
}

Gradle 5.4 to 5.6.x

The plugin must be applied to the root project of the build in addition to the com.gradle.build-scan plugin.

build.gradle.kts

plugins {
    id("com.gradle.build-scan") version "3.10.3"
    id("com.gradle.enterprise.test-distribution") version "2.3.5"
}

gradleEnterprise {
    server = "https://develocity.mycompany.com"
}

build.gradle

plugins {
    id('com.gradle.build-scan') version '3.10.3'
    id('com.gradle.enterprise.test-distribution') version '2.3.5'
}

gradleEnterprise {
    server = 'https://develocity.mycompany.com'
}

Authentication

Builds must be authenticated with Develocity via an access key in order to use Test Distribution. Please see Authenticating Gradle builds with Develocity for how to do this.

Enabling

Test Distribution is enabled and configured per task, via the distribution DSL extension.

build.gradle.kts

tasks.test {
    useJUnitPlatform() (1)
    develocity.testDistribution {
        enabled.set(true) (2)
    }
}

build.gradle

tasks.named('test', Test) {
    useJUnitPlatform() (1)
    develocity.testDistribution {
        enabled = true (2)
    }
}

1	Use JUnit Platform for executing tests (see Test compatibility)
2	Enable Test Distribution

Transferring test inputs & outputs between build and remote agents comes at a cost. For this reason you should not activate Test Distribution for each test task in the build. Good candidates for distribution are test tasks that take more than a few seconds to execute. Test tasks that only execute fast unit tests should not be distributed as they will compete for remote agents thereby slowing other Test Distribution jobs down.

Executors

Tests can be executed by local and/or remote executors.

Local executors

By default, the number of local executors is derived from the maxParallelForks property of the task. It is further constrained by the maximum number of workers (defaults to number of CPU processors/cores) configured to be used for your build. To limit the use of local executors, configure the maxLocalExecutors property. Setting it to zero, causes tests to be only executed remotely.

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        maxLocalExecutors.set(2)
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        maxLocalExecutors = 2
    }
}

Remote executors

By default, test execution uses as many remote executors as possible. To limit the number used, you can configure the maxRemoteExecutors property:

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        maxRemoteExecutors.set(2)
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        maxRemoteExecutors = 2
    }
}

Each test task will request at most maxRemoteExecutors (or as many as possible if not set) and test tasks that are running in parallel will each request their respective configured number of remote executors. So a build running two test tasks in parallel each having maxRemoteExecutors = 10 will request 20 remote executors. The number of remote executors actually used is subject to agent availability and the number of test partitions.

Setting the number of remote executors to zero causes tests to be only executed locally.

Executor preference

Since local executors are typically the first to respond to execution requests, the longest running tests are usually assigned to them. To favor remote over local execution (e.g. to unburden the build machine), you can set the remoteExecutionPreferred property to true.

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        remoteExecutionPreferred.set(true)
        waitTimeout.set(Duration.ofMinutes(2))
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        remoteExecutionPreferred = true
        waitTimeout = Duration.ofMinutes(2)
    }
}

If enabled, remote executors are tried first, falling back to local ones after waitTimeout expires.

When first enabling Test Distribution for an existing test suite, it is a good idea to start with only local executors to verify compatibility of the tests with Test Distribution.

When using Gradle’s offline mode, only local executors will be used.

Executor restrictions

Some tests can only be executed by local or remote executors. Prior to Develocity Gradle Plugin version 3.11, you had to define a separate test task with maxRemoteExecutors=0 or maxLocalExecutors=0 to control where tests are executed. Starting with version 3.11, you can use the a localOnly and remoteOnly configuration.

This can be achieved in two ways: by specifying a class name filter pattern, via the includeClasses configuration property, or in an annotation-driven manner, via the includeAnnotationClasses configuration property. Both configuration properties can be applied simultaneously.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

It is an error when a test is matched by both localOnly and remoteOnly. It is also an error when a test is matched by localOnly and maxLocalExecutors=0 is set and vice versa for remoteOnly and maxRemoteExecutors=0.

You can also use the LocalOnly and RemoteOnly annotations from the develocity-testing-annotations project without any additional configuration other than the dependency (version 1.x of the annotations is supported starting with 3.12, 2.x starting with 3.16).

Local-/remote-only class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        localOnly {
            includeClasses.addAll("com.project.FileIntegrityTest", "*LocalTest")
        }
        remoteOnly {
            includeClasses.addAll("com.project.SpecificTest", "*RemoteTest")
        }
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        localOnly {
            includeClasses = ["com.project.FileIntegrityTest", "*LocalTest"]
        }
        remoteOnly {
            includeClasses = ["com.project.SpecificTest", "*RemoteTest"]
        }
    }
}

Local-/remote-only annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class’ annotations. A class need only have one annotation matching any of the patterns to be included.

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        localOnly {
            includeAnnotationClasses.addAll("com.project.LocalOnly")
        }
        remoteOnly {
            includeAnnotationClasses.addAll("com.project.RemoteOnly")
        }
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        localOnly {
            includeAnnotationClasses = ["com.project.LocalOnly"]
        }
        remoteOnly {
            includeAnnotationClasses = ["com.project.RemoteOnly"]
        }
    }
}

Refer to the Example local-only annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Requirements

Test tasks can specify requirements that agents must fulfill in order to be considered compatible. Requirements are matched against the advertised capabilities of the agents that are connected to your Develocity server. Only agents that fulfill all requirements are considered compatible. Local executors are assumed to provide all required capabilities. This means that local executors will always be used regardless of the declared requirements, unless use of local executors is disabled.

Requirements are strings that may only contain alphanumeric characters, dashes, underscores, periods, and the relational operators =, <, >, <= or >=. It is recommended to use a key-value form (e.g. os=linux), expression form (e.g. memory>=2.5Gi) or tag-like form (e.g. postgres) to model requirements.

When using expression form, the right-hand side of the expression can be any decimal number with an optional unit suffix. See Build Requirements Matching Details for further information.

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        requirements.set(setOf("os=linux", "postgres"))
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        requirements = ['os=linux', 'postgres']
    }
}

To ensure tests are executed using the intended Java version, an implicit jdk=«version» requirement is added to the set of declared requirements for the task. The version is determined from the Java executable set on the task. Only the major version is taken into account (e.g. jdk=8 for Java 8 and jdk=11 for Java 11).

For more information on how to define the capabilities of an agent, please see the Test Distribution Agent User Manual.

Wait timeout

Test Distribution relies on external services in order to execute your build, such as the Develocity server and assigned remote agents. When an agent disconnects during the build, the plugin reschedules the affected test partition on another local or remote executor. Similarly, when the connection to the Develocity server is closed unexpectedly, the plugin attempts to reconnect and reschedules the affected test partitions. However, in cases test execution cannot progress, e.g. if local executors are disabled and reconnection attempts are not successful, the plugin will fail the build after a configurable wait timeout (30 seconds by default) has elapsed in order to prevent the build from blocking indefinitely.

build.gradle.kts

import java.time.Duration

tasks.test {
    develocity.testDistribution {
        enabled.set(true)
        waitTimeout.set(Duration.ofMinutes(2))
    }
}

build.gradle

import java.time.Duration

tasks.named('test', Test) {
    develocity.testDistribution {
        enabled = true
        waitTimeout = Duration.ofMinutes(2)
    }
}

As long as local executors are configured the build can always progress and thus will not time out. If local executors are disabled but there are some compatible remote executors connected, the build will wait as long as it takes for them to become available even if that takes longer than waitTimeout.

Inputs and outputs

All inputs declared on the test task are tracked by Test Distribution and will be transferred to remote executors. In order to add additional input files that are required for your tests use the runtime task inputs API:

build.gradle.kts

tasks.test {
    inputs.file("src/test-data/data.xml")
    develocity.testDistribution {
        enabled.set(true)
    }
}

build.gradle

tasks.named('test', Test) {
    inputs.file 'src/test-data/data.xml'
    develocity.testDistribution {
        enabled = true
    }
}

See Tests reading files from src/test/resources for a common example of missing input declarations.

Tests may produce file outputs, such as reports. After finishing execution of a test partition all output files that are registered on the test task are transferred back from remote executors to the build. Test reports and JaCoCo coverage data are automatically tracked.

build.gradle.kts

tasks.test {
    outputs.dir("$buildDir/additional-outputs")
    develocity.testDistribution {
        enabled.set(true)
    }
}

build.gradle

tasks.named('test', Test) {
    outputs.dir "$buildDir/additional-outputs"
    develocity.testDistribution {
        enabled = true
    }
}

If two partitions produce the same output file, the test task will fail. This means that you must create unique files per test class. Directory structures are merged as long as they don’t contain conflicting files.

Reproducible archives

To maximise the effectiveness of caching and minimise the number of input files transferred, it is recommended to configure Gradle’s AbstractArchiveTask to produce a stable output by discarding file timestamps and applying consistent file ordering in the archive.

For more information about configuring Gradle to produce reproducible archives, refer to the Gradle documentation.

Other task properties

In addition to the above configuration properties of the distribution DSL extension, Test Distribution respects the Test task’s configuration. The following properties influence test partitioning and the type and number of used local and remote executors.

maxParallelForks: Used as default number of local executors.

Test Distribution agents always only run a single fork. Changing the maxParallelForks setting therefore has no impact on remote test execution. If you want to better utilize compute resources of the machines hosting agents you should start several agent instances on a single machine.

debug (--debug-jvm command line option): If enabled, tests are executed on a single executor that you can attach a debugger to as usual. When using Gradle plugin 3.17+, if local executors are disabled or remote execution is preferred, debugging will be attempted on a remote executor; otherwise, a local executor will be used.
forkEvery: If set, the number is used to limit the maximum size of each test partition.
jvmArgs, minHeapSize, maxHeapSize, systemProperties: Used locally and on agents when forking JVMs to execute tests. If maxHeapSize is not set, local and remote executors use the same default as the built-in test task, which is 512 MiB maximum heap size. Paths of inputs and outputs in JVM arguments are automatically replaced to match the agent’s temporary workspace.
environment: All environment variables configured on the test task are passed to locally forked JVMs. For remote JVMs, only environment variables with values different from those of the Gradle daemon running the build are propagated. Paths of inputs and outputs in environment variable values are automatically replaced to match the agent’s temporary workspace. In addition, remote JVMs inherit the environment variables of the agent process. You should prefer using system properties over environment variables to inject values into your tests, whenever possible.

The following properties are currently not supported:

failFast (--fail-fast command line option)
modularity (incubating since Gradle 6.4)

Integration with test coverage tools

JaCoCo

Test Distribution is compatible with Gradle’s JaCoCo plugin. Coverage information from all executors, including remote, are merged to produce a single coverage report. No additional configuration is required.

IntelliJ coverage agent and Kover plugin

Test Distribution is incompatible with IntelliJ’s Run with coverage functionality.

The Kover Gradle plugin, which uses IntelliJ’s coverage agent by default, is also not supported. However, the plugin allows to use the JaCoCo coverage library instead. When JaCoCo library is used, the coverage data collected by the plugin will be automatically tracked.

Tests will run locally if the test coverage data is requested via IntelliJ or Kover plugin.

Integration with test retry

Test Distribution is compatible with the integrated test retry functionality of the Develocity Gradle plugin version 3.12 or above. To configure retries of failed tests, you can use the retry DSL extension provided by the Develocity Gradle plugin:

build.gradle.kts

tasks.test {
    useJUnitPlatform()
    develocity{
        testDistribution {
            enabled.set(true)
            retryInSameJvm.set(true) (1)
        }
        testRetry {
            maxRetries.set(3)
            maxFailures.set(20)
            failOnPassedAfterRetry.set(true)
        }
    }
}

build.gradle

tasks.named('test', Test) {
    useJUnitPlatform()
    develocity {
        testDistribution {
            enabled = true
            retryInSameJvm = true (1)
        }
        testRetry {
            maxRetries = 3
            maxFailures = 20
            failOnPassedAfterRetry = true
        }
    }
}

1	Controls whether the retry is performed in the same JVM (default), or a new JVM. Retrying in a new JVM can help mitigate problems such as static pollution, but it will be slower as it incurs the startup cost of a new JVM.

When using on older version of the Develocity Gradle plugin, you can use the test retry functionality of the test retry plugin version 1.1.4 or above. To configure retries of failed tests, you can use the retry DSL extension provided by the test retry plugin:

build.gradle.kts

plugins {
    id("org.gradle.test-retry") version "1.6.0" (1)
}
tasks.test {
    useJUnitPlatform()
    develocity.testDistribution {
        enabled.set(true)
        retryInSameJvm.set(true)
    }
    retry {
        maxRetries.set(3)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}

build.gradle

plugins {
    id('org.gradle.test-retry') version '1.6.0' (1)
}
tasks.named('test', Test) {
    useJUnitPlatform()
    develocity.testDistribution {
        enabled = true
        retryInSameJvm = true
    }
    retry {
        maxRetries = 3
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}

1	Apply the test retry plugin

Please note that retry semantics change slightly: both maxRetries and maxFailures are applied for each test partition instead of the entire test task. Moreover, while the test retry plugin forks a new JVM for each retry, Test Distribution retries tests in the same JVM in which they originally failed unless retryInSameJvm is set to false. The granularity might also differ slightly in some cases, e.g. Test Distribution retries individual invocations of parameterized JUnit Jupiter tests but retries the whole test class for data-driven Spock tests due to a limitation of the JUnit Vintage Engine.

Integration with other plugins

Gradle plugin development plugin

Running functional tests using the GradleRunner is supported out of the box. No additional configuration is required.

Android Gradle plugin

Test Distribution is only supported for Gradle’s built-in test task. For that reason only local unit tests (located under module-name/src/test/java/) can be distributed. Distributing instrumented tests (located under module-name/src/androidTest/java/) is not supported.

Test dry run mode

(Gradle plugin 3.14.1+)

To simulate the execution of tests without actually running them, Develocity provides a test dry run mode. This mode generates reports, distribute tests to agents, and transfer files from the build to Develocity. This can e.g. be used to verify that your test filtering configuration or Test Distribution configuration is correct, without actually running the tests.

The test dry run mode can be enabled by:

Adding --test-dry-run command line option or setting the dryRun test task property to true (Gradle 8.3+)
Setting the junit.platform.execution.dryRun.enabled system property to true (JUnit 5.10+)

Fallback to regular execution

(Gradle plugin 3.16+)

Test Distribution relies on JUnit Platform to run tests. By default, the test task will fail if it isn’t configured to use JUnit Platform. To avoid that fall back to regular test execution, set fallbackToRegularExecutionOnMissingPrerequisites to true when centrally enabling Test Distribution.

This behavior can be enabled by setting the property in the build script:

build.gradle.kts

tasks.test {
    develocity.testDistribution {
        fallbackToRegularExecutionOnMissingPrerequisites.set(true) // default: false
    }
}

build.gradle

tasks.named('test', Test) {
    develocity.testDistribution {
        fallbackToRegularExecutionOnMissingPrerequisites = true // default: false
    }
}

Or, by setting the corresponding system property:

$ ./gradlew test -Ddevelocity.testing.fallbackToRegularExecutionOnMissingPrerequisites=true

The fallbackToRegularExecutionOnMissingPrerequisites property was called fallbackToRegularExecutionOnMissingJUnitPlatform in Gradle plugin versions 3.16.x.

Maven

(Maven extension 1.8+, Develocity 2020.5+)

The configuration examples in this section assume you are using Develocity Maven extension 1.21 or later. For older versions, please refer to the (Legacy) Gradle Enterprise Maven Extension User Manual.

Applying the extension

A prerequisite of using Test Distribution is applying and configuring the Develocity Maven extension to your build. Refer to the Develocity Maven extension user manual for detailed information on how to do this.

Authentication

Builds must be authenticated with Develocity via an access key in order to use Test Distribution. Please see Authenticating Maven builds with Develocity for how to do this.

Enabling

The examples in this and the following sections reference the maven-surefire-plugin but the same configuration options are available for the maven-failsafe-plugin.

Test Distribution is enabled and configured per goal, via the distribution element in the POM.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled> (1)
      </distribution>
    </properties>
  </configuration>
</plugin>

1	Enable Test Distribution

Transferring test inputs & outputs between build and remote agents comes at a cost. For this reason you should not activate Test Distribution for each test goal in the build. Good candidates for distribution are test goals that take more than a few seconds to execute. Test goals that only execute fast unit tests should not be distributed as they will compete for remote agents thereby slowing other Test Distribution jobs down.

Executors

Tests can be executed by local and/or remote executors.

Local executors

By default, the number of local executors is derived from the forkCount parameter of the goal. To limit the use of local executors, configure the maxLocalExecutors parameter. Setting it to zero, causes tests to be only executed remotely.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <maxLocalExecutors>2</maxLocalExecutors> (1)
      </distribution>
    </properties>
  </configuration>
</plugin>

1	Limit number of local executors to 2

Remote executors

By default, test execution uses as many remote executors as possible. To limit the number used, you can configure the maxRemoteExecutors parameter:

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <maxRemoteExecutors>2</maxRemoteExecutors> (1)
      </distribution>
    </properties>
  </configuration>
</plugin>

1	Limit number of remote executors to 2

Each test goal will request at most maxRemoteExecutors (or as many as possible if not set) and test goals that are running in parallel will each request their respective configured number of remote executors. So a build running two test goals in parallel each having maxRemoteExecutors set to 10 will request 20 remote executors. The number of remote executors actually used is subject to agent availability and the number of test partitions.

Setting the number of remote executors to zero causes tests to be only executed locally.

Executor preference

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <remoteExecutionPreferred>true</remoteExecutionPreferred>
        <waitTimeoutInSeconds>120</waitTimeoutInSeconds>
      </distribution>
    </properties>
  </configuration>
</plugin>

If enabled, remote executors are tried first, falling back to local ones after waitTimeoutInSeconds expires.

When first enabling Test Distribution for an existing test suite, it is a good idea to start with only local executors to verify compatibility of the tests with Test Distribution.

When using Maven’s offline mode, only local executors will be used.

Executor restrictions

Some tests can only be executed by local or remote executors. Prior to Develocity Maven Extension version 1.15, you had to define a separate test goal with maxRemoteExecutors=0 or maxLocalExecutors=0 to control where tests are executed. Starting with version 1.15, you can use the a localOnly and remoteOnly configuration.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

Local-/remote-only class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <localOnly>
          <includeClasses>
            <include>example.project.FileIntegrityTest</include>
            <include>*LocalTest</include>
          </includeClasses>
        </localOnly>
        <remoteOnly>
          <includeClasses>
            <include>example.project.SpecificTest</include>
            <include>*RemoteTest</include>
          </includeClasses>
        </remoteOnly>
      </distribution>
    </properties>
  </configuration>
</plugin>

Local-/remote-only annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class' annotations. A class need only have one annotation matching any of the patterns to be included.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <localOnly>
          <includeAnnotationClasses>
            <include>com.project.annotations.LocalOnly</include>
          </includeAnnotationClasses>
        </localOnly>
        <remoteOnly>
          <includeAnnotationClasses>
            <include>com.project.annotations.RemoteOnly</include>
          </includeAnnotationClasses>
        </remoteOnly>
      </distribution>
    </properties>
  </configuration>
</plugin>

Refer to the Example local-only annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Requirements

Test goals can specify requirements that agents must fulfill in order to be considered compatible. Requirements are matched against the advertised capabilities of the agents that are connected to your Develocity server. Only agents that fulfill all requirements are considered compatible. Local executors are assumed to provide all required capabilities. This means that local executors will always be used regardless of the declared requirements, unless use of local executors is disabled.

When using expression form, the right-hand side of the expression can be any decimal number with an optional unit suffix. See Build Requirements Matching Details for further information.

When using the relational operators < and > in pom.xml you need to write them in entity notation, e.g. memory>=2Gi.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <requirements>
          <requirement>os=linux</requirement> (1)
          <requirement>postgres</requirement> (2)
        </requirements>
      </distribution>
    </properties>
  </configuration>
</plugin>

1	Add `os=linux` requirement
2	Add `postgres` requirement

To ensure tests are executed using the intended Java version, an implicit jdk=«version» requirement is added to the set of declared requirements for the goal. The version is determined from the Java executable set on the goal. Only the major version is taken into account (e.g. jdk=8 for Java 8 and jdk=11 for Java 11).

For more information on how to define the capabilities of an agent, please see the Test Distribution Agent User Manual.

Wait timeout

Test Distribution relies on external services in order to execute your build, such as the Develocity server and assigned remote agents. When an agent disconnects during the build, the extension reschedules the affected test partition on another local or remote executor. Similarly, when the connection to the Develocity server is closed unexpectedly, the extension attempts to reconnect and reschedules the affected test partitions. However, in cases test execution cannot progress, e.g. if local executors are disabled and reconnection attempts are not successful, the extension will fail the build after a configurable wait timeout (30 seconds by default) has elapsed in order to prevent the build from blocking indefinitely.

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <enabled>true</enabled>
        <waitTimeoutInSeconds>120</waitTimeoutInSeconds> (1)
      </distribution>
    </properties>
  </configuration>
</plugin>

1	Configure wait timeout of 120 seconds

Inputs and outputs

All well-known and manually declared inputs of the test goal are tracked by Test Distribution and will be transferred to remote executors. In order to add additional input files that are required for your tests declare them using in the POM configuration of the Develocity Maven extension:

pom.xml

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.5.2</version>
      <configuration>
        <properties>
          <distribution>
            <enabled>true</enabled>
          </distribution>
        </properties>
      </configuration>
    </plugin>
  </plugins>
  <pluginManagement>
    <plugins>
      <plugin>
        <groupId>com.gradle</groupId>
        <artifactId>develocity-maven-extension</artifactId>
        <configuration>
          <develocity>
            <plugins>
              <plugin>
                <artifactId>maven-surefire-plugin</artifactId> (1)
                <inputs>
                  <fileSets>
                    <fileSet>
                      <name>test-data</name>
                      <paths>
                        <path>src/test-data/data.xml</path> (2)
                      </paths>
                    </fileSet>
                  </fileSets>
                </inputs>
              </plugin>
            </plugins>
          </develocity>
        </configuration>
      </plugin>
    </plugins>
  </pluginManagement>
</build>

1	Reference the `maven-surefire-plugin`
2	Declare `src/test-data/data.xml` as an additional input file

See Tests reading files from src/test/resources for a common example of missing input declarations.

Tests may produce file outputs, such as reports. After finishing execution of a test partition all output files that are registered on the test goal are transferred back from remote executors to the build. Test reports and JaCoCo coverage data are automatically tracked.

pom.xml

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.5.2</version>
      <configuration>
        <properties>
          <distribution>
            <enabled>true</enabled>
          </distribution>
        </properties>
      </configuration>
    </plugin>
  </plugins>
  <pluginManagement>
    <plugins>
      <plugin>
        <groupId>com.gradle</groupId>
        <artifactId>develocity-maven-extension</artifactId>
        <configuration>
          <develocity>
            <plugins>
              <plugin>
                <artifactId>maven-surefire-plugin</artifactId> (1)
                <outputs>
                  <directories>
                    <directory>
                      <name>additional-outputs</name>
                      <path>${project.build.directory}/additional-outputs</path> (2)
                    </directory>
                  </directories>
                </outputs>
              </plugin>
            </plugins>
          </develocity>
        </configuration>
      </plugin>
    </plugins>
  </pluginManagement>
</build>

1	Reference the `maven-surefire-plugin`
2	Declare `target/additional-outputs` as an additional output directory

If two partitions produce the same output file, the test goal will fail. This means that you must create unique files per test class. Directory structures are merged as long as they don’t contain conflicting files.

Other configuration parameters

In addition to the above configuration parameters, the Develocity Maven extension respects most of the goal’s configuration. The following parameters influence test partitioning and the type and number of used local and remote executors.

forkCount: Used as default number of local executors. Disabling forking by setting forkCount=0 is not supported.

Test Distribution agents always only run a single fork.Changing the forkCount setting therefore has no impact on remote test execution. If you want to better utilize compute resources of the machines hosting agents you should start several agent instances on a single machine.

debugForkedProcess (maven.surefire.debug user property): If enabled, tests are executed on a single executor that you can attach a debugger to as usual. When using Develocity Maven extension 1.21+, if local executors are disabled or remote execution is preferred, debugging will be attempted on a remote executor; otherwise, a local executor will be used.
reuseForks: If disabled, each test partition will be limited to a single test class.
argLine, systemProperties, systemPropertiesFile, systemPropertyVariables: Used locally and on agents when forking JVMs to execute tests. Paths of inputs and outputs in JVM arguments are automatically replaced to match the agent’s temporary workspace.

By default, Surefire/Failsafe lets the forked JVM determine the maximum heap size based on the machine’s physical memory. Since the outcome depends on the used JVM version and implementation, it’s usually a good idea to configure -Xmx explicitly via the argLine configuration parameter.

environmentVariables, excludedEnvironmentVariables: The environment variables of the Maven process (excluding those in excludedEnvironmentVariables) along with those explicitly configured via environmentVariables are passed to locally forked JVMs. For remote JVMs, only explicitly configured environment variables are propagated. Paths of inputs and outputs in environment variable values are automatically replaced to match the agent’s temporary workspace. In addition, remote JVMs inherit the environment variables of the agent process. You should prefer using system properties over environment variables to inject values into your tests, whenever possible.
rerunFailingTestsCount: If set to a value greater than 0, failing tests will be retried the specified number of times.

The following configuration parameters are currently not supported: enableProcessChecker, forkedProcessExitTimeoutInSeconds, forkNode, junitArtifactName, objectFactory, parallel, parallelOptimized, parallelTestsTimeoutForcedInSeconds, parallelTestsTimeoutInSeconds, perCoreThreadCount, runOrder, shutdown, suiteXmlFiles, testNGArtifactName, threadCount, threadCountClasses, threadCountMethods, threadCountSuites, useManifestOnlyJar, useSystemClassLoader, useUnlimitedThreads.

Integration with other plugins

JaCoCo plugin

Test Distribution is compatible with JaCoCo Maven plugin. Coverage information from all executors, including remote, are merged to produce a single coverage report. No additional configuration is required.

Test dry run mode

(Maven extension 1.18.1+)

To simulate the execution of the tests without actually running them, Develocity provides a test dry run mode. This will still generate reports, distribute tests to agents, and transfer files from the build to Develocity. This can e.g. be used to verify that your test filtering configuration or Test Distribution configuration is correct, without actually running the tests.

The test dry run mode can be enabled by:

Setting the junit.platform.execution.dryRun.enabled configuration parameters property to true (JUnit 5.10+)

Fallback to regular execution

(Maven extension 1.20+)

Test Distribution relies on JUnit Platform and at least Surefire 2.22.2 to run tests. By default, the test goal will fail if these prerequisites are not fulfilled. To avoid that fall back to regular test execution, set fallbackToRegularExecutionOnMissingPrerequisites to true when centrally enabling Test Distribution.

This behavior can be enabled by setting the property in the POM:

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <properties>
      <distribution>
        <fallbackToRegularExecutionOnMissingPrerequisites>true</fallbackToRegularExecutionOnMissingPrerequisites> <!-- default: false -->
      </distribution>
    </properties>
  </configuration>
</plugin>

Or, by setting the corresponding system property:

$ ./mvnw clean test -Ddevelocity.testing.fallbackToRegularExecutionOnMissingPrerequisites=true

The fallbackToRegularExecutionOnMissingPrerequisites property was called fallbackToRegularExecutionOnMissingJUnitPlatform in Maven extension versions 1.20.x.

Example local-only annotation

Any annotation class referenced in the includeAnnotationClasses configuration should be implemented to be retained at runtime and target types (class level). See the listing below for a sample implementation.

LocalOnly.java

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface LocalOnly {}

Annotations present on super-classes with the @Inherited meta-annotation are considered when inspecting subclasses. Note that annotations are not inherited from implemented interfaces.

Build Scans

(Develocity 2023.2+, Gradle plugin 3.14+, Develocity Maven extension 1.18+)

From Develocity 2023.2 onwards, detailed information and metrics about Test Distribution can be accessed in the Build Scans tests view.

In the tests view, task-/goal-level and class-level inspectors summarize the most important data about Test Distribution usage. In those inspectors, you can check whether Test Distribution was enabled, which executors have been used, and how much time has been saved by using Test Distribution.

Task-/goal-level inspector

Class-level inspector

A more comprehensive list of Test Distribution-related information is provided in the Test Distribution tab, in the task/goal view. There, you can view how Test Distribution has been configured, which impact it had on the build’s performance, which input files had to be transferred to Develocity, and which output files back from Develocity.

build scan workunit view test distribution

Configuration

This section shows the values that have been implicitly or explicitly set in the Test Distribution configuration. Please refer to Build configuration for additional details.

Execution

Local executors used and remote executors used: number of distinct local and remote executors that have been used to run the test task/goal
Potentially compatible remote executors: total number of remote executors with matching requirements, that could have been used to run the test task/goal
Average concurrently used executors and max concurrently used executors: average/maximum number of executors that were in use at any point of time during the test task/goal execution
Average concurrently requested executors and max requested executors: average/maximum number of executors that were requested (i.e. pending for execution or in use) at any point of time during the test task/goal execution

Performance

Test discovery duration: time spent by Test Distribution to discover the tests that have to be executed
Time spent waiting on remote executors: time span where remote executors have been requested, but none were assigned
Longest running test class: name and duration of the longest running test class. Its duration is a lower bound on how much the test task/goal can be accelerated with Test Distribution (see Performance for details).
Total duration of the task/goal: total task/goal execution time
File transfer statistics: capture all files that have been transferred from the build to Develocity and from Test Distribution agents back to the build client. If the requested file is already present in the local file cache of Develocity or all used Test Distribution agents, the input file will not be transferred again and it will not be visible in this section.

Troubleshooting

Identifying the executor of a failed test

Executors starting with localhost- are the local executors. All other executors (in our example containing distribution-agent) are the remote executors.

The executor (and the pool, if applicable) where a test has been run are displayed in the test class view from Build Scans:

Gradle

Gradle’s test progress logging can be configured to show partition and executor information for each event.

build.gradle.kts

import org.gradle.api.tasks.testing.logging.TestLogEvent.*

tasks.test {
    testLogging {
        displayGranularity = 1
        events(PASSED, FAILED, SKIPPED)
    }
    develocity.testDistribution {
        enabled.set(true)
    }
}

build.gradle

import static org.gradle.api.tasks.testing.logging.TestLogEvent.*

tasks.named('test', Test) {
    testLogging {
        displayGranularity = 1
        events(PASSED, FAILED, SKIPPED)
    }
    develocity.testDistribution {
        enabled = true
    }
}

$ ./gradlew test
[…]
> Task :test

Partition 1 in session 1 on localhost-executor-1 > TestClass1 > testMethod() PASSED

Partition 2 in session 2 on prj-distribution-agent-1 > TestClass2 > testMethod() FAILED

Partition 3 in session 3 on prj-distribution-agent-2 > TestClass3 > testMethod() SKIPPED

[…]

Maven

The build log contains information about the test executor used and the partition number for every test class. This information can help if the failure is caused by the test executor, such as a remote agent not providing a required resource.

$ mvn clean test
[…]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestClass1 (of partition 1 on localhost-executor-1)
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 s - in TestClass1 (of partition 1 on localhost-executor-1)
[INFO] Running TestClass2 (of partition 2 on prj-distribution-agent-1)
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 s - in TestClass2 (of partition 2 on prj-distribution-agent-1)
[INFO] Running TestClass3 (of partition 3 on prj-distribution-agent-2)
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 s - in TestClass3 (of partition 3 on prj-distribution-agent-2)
[…]

Moreover, the information is persisted in the generated XML reports.

Classes sharing static state

Each test partition executes in a separate JVM. If tests share state via static variables, intentionally or unintentionally, they may not work reliably when distributed. They are also unlikely to work reliably when not using Test Distribution if using maxParallelForks (for Gradle) or forkCount (for Maven) to execute tests in parallel locally.

If your tests already set forkEvery = 1 (for Gradle) or reuseForks = false (for Maven) to mitigate shared static state, they will likely work reliably with Test Distribution but be slower to execute than they otherwise would be. This setting causes a partition to be created per test class, which creates additional overhead when there are many test classes.

To make optimal use of Test Distribution, each test class should be independent of any other test class and not mutate any shared static state. If that is not possible, consider separating test classes that are self-contained from those that aren’t into separate test tasks and configuring them accordingly.

Tests reading files from `src/test/resources`

Some tests access test resource files in their source location, e.g. in src/test/resources. This location is not an input to the test task and files will not be available when tests are executed remotely. The build tool copies these files, and any other additionally declared test resources, to build/resources/test (for Gradle) or target/test-classes (for Maven) before executing tests. This location is declared as an input, and is part of the test runtime classpath.

One potential solution is to access such files from their location in the build (for Gradle) or target (for Maven) folder instead. However, this does not work in IDEs that don’t delegate test execution to Gradle as they use src/test/resources as an input. If you are unable to use Gradle/Maven for all test execution, a better solution is to access the resource via the runtime classpath via Java’s classpath resource loading mechanism.

Provisioning an external resource for many tests

If you need to prepare/cleanup resources, start/stop a server etc. before/after your tests are executed, you can use one of JUnit Platform’s listener mechanisms. For that purpose, you need to add a dependency on junit-platform-launcher to your project:

build.gradle.kts

plugins {
    java
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation(platform("org.junit:junit-bom:5.10.3"))
    testImplementation("org.junit.jupiter:junit-jupiter") (1)
    testImplementation("org.junit.platform:junit-platform-launcher") (2)
}

tasks.test {
    useJUnitPlatform()
    develocity.testDistribution {
        enabled.set(true)
    }
}

build.gradle

plugins {
    id 'java'
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation platform('org.junit:junit-bom:5.10.3')
    testImplementation 'org.junit.jupiter:junit-jupiter' (1)
    testImplementation 'org.junit.platform:junit-platform-launcher' (2)
}

tasks.named('test', Test) {
    useJUnitPlatform()
    develocity.testDistribution {
        enabled = true
    }
}

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>test-execution-listener</artifactId>
  <version>1.0</version>
  <dependencies>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId> (1)
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId> (2)
      <scope>test</scope>
    </dependency>
  </dependencies>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>5.10.3</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.9.0</version>
        <configuration>
          <release>11</release>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.5.2</version>
        <configuration>
          <properties>
            <distribution>
              <enabled>true</enabled>
            </distribution>
          </properties>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

1	Use JUnit Platform for executing tests
2	Use JUnit Platform Launcher API for implementing a custom listener

JUnit 5.8 and later

With LauncherSessionListener, JUnit 5.8 introduced new hooks that are called before the first and after the last test in a forked test JVM, respectively, and are therefore well suited for implementing once-per-JVM setup/teardown behavior. A custom listener that starts an HTTP server before executing the first test and stops it after the last test has been executed, could look like this:

src/test/java/com/example/GlobalSetupTeardownListener.java

package com.example;

import com.sun.net.httpserver.HttpServer;
import org.junit.platform.launcher.LauncherSession;
import org.junit.platform.launcher.LauncherSessionListener;
import org.junit.platform.launcher.TestExecutionListener;
import org.junit.platform.launcher.TestPlan;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class GlobalSetupTeardownListener implements LauncherSessionListener {

    private Fixture fixture;

    @Override
    public void launcherSessionOpened(LauncherSession session) {
        // Avoid setup for test discovery by delaying it until tests are about to be executed
        session.getLauncher().registerTestExecutionListeners(new TestExecutionListener() {
            @Override
            public void testPlanExecutionStarted(TestPlan testPlan) {
                if (fixture == null) {
                    fixture = new Fixture();
                    fixture.setUp();
                }
            }
        });
    }

    @Override
    public void launcherSessionClosed(LauncherSession session) {
        if (fixture != null) {
            fixture.tearDown();
            fixture = null;
        }
    }

    static class Fixture {

        private HttpServer server;
        private ExecutorService executorService;

        void setUp() {
            try {
                server = HttpServer.create(new InetSocketAddress(0), 0);
            } catch (IOException e) {
                throw new UncheckedIOException("Failed to start HTTP server", e);
            };
            server.createContext("/test", exchange -> {
                exchange.sendResponseHeaders(204, -1);
                exchange.close();
            });
            executorService = Executors.newCachedThreadPool();
            server.setExecutor(executorService);
            server.start(); (1)
            System.setProperty("http.server.port", String.valueOf(server.getAddress().getPort())); (2)
        }

        void tearDown() {
            server.stop(0); (3)
            executorService.shutdownNow();
        }
    }

}

1	Start the HTTP server
2	Export its dynamic port as a system property for consumption by tests
3	Stop the HTTP server

This sample uses the HTTP server implementation from the jdk.httpserver module that comes with the JDK but would work similarly with any other server or resource. In order for the listener to be picked up by JUnit Platform, you need to register it as a service by adding a resource file with the following name and contents to your test runtime classpath (e.g. by adding the file to src/test/resources):

src/test/resources/META-INF/services/org.junit.platform.launcher.LauncherSessionListener

com.example.GlobalSetupTeardownListener

You can now use the resource from your test:

src/test/java/com/example/HttpTests.java

package com.example;

import org.junit.jupiter.api.Test;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

import static org.junit.jupiter.api.Assertions.assertEquals;

class HttpTests {

    @Test
    void respondsWith204() throws Exception {
        var httpClient = HttpClient.newHttpClient();
        var port = System.getProperty("http.server.port"); (1)
        var request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/test")).build();

        var response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); (2)

        assertEquals(204, response.statusCode()); (3)
    }
}

1	Read the port of the server from the system property set by the listener
2	Send a request to the server
3	Check the status code of the response

JUnit 5.7 and earlier

If you’re using a JUnit version prior to 5.8, you can implement a TestExecutionListener to achieve custom setup/teardown behavior.

Implementations of TestExecutionListener are called for every execution request, i.e. potentially multiple times per forked test JVM in the case of retries and when a forked JVM is reused. If possible, you should therefore upgrade to JUnit 5.8 or later and implement a LauncherSessionListener instead.

A custom listener that starts an HTTP server before executing the first test and stops it after the last test has been executed, could look like this:

src/test/java/com/example/CustomTestExecutionListener.java

package com.example;

import com.sun.net.httpserver.HttpServer;
import org.junit.platform.launcher.TestExecutionListener;
import org.junit.platform.launcher.TestPlan;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class CustomTestExecutionListener implements TestExecutionListener {

    private HttpServer server;
    private ExecutorService executorService;

    @Override
    public void testPlanExecutionStarted(TestPlan testPlan) {
        try {
            server = HttpServer.create(new InetSocketAddress(0), 0);
        } catch (IOException e) {
            throw new UncheckedIOException("Failed to start HTTP server", e);
        };
        server.createContext("/test", exchange -> {
            exchange.sendResponseHeaders(204, -1);
            exchange.close();
        });
        executorService = Executors.newCachedThreadPool();
        server.setExecutor(executorService);
        server.start(); (1)
        System.setProperty("http.server.port", String.valueOf(server.getAddress().getPort())); (2)
    }

    @Override
    public void testPlanExecutionFinished(TestPlan testPlan) {
        server.stop(0); (3)
        executorService.shutdownNow();
    }
}

1	Start the HTTP server
2	Export its dynamic port as a system property for consumption by tests
3	Stop the HTTP server

src/test/resources/META-INF/services/org.junit.platform.launcher.TestExecutionListener

com.example.CustomTestExecutionListener

You can now use the resource from your test:

src/test/java/com/example/HttpTests.java

package com.example;

import org.junit.jupiter.api.Test;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

import static org.junit.jupiter.api.Assertions.assertEquals;

class HttpTests {

    @Test
    void respondsWith204() throws Exception {
        var httpClient = HttpClient.newHttpClient();
        var port = System.getProperty("http.server.port"); (1)
        var request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/test")).build();

        var response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); (2)

        assertEquals(204, response.statusCode()); (3)
    }
}

1	Read the port of the server from the system property set by the listener
2	Send a request to the server
3	Check the status code of the response

Accessing JDK internal classes from your tests

When your tests try to access JDK internal classes, and you’re running them with Test Distribution enabled, you might run into exceptions of the following form:

java.lang.IllegalAccessException: module java.base does not open java.lang to unnamed module ...

The execution infrastructure employed by Test Distribution does not automatically open any modules/packages for reflective access. You can fix this issue by configuring the following JVM arguments for your test task or goal.

Gradle

Up until version 7.5, Gradle’s vanilla test executor automatically opened the modules java.util and java.lang for internal reasons. This is why you won’t run into problems when not using Test Distribution on older versions of Gradle.

build.gradle.kts

tasks.test {
    jvmArgs(
    	"--add-opens", "java.base/java.lang=ALL-UNNAMED",
    	"--add-opens", "java.base/java.util=ALL-UNNAMED"
  	)
    develocity.testDistribution {
        enabled.set(true)
    }
}

build.gradle

tasks.named('test', Test) {
    jvmArgs(
    	'--add-opens', 'java.base/java.lang=ALL-UNNAMED',
    	'--add-opens', 'java.base/java.util=ALL-UNNAMED'
  	)
    develocity.testDistribution {
        enabled = true
    }
}

Note that such exceptions can also happen indirectly. For example, using Gradle’s ProjectBuilder test fixture in your tests will indirectly lead to the problem above.

Maven

pom.xml

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.2</version>
  <configuration>
    <argLine>
      --add-opens java.base/java.lang=ALL-UNNAMED
      --add-opens java.base/java.util=ALL-UNNAMED
    </argLine>
    <properties>
      <distribution>
        <enabled>true</enabled>
      </distribution>
    </properties>
  </configuration>
</plugin>

Test JVM fails to start

When running the dockerized version of the Test Distribution on Apple Silicon, it is possible to see the following error:

Test execution failed on remote executor
> Failed to fork test JVM
  > Cannot run program "/opt/java/openjdk/bin/java": error=0, Failed to exec spawn helper: pid: 42, exit value: 1
    > error=0, Failed to exec spawn helper: pid: 42, exit value: 1

This problem only affects the old Test Distribution agent v2.1 (which uses JDK 17 internally) when it is started as a Docker container running on an M1 Apple Silicon. The problem indicates, that the agent could not start the test JVM via the jspawnhelper utility, which is part of the JDK distribution.

To circumvent this problem on an M1 Apple Silicon, you can configure the JDK of the Test Distribution agent to use an older launching mechanism by adding ‑Djdk.lang.Process.launchMechanism=vfork to the Test Distribution Agent JVM options. This configuration should not be used in other environments though. While evaluating Test Distribution, it may be easiest to use the JAR-based Test Distribution agent instead.

Test JVM exiting unexpectedly during test execution

When the JVM that executes tests exits unexpectedly, Test Distribution stops executing any remaining tests, and the test task/goal fails. Any remaining tests are silently skipped. The failure tab in the corresponding Build Scan shows test tasks/goals, where a test JVM exited unexpectedly. The JVM’s exit code is shown as well as the standard output and standard error if they are not empty.

For Gradle builds, this information is also part of the build output when running the build with the option --stacktrace. For Maven builds, option -e (or its long form --errors) ensures that this information is part of the build output.

Logged output of the test JVM

Often, the standard output and error of the test JVM already contain enough information:

> Failure during test discovery: Forked test JVM terminated unexpectedly with exit value 1
  Standard error from JVM:
      Terminating due to fatal error
      org.junit.platform.commons.PreconditionViolationException: Cannot create Launcher without at least one TestEngine; consider adding an engine implementation JAR to the classpath

In the example above, the output indicates that a JUnit test dependency is missing (either org.junit.jupiter:junit-jupiter for JUnit5-based tests or org.junit.vintage:junit-vintage-engine when using the older JUnit4 test framework).

Exit code of the test JVM

But sometimes the exit code of the test JVM can also be helpful. This is especially true in out-of-memory situations, where the remote agent is running in a Kubernetes cluster.

Out-of-memory errors

> Failure during test discovery: Forked test JVM terminated unexpectedly with exit value 137

When the JVM’s exit code is above 128, it typically means that the JVM got killed by the operating system via a signal. The signal number can be deduced by subtracting 128 from the exit value. In the case above this leads to signal number 9, which is also known as the KILL signal.

This is almost always a sign that the test JVM has tried to allocate more memory than the underlying node actually provides. If this happens, the operating system sends a KILL signal to the process that uses the most memory - which is typically the test JVM in this scenario.

Maybe tests are allocating memory which never gets freed by the JVM’s garbage collector. The more tests the test JVM is executing, the more memory is allocated but never freed, until the test JVM gets killed. Or maybe memory resources for the remote agent are simply too low. If this is the case, increase the memory resource limits for the remote agents.

Memory resource limits for the remote agents should be higher than the configured maximum heap space for your test task/goal. Apart from the heap, the JVM requires additional off-heap memory for internal data structures or when doing native I/O. A good rule of thumb is to configure memory resource limits equal to max-heap multiplied by 1.5.

Tests exiting the JVM

> Failure during test discovery: Forked test JVM terminated unexpectedly with exit value 0

When the test JVM exits with exit code 0, this could be an indication that a test is explicitly exiting the JVM via System.exit(0). This should be avoided.

Performance

There are multiple factors that can negatively impact the performance of a build using Test Distribution. The Test Distribution view (see Build Scans) at the task/goal level can provide valuable information to troubleshoot such issues.

Usually, degraded performance is caused by one (or a combination) of the issues below:

Too few executors available to execute the test task/goal: in cases of high contention, compatible remote executors might get assigned to different test tasks/goals or builds. To verify whether this was the case, you can refer to the Execution section in the Test Distribution view. Ideally, Remote executors used should be close to Max remote executors, and average concurrently used executors close to average concurrently requested executors. A small value in Potentially compatible remote executors indicates that only few executors matched the task/goal’s requirements. Starting up additional compatible executors could have a positive impact on performance in such cases.
Suboptimal build configuration settings: performance might be impeded by the values set in maxLocalExecutors/maxRemoteExecutors. In the Test Distribution view, you can double-check the values that have been configured for those settings, and whether average concurrently used executors is approximately equal to average concurrently requested executors.
Slow startup of remote executors: performance can be negatively impacted for example when auto-scaling is used, and remote executors are slow to get online and pick up tests to execute. You can check the time spent waiting on remote executors in the Test Distribution view.
Long running test classes: such classes prevent an optimal distribution of the workload between executors. You can check the duration of the longest test class in the Test Distribution view: it should ideally be much smaller than the total duration of the task/goal. If this is not the case, it can be reduced e.g. by splitting up the long running test class into multiple classes.
Long file transfers to/from Develocity: all well-known and manually declared input files required to execute requested tests have to be transferred to remote Test Distribution agents (and back to the build for output files). File transfer statistics in the Test Distribution view capture all files that have been transferred from the build to Develocity and from Test Distribution agents back to the build client, and can be used to verify the overhead induced by file transfers. These files will be cached in the local file cache of Develocity or the Test Distribution agents, and the overhead should become smaller over time.

Appendix A: Release history

Please refer to the release history of Develocity, the Develocity Test Distribution Agent, and the Develocity Gradle plugin or Develocity Maven extension.

Test Distribution Gradle plugin

The functionality of the Develocity Test Distribution Gradle plugin was merged into the Develocity Gradle plugin as of version 3.11 as part of the Develocity 2022.3 release. For future changes, please see the release history of the Develocity Gradle plugin.