Gradle Enterprise Test Distribution takes your existing test suites and distributes them across remote agents to execute them faster.

How it works

network diagram

Historical test execution data provided by Gradle Enterprise is used to create balanced partitions of similar expected execution times in order to optimize distribution across remote agents. The tests and their supporting files are transferred to each agent and executed, with their logging and results streamed back to the build in real time. If there are no remote agents available, tests can still be executed locally on the build machine.

Gradle Enterprise provides an interface for administering remote agents, and viewing the current and historical system usage. Please see the Gradle Enterprise Admin Manual for more information.

Compatible types of tests

The plugin enhances Gradle’s built-in Test task type and requires tests to run with Java 8 or later.

Tests must run via JUnit Platform, which is part of JUnit 5. All JUnit Platform compatible test engines are supported, such as JUnit Jupiter, jqwik, Kotest, Spek, and others.

JUnit 3 and JUnit 4 based tests are supported when using the JUnit Vintage Engine, including extensions such as Spock.

TestNG is currently not supported.

External test resources

Consideration must be given to any supporting resources required by tests.

Test distribution takes your tests and executes them on remote agents. The test runtime classpath and any additionally declared file inputs are transferred to the agents. If your tests are expecting any other resources to be locally available, such as a running database, the test agents must also provide the resource or the tests will fail.

There are two possible approaches to using such resources in your tests:

  1. Have the tests themselves provision necessary resources (e.g. using Docker containers)

  2. Deploy test distribution agents in environments that provide all required resources

The first approach is recommended where possible as it is more portable and minimizes the need to change agent environments when test resources change. The Testcontainers project provides an easy way to utilize Docker containers within tests.

Depending on how you deploy test distribution agents, additional configuration may be required in order to make Docker accessible to the tests running on those agents. Please refer to the Test Distribution Agent User Manual for more information on this topic.

When using the second approach of having the agent environment provide resources, it is important to declare the requirements of tests so that appropriate agents that provide the capability can be assigned.

Prerequisites

Test distribution requires a Gradle Enterprise 2020.2 or later installation with connected test distribution agents. For information on configuring a Gradle Enterprise installation to enable test distribution, please see the Gradle Enterprise Admin Manual.

Your build must use Gradle 5.4 or later, and use the Gradle Enterprise Plugin to connect to the Gradle Enterprise server. For more information on using the Gradle Enterprise plugin, please see the Gradle Enterprise Gradle Plugin User Manual.

Your build must also authenticate with Gradle Enterprise and the user be granted the “Test distribution” access control role. Unauthenticated usage of test distribution is prohibited due to the inherent security implications of such a remote code execution service.

Configuration

Applying the plugin

The ID of the Gradle Enterprise Test Distribution plugin is com.gradle.enterprise.test-distribution. Version 1.1.1 is the latest version.

build.gradle
plugins {
    id 'com.gradle.enterprise.test-distribution' version '1.1.1'
}
build.gradle.kts
plugins {
    id("com.gradle.enterprise.test-distribution") version "1.1.1"
}

Note that while the Gradle Enterprise plugin is only applied once to your build, the Test Distribution plugin must be applied to each project that contains Test tasks that should be distributed.

Enabling test distribution

Test distribution is enabled and configured per task, via the distribution DSL extension.

build.gradle
tasks.named("test", Test).configure {
    useJUnitPlatform() (1)
    distribution {
        enabled = true (2)
    }
}
build.gradle.kts
tasks.test {
    useJUnitPlatform() (1)
    distribution {
        enabled.set(true) (2)
    }
}
1 Use JUnit Platform for executing tests (see compatible types of tests)
2 Enable test distribution

Executors

Tests can be executed by local and/or remote executors.

By default, the number of local executors is derived from the maxParallelForks property of the task. It is further constrained by the maximum number of workers (defaults to number of CPU processors/cores) configured to be used for your build. To limit the use of local executors, configure the maxLocalExecutors property. Setting it to zero, causes tests to be only executed remotely.

build.gradle
tasks.named("test", Test).configure {
    distribution {
        enabled = true
        maxLocalExecutors = 2
    }
}
build.gradle.kts
tasks.test {
    distribution {
        enabled.set(true)
        maxLocalExecutors.set(2)
    }
}

By default, test execution uses as many remote executors as possible. To limit the number used, you can configure the maxRemoteExecutors property:

build.gradle
tasks.named("test", Test).configure {
    distribution {
        enabled = true
        maxRemoteExecutors = 2
    }
}
build.gradle.kts
tasks.test {
    distribution {
        enabled.set(true)
        maxRemoteExecutors.set(2)
    }
}

The number of remote executors actually used is subject to agent availability and the number of test partitions.

Setting the number of remote executors to zero causes tests to be only executed locally.

When first enabling test distribution for an existing test suite, it is a good idea to start with only local executors to verify compatibility of the tests with test distribution.
When using Gradle’s offline mode, only local executors will be used.

Requirements

Test tasks can specify requirements that agents must fulfill in order to be considered compatible. Requirements are matched against the advertised capabilities of the agents that are connected to your Gradle Enterprise server. Only agents that fulfill all requirements are considered compatible. Local executors are assumed to provide all required capabilities. This means that local executors will always be used regardless of the declared requirements, unless use of local executors is disabled.

Requirements are strings that may only contain alphanumeric characters, dashes, underscores, periods, and a single equals sign. It is recommended to use a key-value form (e.g. os=linux) or tag-like form (e.g. postgres) to model requirements.

build.gradle
tasks.named("test", Test).configure {
    distribution {
        enabled = true
        requirements = ["os=linux", "postgres"]
    }
}
build.gradle.kts
tasks.test {
    distribution {
        enabled.set(true)
        requirements.set(setOf("os=linux", "postgres"))
    }
}

To ensure tests are executed using the intended Java version, an implicit jdk=«version» requirement is added to the set of declared requirements for the task. The version is determined from the Java executable set on the task. Only the major version is taken into account (e.g. jdk=8 for Java 8 and jdk=11 for Java 11).

For more information on how to define the capabilities of an agent, please see the Test Distribution Agent User Manual.

Other task properties

In addition to the above configuration properties of the distribution DSL extension, the Test Distribution plugin respects the Test task’s configuration. The following properties influence test partitioning and the type and number of used local and remote executors.

maxParallelForks

Used as default number of local executors.

debug (--debug-jvm command line option)

If enabled, tests are executed on a single local executor that you can attach a debugger to as usual.

forkEvery

If set, the number is used to limit the maximum size of each test partition.

jvmArgs, minHeapSize, maxHeapSize, systemProperties

Used locally and on agents when forking JVMs to execute tests. Paths in these JVM arguments are automatically replaced to match the agent’s temporary workspace.

The following properties are currently not supported:

  • failFast (--fail-fast command line option)

  • modularity (incubating since Gradle 6.4)

Output files

Tests may produce file outputs, such as reports. After finishing execution of a test partition, agents send all declared output files and directories back to the build.

Output directories are merged by copying them to the same destination directory. If two partitions produce the same output file, the task will fail. You must create unique files per test class.

Integration with other plugins

Test Retry plugin

Test distribution is compatible with the Test Retry plugin version 1.1.4 or above. To configure retries of failed tests, you can use the retry DSL extension provided by the Test Retry plugin:

build.gradle
plugins {
    id 'java'
    id 'com.gradle.enterprise.test-distribution' version '1.1.1' (1)
    id 'org.gradle.test-retry' version '1.1.6' (2)
}
tasks.named("test", Test).configure {
    useJUnitPlatform()
    distribution {
        enabled = true
    }
    retry {
        maxRetries = 3
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}
build.gradle.kts
plugins {
    java
    id("com.gradle.enterprise.test-distribution") version "1.1.1" (1)
    id("org.gradle.test-retry") version "1.1.6" (2)
}
tasks.test {
    useJUnitPlatform()
    distribution {
        enabled.set(true)
    }
    retry {
        maxRetries.set(3)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}
1 Apply the Test Distribution plugin
2 Apply the Test Retry plugin

Please note that retry semantics change slightly: both maxRetries and maxFailures are applied for each test partition instead of the entire test task. Moreover, while the Test Retry plugin forks a new JVM for each retry, the Test Distribution plugin retries tests in the same JVM in which they originally failed. The granularity might also differ slightly in some cases, e.g. the Test Distribution plugin retries individual invocations of parameterized JUnit Jupiter tests but retries the whole test class for data-driven Spock tests due to a limitation of the JUnit Vintage Engine.

JaCoCo plugin

Test distribution is compatible with Gradle’s JaCoCo plugin. Coverage information from all executors, including remote, are merged to produce a single coverage report. No additional configuration is required.

Gradle plugin development plugin

Running functional tests using the GradleRunner is supported out of the box. No additional configuration is required.

Dealing with common problems

Identifying the executor of a failed test

When a test has failed, the first line of its logging output contains information about the test executor used and the partition number. This information can help if the failure is caused by the test executor, such as a remote agent not providing a required resource. Any other output the test produces follows this information.

This can be seen everywhere test logging output is displayed, including build scans.

build scan failed test

Gradle’s test progress logging can also be configured to show partition and executor information for each event.

build.gradle
import static org.gradle.api.tasks.testing.logging.TestLogEvent.*

tasks.named("test", Test).configure {
    testLogging {
        displayGranularity = 0
        events(PASSED, FAILED, SKIPPED)
    }
    distribution {
        enabled = true
    }
}
build.gradle.kts
import org.gradle.api.tasks.testing.logging.TestLogEvent.*

tasks.test {
    testLogging {
        displayGranularity = 0
        events(PASSED, FAILED, SKIPPED)
    }
    distribution {
        enabled.set(true)
    }
}
$ ./gradlew test
[…]
> Task :test

Distributed Test Run :test > Partition 1 on localhost-executor-1 > TestClass1 > testMethod() PASSED

Distributed Test Run :test > Partition 2 on prj-distribution-agent-1 > TestClass2 > testMethod() FAILED

Distributed Test Run :test > Partition 3 on prj-distribution-agent-2 > TestClass3 > testMethod() SKIPPED

[…]

Executors starting with localhost- are the local executors. All other executors (in our example containing distribution-agent) are the remote executors.

Classes sharing static state

Each test partition executes in a separate JVM. If tests share state via static variables, intentionally or unintentionally, they may not work reliably when distributed. They are also unlikely to work reliably when not using test distribution if using maxParallelForks to execute tests in parallel locally.

If your tests already set forkEvery = 1 to mitigate shared static state, they will likely work reliably with test distribution but be slower to execute than they otherwise would be. This setting causes a partition to be created per test class, which creates additional overhead when there are many test classes.

To make optimal use of test distribution, each test class should be independent of any other test class and not mutate any shared static state. If that is not possible, consider separating test classes that are self-contained from those that aren’t into separate test tasks and configuring them accordingly.

Tests reading files from src/test/resources

Some tests access test resource files in their source location, e.g. in src/test/resources. This location is not an input to the test task and files will not be available when tests are executed remotely. Gradle copies these files, and any other additionally declared test resources, to build/resources/test before executing tests. This location is declared as an input, and is part of the test runtime classpath.

One potential solution is to access such files from their location in the build folder instead. However, this does not work in IDEs that don’t delegate test execution to Gradle as they use src/test/resources as an input. If you are unable to use Gradle for all test execution, a better solution is to access the resource via the runtime classpath via Java’s classpath resource loading mechanism.

Provisioning an external resource for many tests

If you need to prepare/cleanup resources, start/stop a server etc. before/after your tests are executed, you can implement a JUnit Platform TestExecutionListener.

build.gradle
plugins {
    id 'java'
    id 'com.gradle.enterprise.test-distribution' version '1.1.1'
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation platform("org.junit:junit-bom:5.6.2")
    testImplementation "org.junit.jupiter:junit-jupiter" (1)
    testImplementation "org.junit.platform:junit-platform-launcher" (2)
}

tasks.named("test", Test).configure {
    useJUnitPlatform()
    distribution {
        enabled = true
    }
}
build.gradle.kts
plugins {
    java
    id("com.gradle.enterprise.test-distribution") version "1.1.1"
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation(platform("org.junit:junit-bom:5.6.2"))
    testImplementation("org.junit.jupiter:junit-jupiter") (1)
    testImplementation("org.junit.platform:junit-platform-launcher") (2)
}

tasks.test {
    useJUnitPlatform()
    distribution {
        enabled.set(true)
    }
}
1 Use JUnit Platform for executing tests
2 Use JUnit Platform Launcher API for implementing a custom TestExecutionListener

A custom listener that starts an HTTP server before executing the first test and stops it after the last test has been executed, could look like this:

CustomTestExecutionListener.java
package com.example;

import com.sun.net.httpserver.HttpServer;
import org.junit.platform.launcher.TestExecutionListener;
import org.junit.platform.launcher.TestPlan;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class CustomTestExecutionListener implements TestExecutionListener {

    private HttpServer server;
    private ExecutorService executorService;

    @Override
    public void testPlanExecutionStarted(TestPlan testPlan) {
        try {
            server = HttpServer.create(new InetSocketAddress(0), 0);
        } catch (IOException e) {
            throw new UncheckedIOException("Failed to start HTTP server", e);
        };
        server.createContext("/test", exchange -> {
            exchange.sendResponseHeaders(204, -1);
            exchange.close();
        });
        executorService = Executors.newCachedThreadPool();
        server.setExecutor(executorService);
        server.start(); (1)
        System.setProperty("http.server.port", String.valueOf(server.getAddress().getPort())); (2)
    }

    @Override
    public void testPlanExecutionFinished(TestPlan testPlan) {
        server.stop(0); (3)
        executorService.shutdownNow();
    }
}
1 Start the HTTP server
2 Export its dynamic port as a system property for consumption by tests
3 Stop the HTTP server
If retries are enabled, the listener is additionally called once per retried set of tests.

This sample uses the HTTP server implementation from the jdk.httpserver module that comes with the JDK but would work similarly with any other server or resource. In order for the listener to be picked up by JUnit Platform, you need to register it as a service by adding a resource file with the following name and contents to your test runtime classpath (e.g. by adding the file to src/test/resources):

META-INF/services/org.junit.platform.launcher.TestExecutionListener
com.example.CustomTestExecutionListener

You can now use the resource from your test:

HttpTests.java
package com.example;

import org.junit.jupiter.api.Test;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

import static org.junit.jupiter.api.Assertions.assertEquals;

class HttpTests {

    @Test
    void respondsWith204() throws Exception {
        var httpClient = HttpClient.newHttpClient();
        var port = System.getProperty("http.server.port"); (1)
        var request = HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/test")).build();

        var response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); (2)

        assertEquals(204, response.statusCode()); (3)
    }
}
1 Read the port of the server from the system property set by the listener
2 Send a request to the server
3 Check the status code of the response

Appendix A: API reference

Please see the Javadoc.

Appendix B: Release history

1.1.1 - 29th July 2020

  • Performance improvements for parallel multi-project builds that use local and remote executors

1.1 - 27th July 2020

  • Input file uploads are multiplexed and deduplicated for improved performance

  • Plugin no longer adds custom values and tags to build scans

  • Compatibility with Gradle 6.7

  • Test executor failures from agents are reported correctly to build scans

  • Tests without corresponding test methods (e.g. for JUnit 4 test runners like Exemplar) can now be filtered

  • Test classes nested within top-level classes that use the Enclosed runner are executed only once

1.0.2 - 5th June 2020

  • The Gradle Enterprise Test Distribution Plugin jar is now digitally signed. Check the documentation’s appendix section on how to verify the signature

  • Long classpaths are supported on Windows when running on JDK 8

  • Handling of unexpected test events is improved

1.0.1 - 11th May 2020

  • Java agent JVM arguments without options are now parsed correctly

  • Java agent JVM arguments with relative path to Java agent are now supported

  • Test output before/after the execution of a test class is now handled correctly

  • Race condition when capturing test output from different threads is fixed

  • Timeout for connecting to forked test VMs is increased to match Gradle’s

  • Add option to suppress capturing custom values in build scans

1.0 - 5th May 2020

  • Initial release

Appendix C: Compatibility with Gradle Build Tool and Gradle Enterprise

Compatibility between versions of Gradle, Gradle Enterprise and the Gradle Enterprise test distribution Gradle plugin can be found here.

Appendix D: Verifying the signature of the plugin jar

(plugin 1.0.2+)

The plugin jar is published to plugins.gradle.org alongside its signature. The public key is published to http://pool.sks-keyservers.net and https://keys.openpgp.org. You can verify the signature as follows:

curl -OL https://plugins.gradle.org/m2/com/gradle/enterprise/test-distribution-gradle-plugin/1.1.1/test-distribution-gradle-plugin-1.1.1.jar
curl -OL https://plugins.gradle.org/m2/com/gradle/enterprise/test-distribution-gradle-plugin/1.1.1/test-distribution-gradle-plugin-1.1.1.jar.asc
gpg --keyserver keys.openpgp.org --recv-key 314FE82E5A4C5377BCA2EDEC5208812E1E4A6DB0
gpg --verify test-distribution-gradle-plugin-1.1.1.jar.asc test-distribution-gradle-plugin-1.1.1.jar

The output of the last command should look similar to the following:

gpg: Signature made Tue May  5 08:36:01 2020 UTC
gpg:                using RSA key 5208812E1E4A6DB0
gpg: Good signature from "Gradle Inc. <info@gradle.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 314F E82E 5A4C 5377 BCA2  EDEC 5208 812E 1E4A 6DB0

This verifies that the artifact was signed with the private key that corresponds to the imported public key. The warning is emitted because you haven’t explicitly trusted the imported key (hence [unknown]). One way of establishing trust is to verify the fingerprint over a secure channel. Please contact technical support should you wish to do so.