Develocity Predictive Test Selection increases developer productivity by automatically and intelligently selecting and executing the subset of tests that are most relevant to a code change, providing faster feedback. It is available for Gradle and Apache Maven™ builds and is complementary to other Develocity build acceleration features such as Build Caching and Test Distribution.

The technique of avoiding irrelevant tests in order to provide faster feedback to developers was popularized by Meta and other advanced software organizations. Develocity Predictive Test Selection was inspired by their work, and makes the benefits of this technique available to all Develocity users.

Develocity Predictive Test Selection is an extension and may not be enabled for your installation. Please contact your customer representative to enable this extension.

How it works

Before executing a test suite, Predictive-Test-Selection-enabled builds ask Develocity to predict which of the tests-to-be-executed will provide useful feedback on the changes under test, selecting only those tests for execution. The Build Scan® indicates which tests were selected and which weren’t along with the reason why. It also indicates how much time was saved by skipping the not selected tests.

The predictive model

Develocity develops a predictive model by observing code changes and test outcomes from your Build Scan data. It combines ongoing learning about your project structure and tests with training from millions of test executions across many projects in order to make accurate predictions.

A snapshot of the test suite and the code-under-test is cross-referenced with change and test result history to predict the subset of relevant tests, which is typically a small portion of the entire test suite.

how predictive test selection works

As each new Build Scan is uploaded to Develocity, the predictive model is updated based on the new results.

Selection profiles

(Develocity 2023.2+)

The selection profile influences the number of tests that will be selected for execution by Predictive Test Selection. Develocity offers three predefined selection profiles: Conservative, Standard, and Fast. The selection profile can be configured for each Gradle test task or Maven test goal.

The Standard profile is the default and provides the same experience as previous versions of Develocity, where the selection profile was not configurable. It aims to strike a good balance between saving time and selecting relevant tests.

The Conservative profile favors increasing the likelihood of selecting all tests that provide useful feedback, which often requires selecting more tests, which reduces time savings compared to the Standard profile.

The Fast profile favors saving time by using a higher selection threshold, which often selects fewer tests and saves more time than the Standard profile.

The Predictive Test Selection Simulator shows the simulated results of the different selection profiles, making it easy to assess the impact of the different profiles.

Flaky test handling

As flaky tests are inherently unpredictable, Predictive Test Selection will always select recently flaky tests.

To maximize the effectiveness of Predictive Test Selection, it is recommended that Develocity Test Analytics be used to identify and prioritize the stabilization of flaky tests in your project.

Relationship with other acceleration techniques and technologies

Build Cache

Predictive Test Selection is complementary to Develocity Build Cache.

When Predictive Test Selection is enabled, test tasks/goals will store their results in the build cache if build caching is enabled, all tests are selected for execution, and the task/goal was successful. The results from running only a subset of all tests or failing tests are never stored in the build cache. Test tasks/goals will aim to reuse the test results from a prior build invocation if the build cache is enabled, regardless of whether Predictive Test Selection is currently enabled or not (requires Gradle 8.2+ and Develocity Gradle plugin 3.13.1+ or Develocity Maven extension 1.16+).

On a cache miss, rerunning the same test suite for the same code-under-test with Predictive Test Selection enabled, previously selected tests that passed will not be selected again. Instead, Predictive Test Selection will select different tests, and typically fewer, for subsequent executions.

Before Gradle 8.2 and Develocity Gradle plugin 3.13.1, enabling Predictive Test Selection on a test task disabled build caching for that task. Prior to Develocity Maven extension 1.16, enabling Predictive Test Selection on a test goal disabled writing to the Build Cache for that goal even if all tests were selected.

Test Distribution

Predictive Test Selection is complementary to Develocity Test Distribution. The tests selected for execution can be accelerated by distribution, further increasing build speed.

Flaky test retry

Enabling flaky test retry functionality is encouraged as it allows test selection predictions to better account for non-deterministic flaky test outcomes. This is especially problematic for long-running tests such as integration tests and end-to-end tests.

Predictive Test Selection is compatible with the integrated test retry functionality of the Develocity Gradle plugin version 3.12 or above. To configure retries of failed tests, you can use the retry DSL extension provided by the Develocity Gradle plugin:

build.gradle.kts
import com.gradle.enterprise.gradleplugin.testretry.retry (1)

tasks.test {
    useJUnitPlatform()
    distribution {
        enabled.set(true)
        retryInSameJvm.set(true)
    }
    retry {
        maxRetries.set(3)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}
build.gradle
tasks.named('test', Test) {
    useJUnitPlatform()
    distribution {
        enabled = true
        retryInSameJvm = true
    }
    retry {
        maxRetries = 3
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}
1 For a Kotlin-based build script, you have to include the retry DSL extension provided by the Develocity Gradle plugin.

When using an older version of the Develocity Gradle plugin, you can use the test retry functionality of the test retry plugin version 1.1.4 or above. To configure retries of failed tests, you can use the retry DSL extension provided by the test retry plugin:

build.gradle.kts
plugins {
    id("org.gradle.test-retry") version "1.5.8" (1)
}
tasks.test {
    useJUnitPlatform()
    distribution {
        enabled.set(true)
        retryInSameJvm.set(true)
    }
    retry {
        maxRetries.set(3)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}
build.gradle
plugins {
    id('org.gradle.test-retry') version '1.5.8' (1)
}
tasks.named('test', Test) {
    useJUnitPlatform()
    distribution {
        enabled = true
        retryInSameJvm = true
    }
    retry {
        maxRetries = 3
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}
1 Apply the test retry plugin

Static analysis and code coverage based alternatives

Other test selection tools are available that employ static analysis or historical code coverage information to select tests. Develocity Predictive Test Selection uses a more intelligent selection process that takes more factors into account, such as the historical relationship between a test and the code-under-test. This is particularly significant for long-running tests such as integration tests and end-to-end tests where a large amount of code-under-test is typically executed during a test.

When to use Predictive Test Selection

Predictive Test Selection intelligently trades testing comprehensiveness for faster feedback, making it worthwhile for many test executions where reducing feedback time is critical, such as local and pre-merge/pull-request builds.

To benefit from the faster feedback offered by Predictive Test Selection during development while still ensuring that changesets are tested comprehensively before releasing, additional test executions that run remaining tests or do not enable Predictive Test Selection should occur later in the change lifecycle. For example, as part of post-merge, nightly or ready-for-release builds.

Relevant vs. remaining tests

(Develocity 2023.2+)

By default, Predictive Test Selection enabled builds will select the most relevant subset of tests for execution for the given changeset and prior test execution/change history to provide the fastest feedback possible. However, it’s advisable to run the remaining tests in a subsequent stage of the CI pipeline to ensure that the changeset is tested comprehensively. Therefore, the Develocity Gradle plugin and Maven extension provide a “remaining tests” selection mode (see instructions below for Gradle and Maven). A typical usage pattern is to use the “relevant tests” selection mode for early and frequent QA stages, such as local pre-commit checks and pre-merge CI jobs, and the “remaining tests” selection mode for later and less frequent extended coverage QA stages, such as nightly or pre-release builds, to ensure that all tests are ultimately executed. This avoids executing any given test more than once for a given change.

ci pipeline

Shift-left integration testing

An essential aspect of modern software development is testing early and often during the development process instead of at the end, known as “shift-left testing”. However, a common approach is to have fast tests, such as unit tests, execute early in the lifecycle of a change and defer all long-running tests such as integration or end-to-end tests. Despite such tests being very good at detecting defects, they are often delayed until later in the development process due to being slower than unit tests. Predictive Test Selection allows a more productive approach where the most valuable subset of integration and end-to-end tests are also “shifted left”.

By executing the most relevant subset of long-running integration and end-to-end tests run early during the lifecycle of a change, Predictive Test Selection gives developers the high-quality feedback such tests provide quickly.

Test compatibility

Frameworks and languages

Tests must be run via JUnit Platform, which is part of JUnit 5. Many popular test frameworks are compatible with JUnit Platform and are supported by Predictive Test Selection, such as JUnit (including JUnit 3 and JUnit 4), Spock, Kotest (version 5.6.0 and later), TestNG, jqwik, ArchUnit and more. Cucumber tests are also supported, but they do require some extra configuration via the Cucumber-Companion plugin. Please see Gradle Testing in Java & JVM projects guide or Maven Surefire’s documentation for guidance on configuring your build to use JUnit Platform. When in doubt, please contact Develocity support to assess whether your setup is compatible with Predictive Test Selection.

Tests and sources authored in popular JVM languages are supported, including Java, Kotlin, Groovy, and Scala.

Some JUnit Platform test engines, such as Spek, are currently not supported. Please contact Develocity support if you are interested in using this or other incompatible test engines with Predictive Test Selection.

Code coverage tools

When Predictive Test Selection is enabled, only a subset of tests may be executed, resulting in a lower code coverage than usual. If you rely on code coverage measurement, we recommend enabling it only for builds with Predictive Test Selection disabled to get consistent values.

Gradle

Develocity Predictive Test Selection is compatible with Gradle 5.4 and later, and works with Gradle’s built-in Test task and its subclasses. This includes Android unit test tasks, but does not include most Android device tests.

Tests must be run with Java 8 or later.

Tests from the buildSrc project and from included builds are currently not supported.

Maven

Develocity Predictive Test Selection is compatible with the Maven Surefire plugin 2.22.2 and later, and Maven Failsafe plugin 2.22.2 and later.

Tests must be run with Java 8 or later.

The Predictive Test Selection Simulator

Develocity provides a Predictive Test Selection Simulator that facilitates making an informed decision about which projects and builds should enable Predictive Test Selection.

predictive test selection simulator

The simulator works by comparing actual test results from Build Scans to what would have happened if the build used Predictive Test Selection, visualizing the following:

  • % Task/Goal failures predicted— Predicted test tasks/goals with failures versus all test tasks/goals with failures

  • % Test failures predicted— Predicted test failures versus all test failures

  • Savings potential— Time taken to run selected tests vs time taken to run all tests

  • Avoidable tests— Count of test classes which would have been “NOT SELECTED” versus all test classes

  • Unavoidable tests— Count of test class executions which would have been "SELECTED" because considered either relevant or likely to fail

Simulation results are available for each test task or goal execution, and can be filtered using the filtering criteria atop the dashboard.

Moreover, the simulator can be used to compare different selection profiles by switching between them using the toggle in the top right.

Prerequisites

The Predictive Test Selection Simulator requires Develocity 2022.1 or later, and the Develocity Predictive Test Selection extension. Please contact your customer representative to enable this extension.

It requires that builds are configured to capture task/goal input files as part of Build Scans. For Gradle, see Capturing task input files for Gradle. For Maven, see Capturing goal input files for Maven.

Interpreting simulation results

The overview metrics and charts help you quickly estimate the expected benefits and risks of enabling Predictive Test Selection for selected builds.

Measuring expected risk

The task/goal failures predicted chart indicates the likelihood that Predictive Test Selection will miss a breaking change. It shows test failures per day to highlight trends and outliers. For example, you may find that there were many failures on a particular day and that test selection had more incorrect predictions that same day. This may indicate some kind of test infrastructure problem which caused unpredictable failures for a short time.

“Correct” predictions would have caught a breaking change, while “incorrect” predictions would not have. However, it is possible for flaky test failures to be misclassified as “incorrect” predictions.

Flaky test failures detected by Develocity are excluded from these predicted failures because they do not reliably represent faults in production code. Some flaky test failures, especially infrastructure failures, may not be detected as flaky by Develocity, and will be classified as “incorrect”.

Analyzing possibly incorrect predictions

Tests are intended to indicate correctness of production code to developers and not the stability of the infrastructure that runs the tests themselves. Unfortunately, there exists no test environment which is completely reliable, and sometimes tests fail due to infrastructure problems.

Predictive Test Selection attempts to detect flaky test failures and omit them from simulation results.

In order to obtain an accurate estimate of expected Predictive Test Selection quality, it is recommended to inspect incorrect predictions and omit simulations (via search filters) for test builds in which all test failures were caused by test flakiness or test environment problems not relating to the code.

Misconfigured or incomplete test task/goal inputs is another common reason for incorrect predictions. Predictive Test Selection needs to know about all additional test inputs impacting the outcome of the test in order to make accurate predictions.

Measuring expected benefit

The savings potential chart indicates the amount and ratio of serial test time that could be saved if Predictive Test Selection had been applied to all matching builds.

The system is calibrated to catch over 99% of non-flaky test task/goal failures for many different types of projects and changesets, which allows it to adapt to unforeseen situations as your project evolves. This implies that benefits are not uniformly distributed: riskier code changes require more testing to maintain high confidence.

The avoidable tests chart indicates the number and ratio of test executions which would have been skipped. This estimates the potential for future Predictive Test Selection improvements.

Enabling Predictive Test Selection

Prerequisites

Predictive Test Selection requires Develocity 2022.2 or later, and the Develocity Predictive Test Selection extension. Please contact your customer representative to enable this extension.

After enabling the extension, recent build data must be processed to construct the prediction model. Depending on the amount of build data available, this process may take up to 3 days. For most installations, this processing is able to complete overnight.

The “Use Predictive Test Selection” permission is required to use Predictive Test Selection in a build. If anonymous access is disabled for viewing build scans, builds must authenticate with Develocity via an access key that has the "Use Predictive Test Selection" permission enabled. See below for build tool-specific instructions.

Builds must be configured to publish Build Scan data with task/goal input files capturing enabled. See below for build tool-specific instructions.

Gradle

Applying the Gradle plugin

Develocity 2022.3 and later

Starting with Develocity 2022.3, the Develocity Gradle plugin is the only Gradle plugin that needs to be applied to your build as a prerequisite of using Predictive Test Selection. Please refer to the Develocity Gradle Plugin User Manual for detailed information on how to do this or see below for the short version.

Gradle 6.0 and above

The plugin must be applied in the settings file of the build.

settings.gradle.kts
plugins {
    id("com.gradle.enterprise") version "3.16.2"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}
settings.gradle
plugins {
    id('com.gradle.enterprise') version '3.16.2'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}
Gradle 5.4 to 5.6.x

The plugin must be applied to the root project of the build.

build.gradle.kts
plugins {
    id("com.gradle.build-scan") version "3.16.2"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}
build.gradle
plugins {
    id('com.gradle.build-scan') version '3.16.2'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}
Develocity 2022.2.x and earlier

Earlier versions of Develocity require applying the Develocity Test Distribution Gradle plugin in addition to the Develocity Gradle plugin as a prerequisite of using Predictive Test Selection. Its plugin ID is com.gradle.enterprise.test-distribution (last version: 2.3.5).

Gradle 6.0 and above

The plugin must be applied in the settings file of the build in addition to the com.gradle.enterprise plugin.

settings.gradle.kts
plugins {
    id("com.gradle.enterprise") version "3.10.3"
    id("com.gradle.enterprise.test-distribution") version "2.3.5"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}
settings.gradle
plugins {
    id('com.gradle.enterprise') version '3.10.3'
    id('com.gradle.enterprise.test-distribution') version '2.3.5'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}
Gradle 5.4 to 5.6.x

The plugin must be applied to the root project of the build in addition to the com.gradle.build-scan plugin.

build.gradle.kts
plugins {
    id("com.gradle.build-scan") version "3.10.3"
    id("com.gradle.enterprise.test-distribution") version "2.3.5"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}
build.gradle
plugins {
    id('com.gradle.build-scan') version '3.10.3'
    id('com.gradle.enterprise.test-distribution') version '2.3.5'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}

Authentication

If anonymous access is disabled for viewing build scans, builds must authenticate with Develocity via an access key that has the "Use Predictive Test Selection" permission enabled to use Predictive Test Selection. Please see Authenticating Gradle builds with Develocity for how to do this.

Capturing task input files

Builds must be configured to capture task input files in order to use Predictive Test Selection. Please see Capturing task input files for Gradle for instructions.

Enabling

Predictive Test Selection is enabled and configured per Test task, via the predictiveSelection DSL extension.

Conditionally enabling

You can enable PTS using the built-in property or a custom property.

Using the built-in property

(Gradle plugin 3.15+)

Instead of using a static boolean value in the DSL extension, it is possible to toggle test selection dynamically via the pts.enabled system property:

$ ./gradlew test -Dpts.enabled=true
Using a custom property

Alternatively, test selection can be toggled dynamically via project properties specified at the time of build invocation.

The build script example below reads the enablePTS project property to determine if test selection should be enabled for execution of the test task.

build.gradle.kts
tasks.test {
    useJUnitPlatform() (1)
    predictiveSelection {
        enabled.set(providers.gradleProperty("enablePTS").map { it != "false" }.orElse(false)) (2)
    }
}
build.gradle
tasks.named('test', Test) {
    useJUnitPlatform() (1)
    predictiveSelection {
        enabled = providers.gradleProperty('enablePTS').map { it != 'false' }.orElse(false) (2)
    }
}
1 Use JUnit Platform for executing tests (see Test compatibility)
2 Enable Predictive Test Selection

The project property is specified as a system property at runtime.

$ ./gradlew test -PenablePTS

The project property can also be specified via an environment variable:

ORG_GRADLE_PROJECT_enablePTS=true

For more information on project properties, refer to the Gradle documentation.

Declaring additional test inputs

For optimal prediction accuracy it is vital that any additional file inputs to the test task are declared.

Additional inputs can be declared via the build script by configuring the test task.

build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
    }
    inputs.file("test-fixtures/data/accounts.csv").withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.file("randomized-test-data").withPathSensitivity(PathSensitivity.RELATIVE)
}
build.gradle
tasks.named('test', Test) {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
    }
    inputs.file('test-fixtures/data/accounts.csv').withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.file('randomized-test-data').withPathSensitivity(PathSensitivity.RELATIVE)
}

Refer to the Gradle build tool Authoring tasks guide for more information about input declaration.

Configuring must-run tests

(Gradle plugin 3.10+, Develocity 2022.2+)

It is possible to configure the plugin to always select tests for execution based on custom criteria.

This can be achieved in two ways: by specifying a class name filter pattern, via the includeClasses configuration property, or in an annotation-driven manner, via the includeAnnotationClasses configuration property. Both configuration properties can be applied simultaneously.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

Specifying tests to run on the command line via --tests …​ has the same effect as including them in must-run configuration. If you wish test selection to avoid tests for filtered test sets, follow instructions for enabling Predictive Test Selection for filtered test sets.

From version 3.12 onward, you can also use the com.gradle.enterprise.testing.annotations.MustRun annotation from the gradle-enterprise-testing-annotations project without any additional configuration other than the dependency.

Must-run class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
        mustRun {
            includeClasses.addAll("com.project.FileIntegrityTest", "*.project.*SanityTest")
        }
    }
}
build.gradle
tasks.named('test', Test) {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
        mustRun {
            includeClasses.addAll('com.project.FileIntegrityTest', '*.project.*SanityTest')
        }
    }
}
Must-run annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class’ annotations. A class need only have one annotation matching any of the patterns to be included.

build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
        mustRun {
            includeAnnotationClasses.add("com.project.annotations.MustRun")
        }
    }
}
build.gradle
tasks.named('test', Test) {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
        mustRun {
            includeAnnotationClasses.add('com.project.annotations.MustRun')
        }
    }
}

Refer to the Example must-run annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Selection mode

(Gradle plugin 3.14+, Develocity 2023.2+)

The selection mode determines how tests are selected for execution (see Relevant vs. remaining tests for details). By default, the most relevant tests according to the used selection profile are selected in order to provide fast feedback. To run the remaining tests, e.g. post-merge, the selection mode can be changed accordingly.

In the “relevant tests” selection mode, all test classes that have not yet had a successful or flaky outcome for the same inputs will be selected. Test classes configured as must-run will only be selected if there was no prior execution with a successful or flaky outcome for the same inputs.

Using the pts.mode system property is the recommended way to change the selection mode:

$ ./gradlew test -Dpts.mode=REMAINING_TESTS

Alternatively, the selection mode can be changed via the mode property in the build script.

build.gradle.kts
import com.gradle.enterprise.gradleplugin.testselection.PredictiveTestSelectionMode.REMAINING_TESTS

tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        mode.set(REMAINING_TESTS) // default: RELEVANT_TESTS
    }
}
build.gradle
import static com.gradle.enterprise.gradleplugin.testselection.PredictiveTestSelectionMode.REMAINING_TESTS

tasks.named('test', Test) {
    useJUnitPlatform()
    predictiveSelection {
        mode = REMAINING_TESTS // default: RELEVANT_TESTS
    }
}
  • Gradle plugin 3.15+: the mode configured via the system property takes precedence over the configuration in the build script.

  • Gradle plugin 3.14.1 and earlier: the mode configured in the build script takes precedence over the system property.

Selection profile

(Gradle plugin 3.14+, Develocity 2023.2+)

The selection profile allows adjusting the desired level of trade-off between the PTS system’s confidence (i.e. percentage of correctly predicted failures) vs. savings (i.e. percentage of testing time avoided). You can choose between the values CONSERVATIVE, STANDARD, and FAST. Setting the selection profile is optional; if none is configured, STANDARD will be used.

build.gradle.kts
import com.gradle.enterprise.gradleplugin.testselection.PredictiveTestSelectionProfile.FAST

tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        profile.set(FAST) // default: STANDARD
    }
}
build.gradle
import static com.gradle.enterprise.gradleplugin.testselection.PredictiveTestSelectionProfile.FAST

tasks.named('test', Test) {
    useJUnitPlatform()
    predictiveSelection {
        profile = FAST // default: STANDARD
    }
}

The selection profile can alternatively be passed as system property:

$ ./gradlew test -Dpts.profile=FAST
The profile configured via the system property takes precedence over what is configured in the build script.

Test dry run mode

(Gradle plugin 3.14.1+)

To simulate the execution of tests without actually running them, Develocity provides a test dry run mode. This mode generates reports, and tests are selected by Predictive Test Selection. This can e.g. be used to verify that your test filtering configuration or Predictive Test Selection configuration is correct, without running tests.

The test dry run mode can be enabled by:

  • Adding the --test-dry-run command line option or setting the dryRun test task property to true (Gradle 8.3+)

  • Setting the junit.platform.execution.dryRun.enabled system property to true (JUnit 5.10+)

Fallback to regular execution

(Gradle plugin 3.16+)

Predictive Test Selection relies on JUnit Platform to run tests. By default, the test task will fail if it isn’t configured to use JUnit Platform. To avoid that, e.g. when centrally enabling Predictive Test Selection, fallbackToRegularExecutionOnMissingJUnitPlatform can be set to true to fall back to regular test execution instead.

This behavior can be enabled by setting the property in the build script:

build.gradle.kts
tasks.test {
    predictiveSelection {
        fallbackToRegularExecutionOnMissingJUnitPlatform.set(true) // default: false
    }
}
build.gradle
tasks.named('test', Test) {
    predictiveSelection {
        fallbackToRegularExecutionOnMissingJUnitPlatform = true // default: false
    }
}

Or, by setting the corresponding system property:

$ ./gradlew test -Ddevelocity.testing.fallbackToRegularExecutionOnMissingJUnitPlatform=true

Maven

Applying the extension

A prerequisite of using Predictive Test Selection is applying and configuring the Develocity Maven extension to your build. Refer to the Develocity Maven extension user manual for detailed information on how to do this.

Authentication

If anonymous access is disabled for viewing build scans, builds must authenticate with Develocity via an access key that has the "Use Predictive Test Selection" permission enabled to use Predictive Test Selection. Please see Authenticating Maven builds with Develocity for how to do this.

Capturing task input files

Builds must be configured to capture goal input files in order to use Predictive Test Selection. Please see Capturing goal input files for Maven for instructions.

Enabling

The examples in this section all reference the Surefire plugin, but the same configuration options are available for the Failsafe plugin. Examples should only require changing the artifactId from maven-surefire-plugin to maven-failsafe-plugin to work with Failsafe.

Predictive Test Selection is enabled and configured per goal, via the predictiveSelection configuration element in the POM.

Dynamically enabling
Using the built-in property

(Maven extension 1.19.2+)

The simplest way to toggle test selection dynamically is by using the pts.enabled system property:

mvn -Dpts.enabled=true test
Using a custom property

Another way to toggle test selection dynamically is by adding a custom property to the Surefire or Failsafe plugin.

The configuration example below introduces a enablePTS user property which expects a boolean value.

pom.xml
<properties>
  <enablePTS>true</enablePTS> (1)
</properties>

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.2.5</version>
      <configuration>
        <properties>
          <predictiveSelection>
            <enabled>${enablePTS}</enabled> (2)
          </predictiveSelection>
        </properties>
      </configuration>
    </plugin>
  </plugins>
</build>
1 Define enablePTS user property with default of true
2 Toggle Predictive Test Selection via enablePTS user property

The custom property can then be overridden at goal execution time, from the command line, using -DenablePTS=false.

mvn -DenablePTS=false test

Declaring additional test inputs

For optimal prediction accuracy it is vital that any additional file inputs required by the test goal are declared for the Surefire or Failsafe plugins.

pom.xml
  <plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
      <properties>
        <predictiveSelection>
          <enabled>true</enabled>
        </predictiveSelection>
      </properties>
    </configuration>
  </plugin>
</plugins>
<pluginManagement>
  <plugins>
    <plugin>
      <groupId>com.gradle</groupId>
      <artifactId>gradle-enterprise-maven-extension</artifactId>
      <configuration>
        <gradleEnterprise>
          <plugins>
            <plugin>
              <artifactId>maven-surefire-plugin</artifactId>
              <inputs>
                <fileSets>
                  <fileSet>
                    <name>accounts-csv</name>
                    <paths>
                      <path>test-fixtures/data/accounts.csv</path>
                    </paths>
                    <normalization>RELATIVE_PATH</normalization>
                  </fileSet>
                  <fileSet>
                    <name>randomized-test-data</name>
                    <paths>
                      <path>randomized-test-data</path>
                    </paths>
                    <normalization>RELATIVE_PATH</normalization>
                  </fileSet>
                </fileSets>
              </inputs>
            </plugin>
          </plugins>
        </gradleEnterprise>
      </configuration>
    </plugin>
  </plugins>
</pluginManagement>

Refer to the Maven extension user manual for more information about declaring additional inputs.

Configuring must-run tests

(Maven extension 1.14+, Develocity 2022.2+)

It is possible to configure the extension to always select tests for execution based on custom criteria.

This can be achieved in two ways: by specifying a class name filter pattern, via the includeClasses configuration element, or in an annotation-driven manner, via the includeAnnotationClasses configuration element. Both configuration properties can be applied simultaneously.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

Specifying tests to run on the command line via -Dtest=…​ (for Surefire) or -Dit.test=…​ (for Failsafe) has the same effect as including them in must-run configuration. If you wish test selection to avoid tests for filtered test sets, follow instructions for enabling Predictive Test Selection for filtered test sets.

From version 1.16 onward, you can also use the com.gradle.enterprise.testing.annotations.MustRun annotation from the gradle-enterprise-testing-annotations project without any additional configuration other than the dependency.

Must-run class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <enabled>true</enabled>
        <mustRun>
          <includeClasses>
            <include>example.project.FileIntegrityTest</include>
            <include>*.project.*SanityTest</include>
          </includeClasses>
        </mustRun>
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>
Must-run annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class' annotations. A class need only have one annotation matching any of the patterns to be included.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <enabled>true</enabled>
        <mustRun>
          <includeAnnotationClasses>
            <include>com.project.annotations.MustRun</include>
          </includeAnnotationClasses>
        </mustRun>
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>

Refer to the Example must-run annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Example must-run annotation

Any annotation class referenced in the includeAnnotationClasses configuration should be implemented to be retained at runtime and target types (class level). See the listing below for a sample implementation.

MustRun.java
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface MustRun {}

Annotations present on super-classes with the @Inherited meta-annotation are considered when inspecting sub-classes.

Selection mode

(Maven extension 1.18+, Develocity 2023.2+)

The selection mode determines how tests are selected for execution (see Relevant vs. remaining tests for details). By default, the most relevant tests according to the used selection profile are selected in order to provide fast feedback. To run the remaining tests, e.g. post-merge, the selection mode can be changed accordingly.

In the “relevant tests” selection mode, all test classes that have not yet had a successful or flaky outcome for the same inputs will be selected. Test classes configured as must-run will only be selected if there was no prior execution with a successful or flaky outcome for the same inputs.

Using the pts.mode system property is the recommended way to change the selection mode:

$ ./mvnw clean test -Dpts.mode=REMAINING_TESTS

Alternatively, the selection mode can be changed via the mode property in the POM.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <mode>REMAINING_TESTS</mode> <!-- default: RELEVANT_TESTS -->
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>
The mode configured via the system property takes precedence over the configuration in the POM.

Selection profile

(Maven extension 1.18+, Develocity 2023.2+)

The selection profile allows adjusting the desired level of trade-off between the PTS system’s confidence (i.e. percentage of correctly predicted failures) vs. savings (i.e. percentage of testing time avoided). You can choose between the values CONSERVATIVE, STANDARD, and FAST. Setting the selection profile is optional; if none is configured, STANDARD will be used.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <profile>FAST</profile> <!-- default: STANDARD -->
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>

The selection profile can alternatively be passed as system property:

$ ./mvnw clean test -Dpts.profile=FAST
The profile configured via the system property takes precedence over the configuration in the POM.

Other configuration parameters

In addition to the above configuration parameters, the Develocity Maven extension respects most of the goal’s configuration.

The following configuration parameters are currently not supported: enableProcessChecker, forkedProcessExitTimeoutInSeconds, forkNode, junitArtifactName, objectFactory, parallel, parallelOptimized, parallelTestsTimeoutForcedInSeconds, parallelTestsTimeoutInSeconds, perCoreThreadCount, runOrder, shutdown, suiteXmlFiles, testNGArtifactName, threadCount, threadCountClasses, threadCountMethods, threadCountSuites, useManifestOnlyJar, useSystemClassLoader, useUnlimitedThreads.

Test dry run mode

(Maven extension 1.18.1+)

To simulate the execution of tests without actually running them, Develocity provides a test dry run mode. This mode generates reports, and tests will still be selected by Predictive Test Selection. This can e.g. be used to verify that your test filtering configuration or Predictive Test Selection configuration is correct, without actually running tests.

The test dry run mode can be enabled by:

  • Setting junit.platform.execution.dryRun.enabled configuration parameters property to true (JUnit 5.10+)

Fallback to regular execution

(Maven extension 1.20+)

Predictive Test Selection relies on JUnit Platform to run tests. By default, the test goal will fail if it isn’t configured to use JUnit Platform. To avoid that, e.g. when centrally enabling Predictive Test Selection, fallbackToRegularExecutionOnMissingJUnitPlatform can be set to true to fall back to regular test execution instead.

This behavior can be enabled by setting the property in the POM:

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <fallbackToRegularExecutionOnMissingJUnitPlatform>true</fallbackToRegularExecutionOnMissingJUnitPlatform> <!-- default: false -->
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>

Or, by setting the corresponding system property:

$ ./mvnw clean test -Ddevelocity.testing.fallbackToRegularExecutionOnMissingJUnitPlatform=true

Observing Predictive Test Selection in Develocity

After Predictive Test Selection has been enabled in your Gradle or Maven build, you will see build logs indicating its impact when running tests:

Predictive Test Selection: 5 of 22 test classes selected with profile 'Standard' (saving 23m 45s serial time)

Once the test run completes, results can be inspected via the generated Build Scan or the Predictive Test Selection dashboard.

Inspecting selected and not selected tests

Each Build Scan contains a summary and full details for test runs in a single build.

The count and estimated serial time savings is reported on the Summary view.

build scan summary

Navigate to the “Tests” view to see a structured and searchable list of tests. Test tasks/goals and test classes affected by Predictive Test Selection are indicated with a wire brain symbol. You can click this symbol to view details for the test task/goal or test class.

not selected test details

The details show estimated saved serial time based on recent executions, as well as a logical prediction explanation for selected and not selected test classes. If your Develocity instance has the Test Analytics extension enabled, you can view the test history to inspect recent test outcomes, including “NOT SELECTED” results.

not selected test trend

Reporting use and impact over time

Test selection results from all test runs are stored in Develocity, and can be aggregated, filtered, and analyzed using the Predictive Test Selection Dashboard.

usage dashboard

The Usage view of the dashboard visualizes the usage of Predictive Test Selection and estimated savings trends, as well as a summary for matching tasks/goals. Here you can see exactly which tasks/goals enabled test selection, how often, and the impact of doing so. This allows you to understand how test selection is being used and the resulting impact.

You can use the search or click on any task/goal to view trends for the specific task or goal, and to see details for individual test runs which enabled test selection.

Clicking any test run navigates to the associated Build Scan, where you can inspect full details of each test, including prediction explanations and estimated savings for each test.

You can also see test runs where Predictive Test Selection was unavailable, that is, selection was attempted but could not be completed for some reason, resulting in all tests being executed. Each list item shows the reason test selection was unavailable so you can quickly identify and respond to problems.

test selection unavailable

Investigating “missed” test failures

Predictive Test Selection will rarely skip a test which would have failed, which may be caught when all tests are run. The process for investigating such occurrences is similar to git bisect:

Step 1: Analyze the build and test logs in the Build Scan to identify the nature of the failure. Many test failures are caused by an unstable environment. It is helpful to quickly rule out flakiness if possible.

Step 2: Analyze test execution history to identify last known good state. You can view the test history for a specific test by clicking “View test history” when viewing test details in a Build Scan, or via searching the Tests Dashboard. Using custom metadata you can quickly isolate a specific stream of builds (e.g. test runs on a specific branch).

test history summary

Step 3: Identify the changes from the last passing execution leading up to the failed test. Build comparison between the last passing and failing test builds is a useful way to identify what changed. It provides snapshots at a different granularity, between builds instead of version control commits, which may allow you to more quickly identify the breaking change.

If you identify that Predictive Test Selection did “miss” a test failure, you have the option to add the test class to must-run criteria to be certain it does not happen again. Predictive Test Selection will learn from every test failure, thus making rigorous “correcting” of mistakes unnecessary.

Troubleshooting

Predictive Test Selection is not enabled

Predictive Test Selection is an extension of Develocity and may not be enabled for your installation. Please contact your customer representative to arrange a trial of the extension if you wish to try it.

Simulator shows “No simulations found”

After the Predictive Test Selection extension is enabled initially, the Develocity server starts to analyze existing builds and perform simulations. Depending on the amount and size of existing Build Scans, this process can take several hours or days. You can track its progress by adjusting the “Start time” filter on the Simulator. When in doubt, please contact Develocity support.

Test execution fails due to no test engine being registered

If your test task/goal fails with an error message similar to the following, you have probably not been executing your tests via the JUnit Platform before enabling Predictive Test Selection.

Failure during test discovery: Forked test JVM terminated unexpectedly with exit value 1
Standard error from JVM:
    Terminating due to fatal error
    org.junit.platform.commons.PreconditionViolationException: Cannot create Launcher without at least one TestEngine; consider adding an engine implementation JAR to the classpath
      at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
      at org.junit.platform.launcher.core.DefaultLauncher.<init>(DefaultLauncher.java:55)
      ...

As indicated by JUnit’s exception message, there needs to be at least one "test engine" (a JUnit Platform concept) registered in order to run tests. If your tests are written in JUnit 3/4, you should add a test runtime dependency on the JUnit Vintage engine. If they use TestNG, you need an additional dependency to the TestNG engine.

If you use JUnit Jupiter, Spock 2.x, or any other supported test framework that runs natively on the JUnit Platform, you should not have to add any dependency.

build.gradle.kts
repositories {
    mavenCentral()
}
dependencies {
    // For JUnit 3/4
    testImplementation("junit:junit:4.13.2")
    testRuntimeOnly("org.junit.vintage:junit-vintage-engine:5.10.0")
    // For TestNG
    testImplementation("org.testng:testng:7.8.0")
    testRuntimeOnly("org.junit.support:testng-engine:1.0.4")
}
build.gradle
repositories {
    mavenCentral()
}
dependencies {
    // For JUnit 3/4
    testImplementation('junit:junit:4.13.2')
    testRuntimeOnly('org.junit.vintage:junit-vintage-engine:5.10.0')
    // For TestNG
    testImplementation('org.testng:testng:7.8.0')
    testRuntimeOnly('org.junit.support:testng-engine:1.0.4')
}
pom.xml
<dependencies>

  <!-- For JUnit 3/4 -->
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.13.2</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.10.0</version>
    <scope>test</scope>
  </dependency>

  <!-- For TestNG -->
  <dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.8.0</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.junit.support</groupId>
    <artifactId>testng-engine</artifactId>
    <version>1.0.4</version>
    <scope>test</scope>
  </dependency>

</dependencies>

Test execution fails when Predictive Test Selection is enabled

If some tests start failing when Predictive Test Selection is enabled but they pass when it is disabled, your tests may have dependencies on each other.

Since Predictive Test Selection does not execute all tests, a test which depends on another test that is not selected will fail. Furthermore, tests may be executed in a different order when Predictive Test Selection is enabled. There’s no way to force Predictive Test Selection to execute test in alphabetical order, for example.

All tests selected due to "must-run criteria"

If your Gradle invocation filters tests using --tests (or Test.setTestNameIncludePatterns, the programmatic equivalent) or your Maven goal filters tests using -Dtest=…​ or -Dit.test=…​ (or uses the <test> property in the POM), Predictive Test Selection will always select all matching tests to ensure a consistent CLI and IDE user experience.

Take one of the following actions to allow Predictive Test Selection to avoid tests normally:

For Gradle, register a custom task with a TestFilter to split a test suite.

build.gradle.kts
tasks.register<Test>("featureTest") {
    filter {
        includeTestsMatching("com.example.feature.*")
    }
}
build.gradle
tasks.register('featureTest', Test) {
    filter {
        includeTestsMatching 'com.example.feature.*'
    }
}

For Maven, use test exclusions or inclusions to split a test suite.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.5</version>
  <executions>
    <execution>
      <id>feature-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <includes>
          <include>com/example/feature/**/*.java</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

Build was unable to contact Develocity server

When using Predictive Test Selection, before tests are executed, the Develocity server is contacted to decide which tests should be selected. If the server is unreachable, Predictive Test Selection is deactivated and all tests are executed.

Recent test executions have not been processed

In order to select tests for a build, Develocity needs to have processed enough recent builds. If processing lags behind, Predictive Test Selection is temporarily deactivated and all tests are executed to avoid selecting the wrong set of tests.

Test selection request took too long to process

If the selection requests to the Develocity server take too long, test selection is disabled and all tests are executed. If you frequently encounter timeouts, please contact Develocity support.

Insufficient change history available

Predictive Test Selection will not attempt to make predictions for test tasks or goals for which fewer than 14 days of code and dependency changes are available. In this case, all tests except those having previously passed for identical inputs are selected and executed. It is possible that input file capture is not enabled and must be configured for Gradle or Maven.

Client not authorized to use Predictive Test Selection

If anonymous access is disabled for viewing build scans, builds must authenticate with Develocity via an access key that has the “Use Predictive Test Selection” permission enabled to use Predictive Test Selection.

Test task/goal inputs could not be determined

In order to make a prediction, Predictive Test Selection needs to capture test task/goal inputs. Such failures can be caused, for example, by Gradle work validation problems.

Build logs contain “input capturing is disabled” warnings

If task/goal input capturing is disabled, Predictive Test Selection will not be able to use its outcome for predictions of future builds. This, however, does not affect the prediction for the current build.

Build logs contain “not publishing Build Scans” warnings

In order to make predictions, Predictive Test Selection requires data about as many builds as possible. Therefore, we recommend publishing a Build Scan for all builds regardless whether they are running locally or on CI.

Build logs contain “absolute path normalization” warnings

When declaring additional input files or directories on a task/goal, you should avoid using absolute path normalization. Tracking absolute paths as part of a task/goal’s cache key makes the build cache less effective since entries often cannot be reused across machines. Moreover, it convolutes the change history used by Predictive Test Selection and negatively impacts its predictions.

Reporting internal errors

If you encounter builds where Predictive Test Selection was disabled due to “Build plugin encountered an internal error” or “Predictive Test Selection encountered an internal error”, please report it to Develocity support.

FAQs

How does Predictive Test Selection decide which tests are relevant?

The machine learning model takes many variables into account when making a decision. For example, which code changes affect tests in the module, how sensitive the test is to changes, the nature of recent changes made, how flaky the test is, and so on.

Tests will always be chosen if they are recently new, recently changed, recently failed, or recently flaky.

Tests which have already passed against the same sources and dependencies are less likely to be selected.

How is the machine learning model trained?

The machine learning model was trained with millions of test executions and dozens of projects from Gradle’s data partners. The training and test sets cover a variety of JVM-based projects such as libraries, API services, Android apps, server apps, and browser-based apps.

Once enabled, Predictive Test Selection will learn from historical and new code changes and test outcomes for your projects.

Can I influence predictions?

Yes, you can specify test patterns and annotations (for Gradle and Maven) which force matching tests to be selected and executed.

You can also choose to use a different selection profile to adjust the desired level of trade-off between the PTS system’s confidence (i.e. percentage of correctly predicted failures) vs. savings (i.e. percentage of testing time avoided).

Other inference aspects are not customizable.

What does it mean for a test to be not selected?

A test outcome of “NOT SELECTED” indicates that the test class was skipped because Predictive Test Selection’s machine learning system determined that it is unlikely to provide a useful test signal.

What does “insufficient data” mean?

This means that Predictive Test Selection does not have sufficient change history or sufficient test history to make a prediction.

The most common reasons the simulator may report that all test runs have “insufficient data” are:

  • Lack of file input snapshots for test builds. The Develocity Gradle plugin or Develocity Maven extension must be configured to capture input files for relevant test builds to support Predictive Test Selection.

  • Lack of relevant test execution history containing file input snapshots. The system requires at least 50 executions over a 14-day period for the target tests in order to guarantee accurate predictions. This avoids inaccurate predictions by giving the model a minimum amount of learning before making predictions.

  • Insufficient Gradle metadata from old versions of the build tool. Gradle versions prior to version 5.4 lack metadata that is critical for test selection.

The Simulator indicates the number of tests and amount of serial time where there was insufficient inference data, and the system was forced to select all impacted tests.

How to measure code coverage when using Predictive Test Selection?

Many builds use JaCoCo or other tools to measure code coverage. Some builds might even fail if the code coverage is below a configured target threshold. When running only a subset of tests as it is the case when Predictive Test Selection is enabled, code coverage might be lower than expected. Therefore, we recommend to only enable code coverage measurement and checks for builds when Predictive Test Selection is disabled.

Appendix A: Release history

Please refer to the release history of Develocity and the Develocity Gradle plugin or Develocity Maven extension.

Appendix B: Compatibility with Gradle Build Tool and Develocity

Compatibility between versions of Gradle, Develocity and the Develocity Gradle plugin can be found here.