Gradle Enterprise Predictive Test Selection increases developer productivity by automatically and intelligently selecting and executing the subset of tests that are most relevant to a code change, providing faster feedback. It is available for Gradle and Apache Maven™ builds and is complementary to other Gradle Enterprise build acceleration features such as Build Caching and Test Distribution.

The technique of avoiding irrelevant tests in order to provide faster feedback to developers was popularized by Meta and other advanced software organizations. Gradle Enterprise Predictive Test Selection was inspired by their work, and makes the benefits of this technique available to all Gradle Enterprise users.

Gradle Enterprise Predictive Test Selection is an extension and may not be enabled for your installation. Please contact your customer representative to enable this extension.

How it works

Before executing a test suite, Predictive-Test-Selection-enabled builds ask Gradle Enterprise to predict which of the tests-to-be-executed will provide useful feedback on the changes under test, selecting only those tests for execution. The Build Scan™ indicates which tests were selected and which weren’t along with the reason why. It also indicates how much time was saved by skipping the not selected tests.

The predictive model

Gradle Enterprise develops a predictive model by observing code changes and test outcomes from your Build Scan data. It combines ongoing learning about your project structure and tests with training from millions of test executions across many projects in order to make accurate predictions.

A snapshot of the test suite and the code-under-test is cross-referenced with change and test result history to predict the subset of relevant tests, which is typically a small portion of the entire test suite.

how predictive test selection works

As each new Build Scan is uploaded to Gradle Enterprise, the predictive model is updated based on the new results.

Flaky test handling

As flaky tests are inherently unpredictable, Predictive Test Selection will always select recently flaky tests.

To maximize the effectiveness of Predictive Test Selection, it is recommended that Gradle Enterprise Test Analytics be used to identify and prioritize the stabilization of flaky tests in your project.

Relationship with other acceleration techniques and technologies

Build Cache

Predictive Test Selection is complementary to Gradle Enterprise Build Cache.

Test results are not stored in the build cache when Predictive Test Selection is enabled. Therefore, when Predictive Test Selection is not enabled, any previous Predictive Test Selection enabled execution of the same test suite for the same code-under-test will not prevent the test suite from executing.

When executing the same test suite for the same code-under-test with Predictive Test Selection enabled, previously selected tests that passed will not be selected again. Instead, Predictive Test Selection will select different tests, and typically fewer, for subsequent executions.

Test Distribution

Predictive Test Selection is complementary to Gradle Enterprise Test Distribution. The tests selected for execution can be accelerated by distribution, further increasing build speed.

Flaky test retry

Predictive Test Selection incorporates flaky test retry functionality through the Test Distribution Gradle Plugin and Gradle Enterprise Maven Extension. Enabling flaky test retry is encouraged as it allows test selection predictions to better account for non-deterministic flaky test outcomes. This is especially problematic for long-running tests such as integration tests and end-to-end tests.

Static analysis and code coverage based alternatives

Other test selection tools are available that employ static analysis or historical code coverage information to select tests. Gradle Enterprise Predictive Test Selection uses a more intelligent selection process that takes more factors into account, such as the historical relationship between a test and the code-under-test. This is particularly significant for long-running tests such as integration tests and end-to-end tests where a large amount of code-under-test is typically executed during a test.

When to use Predictive Test Selection

Predictive Test Selection intelligently trades testing comprehensiveness for faster feedback, making it worthwhile for many test executions where reducing feedback time is critical, such as local and pre-merge/pull-request builds.

To benefit from the faster feedback offered by Predictive Test Selection during development while still ensuring that changesets are tested comprehensively before releasing, additional test executions that do not enable Predictive Test Selection should occur later in the change lifecycle. For example, as part of post-merge, nightly or ready-for-release builds.

ci pipeline

Shift-left integration testing

An essential aspect of modern software development is testing early and often during the development process instead of at the end, known as “shift-left testing”. However, a common approach is to have fast tests, such as unit tests, execute early in the lifecycle of a change and defer all long-running tests such as integration or end-to-end tests. Despite such tests being very good at detecting defects, they are often delayed until later in the development process due to being slower than unit tests. Predictive Test Selection allows a more productive approach where the most valuable subset of integration and end-to-end tests are also “shifted left”.

By executing the most relevant subset of long-running integration and end-to-end tests run early during the lifecycle of a change, Predictive Test Selection gives developers the high-quality feedback such tests provide quickly.

Test compatibility

Frameworks and languages

Tests must be run via JUnit Platform, which is part of JUnit 5. Many popular test frameworks are compatible with JUnit Platform and are supported by Predictive Test Selection, such as JUnit (including JUnit 3 and JUnit 4), Spock, TestNG, jqwik and more. Please see Gradle Testing in Java & JVM projects guide or Maven Surefire’s documentation for guidance on configuring your build to use JUnit Platform.

Tests and sources authored in popular JVM languages are supported, including Java, Kotlin, Groovy, and Scala.

Some JUnit Platform test engines, such as Cucumber, Kotest, and Spek, are currently not supported. Please contact Gradle Enterprise support if you are interested in using these or other incompatible test engines with Predictive Test Selection.

Gradle

Gradle Enterprise Predictive Test Selection is compatible with Gradle 5.4 and later, and works with Gradle’s built-in Test task and its subclasses. This includes Android unit test tasks, but does not include most Android device tests.

Tests must be run with Java 8 or later.

Tests from the buildSrc project and from included builds are currently not supported.

Maven

Gradle Enterprise Predictive Test Selection is compatible with the Maven Surefire plugin 2.22.2 and later, and Maven Failsafe plugin 2.22.2 and later.

Tests must be run with Java 8 or later.

The Predictive Test Selection Simulator

Gradle Enterprise provides a Predictive Test Selection Simulator that facilitates making an informed decision about which projects and builds should enable Predictive Test Selection.

predictive test selection simulator

The simulator works by comparing actual test results from Build Scans to what would have happened if the build used Predictive Test Selection, visualizing the following:

  • % Task/Goal failures predicted— Predicted test tasks/goals with failures versus all test tasks/goals with failures

  • % Test failures predicted— Predicted test failures versus all test failures

  • Savings potential— Time taken to run selected tests vs time taken to run all tests

  • Avoidable tests— Count of test classes which would have been “NOT SELECTED” versus all test classes

Simulation results are available for each test task or goal execution, and can be filtered using the filtering criteria atop the dashboard.

Prerequisites

The Predictive Test Selection Simulator requires Gradle Enterprise 2022.1 or later, and the Gradle Enterprise Predictive Test Selection extension. Please contact your customer representative to enable this extension.

It requires that builds are configured to capture task/goal input files as part of Build Scans. For Gradle, see Capturing task input files for Gradle. For Maven, see Capturing goal input files for Maven.

Interpreting simulation results

The overview metrics and charts help you quickly estimate the expected benefits and risks of enabling Predictive Test Selection for selected builds.

Measuring expected risk

The task/goal failures predicted chart indicates the likelihood that Predictive Test Selection will miss a breaking change. It shows test failures per day to highlight trends and outliers. For example, you may find that there were many failures on a particular day and that test selection had more incorrect predictions that same day. This may indicate some kind of test infrastructure problem which caused unpredictable failures for a short time.

“Correct” predictions would have caught a breaking change, while “incorrect” predictions would not have. However, it is possible for flaky test failures to be misclassified as “incorrect” predictions.

Flaky test failures detected by Gradle Enterprise are excluded from these predicted failures because they do not reliably represent faults in production code. Some flaky test failures, especially infrastructure failures, may not be detected as flaky by Gradle Enterprise, and will be classified as “incorrect”.

Analyzing possibly incorrect predictions

Tests are intended to indicate correctness of production code to developers and not the stability of the infrastructure that runs the tests themselves. Unfortunately, there exists no test environment which is completely reliable, and sometimes tests fail due to infrastructure problems.

Predictive Test Selection attempts to detect flaky test failures and omit them from simulation results.

In order to obtain an accurate estimate of expected Predictive Test Selection quality, it is recommended to inspect incorrect predictions and omit simulations (via search filters) for test builds in which all test failures were caused by test flakiness or test environment problems not relating to the code.

Measuring expected benefit

The savings potential chart indicates the amount and ratio of serial test time that could be saved if Predictive Test Selection had been applied to all matching builds.

The system is calibrated to catch over 99% of non-flaky test task/goal failures for many different types of projects and changesets, which allows it to adapt to unforeseen situations as your project evolves. This implies that benefits are not uniformly distributed: riskier code changes require more testing to maintain high confidence.

The avoidable tests chart indicates the number and ratio of test executions which would have been skipped. This estimates the potential for future Predictive Test Selection improvements.

Enabling Predictive Test Selection

Prerequisites

Predictive Test Selection requires Gradle Enterprise 2022.2 or later, and the Gradle Enterprise Predictive Test Selection extension. Please contact your customer representative to enable this extension.

After enabling the extension, recent build data must be processed to construct the prediction model. Depending on the amount of build data available, this process may take up to 3 days. For most installations, this processing is able to complete overnight.

To use Predictive Test Selection during a build, the build must authenticate with Gradle Enterprise and the user be granted the “Build scan view” access control permission. See below for build-tool specific authentication instructions.

Builds must be configured to publish Build Scans which capture task/goal input files. See below for build-tool specific instructions on how to enable this.

Gradle

A prerequisite of using Predictive Test Selection is applying and configuring the Gradle Enterprise Gradle Plugin to your build. Please refer to the Gradle Enterprise Gradle Plugin user manual for detailed information on how to do this.

Applying the Gradle plugin

Predictive Test Selection is bundled with the Gradle Enterprise Test Distribution plugin. The shared plugin ID is com.gradle.enterprise.test-distribution. Version 2.3.1 is the latest version.

Gradle 6.0 and above

The plugin must be applied in the settings file of the build in addition to the com.gradle.enterprise plugin.

settings.gradle
plugins {
    id('com.gradle.enterprise') version '3.10.1'
    id('com.gradle.enterprise.test-distribution') version '2.3.1'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}
settings.gradle.kts
plugins {
    id("com.gradle.enterprise") version "3.10.1"
    id("com.gradle.enterprise.test-distribution") version "2.3.1"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}
Gradle 5.4x to 5.6x

The plugin must be applied to the root project of the build in addition to the com.gradle.build-scan plugin.

build.gradle
plugins {
    id('com.gradle.build-scan') version '3.10.1'
    id('com.gradle.enterprise.test-distribution') version '2.3.1'
}

gradleEnterprise {
    server = 'https://gradle-enterprise.mycompany.com'
}
build.gradle.kts
plugins {
    id("com.gradle.build-scan") version "3.10.1"
    id("com.gradle.enterprise.test-distribution") version "2.3.1"
}

gradleEnterprise {
    server = "https://gradle-enterprise.mycompany.com"
}

Enabling

Predictive Test Selection is enabled and configured per Test task, via the predictiveSelection DSL extension.

Conditionally enabling

Instead of using a static boolean value in the DSL extension, it is possible to toggle test selection dynamically via project properties specified at the time of build invocation.

The build script example below reads the isPTSEnabled project property to determine if test selection should be enabled for execution of the test task.

build.gradle
tasks.named('test', Test).configure {
    useJUnitPlatform() (1)
    predictiveSelection {
        enabled = project.hasProperty("isPTSEnabled") (2)
    }
}
build.gradle.kts
tasks.test {
    useJUnitPlatform() (1)
    predictiveSelection {
        enabled.set(project.hasProperty("isPTSEnabled")) (2)
    }
}
1 Use JUnit Platform for executing tests (see Test compatibility)
2 Enable Predictive Test Selection

The project property is specified as a system property at runtime.

./gradlew test -PisPTSEnabled

Alternatively, the project property can be specified via an environment variable.

ORG_GRADLE_PROJECT_isPTSEnabled=true

For more information on project properties, refer to the Gradle documentation.

Declaring additional test inputs

For optimal prediction accuracy it is vital that any additional file inputs to the test task are declared.

Additional inputs can be declared via the build script by configuring the test task.

build.gradle
tasks.named('test', Test).configure {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
    }
    inputs.file('test-fixtures/data/accounts.csv').withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.file('randomized-test-data').withPathSensitivity(PathSensitivity.RELATIVE)
}
build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
    }
    inputs.file("test-fixtures/data/accounts.csv").withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.file("randomized-test-data").withPathSensitivity(PathSensitivity.RELATIVE)
}

Refer to the Gradle build tool Authoring tasks guide for more information about input declaration.

Configuring must-run tests

It is possible to configure the plugin to always select tests for execution based on custom criteria.

This can be achieved in two ways: by specifying a class name filter pattern, via the includeClasses configuration property, or in an annotation-driven manner, via the includeAnnotationClasses configuration property. Both configuration properties can be applied simultaneously.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

Specifying tests to run on the command line via --tests …​ has the same effect as including them in must-run configuration. If you wish test selection to avoid tests for filtered test sets, follow instructions for enabling Predictive Test Selection for filtered test sets.

Must-run class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

build.gradle
tasks.named('test', Test).configure {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
        mustRun {
            includeClasses.addAll('com.project.FileIntegrityTest', '*.project.*SanityTest')
        }
    }
}
build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
        mustRun {
            includeClasses.addAll("com.project.FileIntegrityTest", "*.project.*SanityTest")
        }
    }
}
Must-run annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class’ annotations. A class need only have one annotation matching any of the patterns to be included.

build.gradle
tasks.named('test', Test).configure {
    useJUnitPlatform()
    predictiveSelection {
        enabled = true
        mustRun {
            includeAnnotationClasses.add('com.project.annotations.MustRun')
        }
    }
}
build.gradle.kts
tasks.test {
    useJUnitPlatform()
    predictiveSelection {
        enabled.set(true)
        mustRun {
            includeAnnotationClasses.add("com.project.annotations.MustRun")
        }
    }
}

Refer to the Example must-run annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Maven

Applying the extension

A prerequisite of using Predictive Test Selection is applying and configuring the Gradle Enterprise Maven extension to your build. Refer to the Gradle Enterprise Maven extension user manual for detailed information on how to do this.

Enabling

The examples in this section all reference the Surefire plugin, but the same configuration options are available for the Failsafe plugin. Examples should only require changing the artifactId from maven-surefire-plugin to maven-failsafe-plugin to work with Failsafe.

Predictive Test Selection is enabled and configured per goal, via the predictiveSelection configuration element in the POM.

Dynamically enabling

The simplest way to toggle test selection dynamically is by adding a custom property to the Surefire or Failsafe plugin.

The configuration example below introduces a isPTSEnabled user property which expects a boolean value.

pom.xml
<properties>
  <isPTSEnabled>true</isPTSEnabled> (1)
</properties>

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.0.0-M6</version>
      <configuration>
        <properties>
          <predictiveSelection>
            <enabled>${isPTSEnabled}</enabled> (2)
          </predictiveSelection>
        </properties>
      </configuration>
    </plugin>
  </plugins>
</build>
1 Define isPTSEnabled user property with default of true
2 Toggle Predictive Test Selection via isPTSEnabled user property

The custom property can then be overridden at goal execution time, from the command line, using -DisPTSEnabled=false.

mvn -DisPTSEnabled=false test

Declaring additional test inputs

For optimal prediction accuracy it is vital that any additional file inputs required by the test goal are declared for the Surefire or Failsafe plugins.

pom.xml
  <plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M6</version>
    <configuration>
      <properties>
        <predictiveSelection>
          <enabled>true</enabled>
        </predictiveSelection>
      </properties>
    </configuration>
  </plugin>
</plugins>
<pluginManagement>
  <plugins>
    <plugin>
      <groupId>com.gradle</groupId>
      <artifactId>gradle-enterprise-maven-extension</artifactId>
      <configuration>
        <gradleEnterprise>
          <plugins>
            <plugin>
              <artifactId>maven-surefire-plugin</artifactId>
              <inputs>
                <fileSets>
                  <fileSet>
                    <name>accounts-csv</name>
                    <paths>
                      <path>test-fixtures/data/accounts.csv</path>
                    </paths>
                    <normalization>RELATIVE_PATH</normalization>
                  </fileSet>
                  <fileSet>
                    <name>randomized-test-data</name>
                    <paths>
                      <path>randomized-test-data</path>
                    </paths>
                    <normalization>RELATIVE_PATH</normalization>
                  </fileSet>
                </fileSets>
              </inputs>
            </plugin>
          </plugins>
        </gradleEnterprise>
      </configuration>
    </plugin>
  </plugins>
</pluginManagement>

Refer to the Maven extension user manual for more information about declaring additional inputs.

Configuring must-run tests

It is possible to configure the extension to always select tests for execution based on custom criteria.

This can be achieved in two ways: by specifying a class name filter pattern, via the includeClasses configuration element, or in an annotation-driven manner, via the includeAnnotationClasses configuration element. Both configuration properties can be applied simultaneously.

Wildcards/asterisks are supported in the pattern string and match zero or more characters. Multiple wildcards/asterisks can be used in a pattern string.

Specifying tests to run on the command line via -Dtest=…​ (for Surefire) or -Dit.test=…​ (for Failsafe) has the same effect as including them in must-run configuration. If you wish test selection to avoid tests for filtered test sets, follow instructions for enabling Predictive Test Selection for filtered test sets.

Must-run class matcher

Configures the patterns used to match tests for selection based on their class name.

Patterns are evaluated against fully qualified class names. A class name only has to match one of the patterns to be selected.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0-M6</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <enabled>true</enabled>
        <mustRun>
          <includeClasses>
            <include>example.project.FileIntegrityTest</include>
            <include>*.project.*SanityTest</include>
          </includeClasses>
        </mustRun>
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>
Must-run annotation matcher

Configures the patterns used to match tests for selection based on their class level annotations.

Patterns are evaluated against the fully qualified class names of a test class' annotations. A class need only have one annotation matching any of the patterns to be included.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0-M6</version>
  <configuration>
    <properties>
      <predictiveSelection>
        <enabled>true</enabled>
        <mustRun>
          <includeAnnotationClasses>
            <include>com.project.annotations.MustRun</include>
          </includeAnnotationClasses>
        </mustRun>
      </predictiveSelection>
    </properties>
  </configuration>
</plugin>

Refer to the Example must-run annotation section which contains a code listing for a sample implementation of an “include” annotation class.

Example must-run annotation

Any annotation class referenced in the includeAnnotationClasses configuration should be implemented to be retained at runtime and target types (class level). See the listing below for a sample implementation.

MustRun.java
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface MustRun {}

Annotations present on super-classes with the @Inherited meta-annotation are considered when inspecting sub-classes.

Observing Predictive Test Selection in Gradle Enterprise

After Predictive Test Selection has been enabled in your Gradle or Maven build, you will see build logs indicating its impact when running tests:

Predictive Test Selection: 5 of 22 test classes selected (saving ~23m45s serial time)

Once the test run completes, results can be inspected via the generated Build Scan or the Predictive Test Selection dashboard.

Inspecting selected and not selected tests

Each Build Scan contains a summary and full details for test runs in a single build.

The count and estimated serial time savings is reported on the Summary view.

build scan summary

Navigate to the “Tests” view to see a structured and searchable list of tests. Test tasks/goals and test classes affected by Predictive Test Selection are indicated with a wire brain symbol. You can click this symbol to view details for the test task/goal or test class.

not selected test details

The details show estimated saved serial time based on recent executions, as well as a logical prediction explanation for selected and not selected test classes. If your Gradle Enterprise instance has the Test Analytics extension enabled, you can view the test history to inspect recent test outcomes, including “NOT SELECTED” results.

not selected test trend

Reporting use and impact over time

Test selection results from all test runs are stored in Gradle Enterprise, and can be aggregated, filtered, and analyzed using the Predictive Test Selection Dashboard.

usage dashboard

The Usage view of the dashboard visualizes the usage of Predictive Test Selection and estimated savings trends, as well as a summary for matching tasks/goals. Here you can see exactly which tasks/goals enabled test selection, how often, and the impact of doing so. This allows you to understand how test selection is being used and the resulting impact.

You can use the search or click on any task/goal to view trends for the specific task or goal, and to see details for individual test runs which enabled test selection.

Clicking any test run navigates to the associated Build Scan, where you can inspect full details of each test, including prediction explanations and estimated savings for each test.

You can also see test runs where Predictive Test Selection was unavailable, that is, selection was attempted but could not be completed for some reason, resulting in all tests being executed. Each list item shows the reason test selection was unavailable so you can quickly identify and respond to problems.

test selection unavailable

Investigating “missed” test failures

Predictive Test Selection will rarely skip a test which would have failed, which may be caught when all tests are run. The process for investigating such occurrences is similar to git bisect:

Step 1: Analyze the build and test logs in the Build Scan to identify the nature of the failure. Many test failures are caused by an unstable environment. It is helpful to quickly rule out flakiness if possible.

Step 2: Analyze test execution history to identify last known good state. You can view the test history for a specific test by clicking “View test history” when viewing test details in a Build Scan, or via searching the Tests Dashboard. Using custom metadata you can quickly isolate a specific stream of builds (e.g. test runs on a specific branch).

test history summary

Step 3: Identify the changes from the last passing execution leading up to the failed test. Build comparison between the last passing and failing test builds is a useful way to identify what changed. It provides snapshots at a different granularity, between builds instead of version control commits, which may allow you to more quickly identify the breaking change.

If you identify that Predictive Test Selection did “miss” a test failure, you have the option to add the test class to must-run criteria to be certain it does not happen again. Predictive Test Selection will learn from every test failure, thus making rigorous “correcting” of mistakes unnecessary.

Troubleshooting

Predictive Test Selection is not enabled

Predictive Test Selection is an extension of Gradle Enterprise and may not be enabled for your installation. Please contact your customer representative to arrange a trial of the extension if you wish to try it.

Simulator shows “No simulations found”

After the Predictive Test Selection extension is enabled initially, the Gradle Enterprise server starts to analyze existing builds and perform simulations. Depending on the amount and size of existing Build Scans, this process can take several hours or days. You can track its progress by adjusting the “Start time” filter on the Simulator. When in doubt, please contact Gradle Enterprise support.

All tests selected due to "must-run criteria"

If your Gradle invocation filters tests using --tests (or Test.setTestNameIncludePatterns, the programmatic equivalent) or your Maven goal filters tests using -Dtest=…​ or -Dit.test=…​ (or uses the <test> property in the POM), Predictive Test Selection will always select all matching tests to ensure a consistent CLI and IDE user experience.

Take one of the following actions to allow Predictive Test Selection to avoid tests normally:

For Gradle, register a custom task with a TestFilter to split a test suite.

build.gradle
tasks.register('featureTest', Test) {
    filter {
        includeTestsMatching "com.example.feature.*"
    }
}
build.gradle.kts
tasks.register<Test>("featureTest") {
    filter {
        includeTestsMatching("com.example.feature.*")
    }
}

For Maven, use test exclusions or inclusions to split a test suite.

pom.xml
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0-M6</version>
  <executions>
    <execution>
      <id>feature-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <includes>
          <include>com/example/feature/**/*.java</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

Build was unable to contact Gradle Enterprise server

When using Predictive Test Selection, before tests are executed, the Gradle Enterprise server is contacted to decide which tests should be selected. If the server is unreachable, Predictive Test Selection is deactivated and all tests are executed.

Recent test executions have not been processed

In order to select tests for a build, Gradle Enterprise needs to have processed enough recent builds. If processing lags behind, Predictive Test Selection is temporarily deactivated and all tests are executed to avoid selecting the wrong set of tests.

Test selection request took too long to process

If the selection requests to the Gradle Enterprise server take too long, test selection is disabled and all tests are executed. If you frequently encounter timeouts, please contact Gradle Enterprise support.

Insufficient change history available

Predictive Test Selection will not attempt to make predictions for test tasks or goals for which fewer than 14 days of code and dependency changes are available. In this case, all tests except those having previously passed for identical inputs are selected and executed. It is possible that input file capture is not enabled and must be configured for Gradle or Maven.

Client not authorized to use Predictive Test Selection

In order to use Predictive Test Selection, the build needs to authenticate with Gradle Enterprise and the authenticated users need to have the “Build scan view” access control permission.

Build logs contain “input capturing is disabled” warnings

If task/goal input capturing is disabled, Predictive Test Selection will not be able to analyze the build in order to make predictions.

Build logs contain “not publishing Build Scans” warnings

In order to make predictions, Predictive Test Selection requires data about as many builds as possible. Therefore, we recommend publishing a Build Scan for all builds regardless whether they are running locally or on CI.

Build logs contain “absolute path normalization” warnings

When declaring additional input files or directories on a task/goal, you should avoid using absolute path normalization. Tracking absolute paths as part of a task/goal’s cache key makes the build cache less effective since entries often cannot be reused across machines. Moreover, it convolutes the change history used by Predictive Test Selection and negatively impacts its predictions.

Reporting internal errors

If you encounter builds where Predictive Test Selection was disabled due to “Build plugin encountered an internal error” or “Predictive Test Selection encountered an internal error”, please report it to Gradle Enterprise support.

FAQs

How does Predictive Test Selection decide which tests are relevant?

The machine learning model takes many variables into account when making a decision. For example, which code changes affect tests in the module, how sensitive the test is to changes, the nature of recent changes made, how flaky the test is, and so on.

Tests will always be chosen if they are recently new, recently changed, recently failed, or recently flaky.

Tests which have already passed against the same sources and dependencies are less likely to be selected.

How is the machine learning model trained?

The machine learning model was trained with millions of test executions and dozens of projects from Gradle’s data partners. The training and test sets cover a variety of JVM-based projects such as libraries, API services, Android apps, server apps, and browser-based apps.

Once enabled, Predictive Test Selection will learn from historical and new code changes and test outcomes for your projects.

Can I influence predictions?

Yes, you can specify test patterns and annotations (for Gradle and Maven) which force matching tests to be selected and executed. Other inference aspects are not customizable.

What does it mean for a test to be not selected?

A test outcome of “NOT SELECTED” indicates that the test class was skipped because Predictive Test Selection’s machine learning system determined that it is unlikely to provide a useful test signal.

What does “insufficient data” mean?

This means that Predictive Test Selection does not have sufficient change history or sufficient test history to make a prediction.

The most common reasons the simulator may report that all test runs have “insufficient data” are:

  • Lack of file input snapshots for test builds. The Gradle Enterprise Gradle plugin or Gradle Enterprise Maven extension must be configured to capture input files for relevant test builds to support Predictive Test Selection.

  • Lack of relevant test execution history containing file input snapshots. The system requires at least 50 executions over a 14-day period for the target tests in order to guarantee accurate predictions. This avoids inaccurate predictions by giving the model a minimum amount of learning before making predictions.

  • Insufficient Gradle metadata from old versions of the build tool. Gradle versions prior to version 5.4 lack metadata that is critical for test selection.

The Simulator indicates the number of tests and amount of serial time where there was insufficient inference data, and the system was forced to select all impacted tests.

Appendix A: Release history

19th April 2022

  • Initial release

Compatible with Gradle Enterprise 2022.2 or later.

Appendix B: Compatibility with Gradle Build Tool and Gradle Enterprise

Compatibility between versions of Gradle, Gradle Enterprise and the Gradle Enterprise test distribution Gradle plugin can be found here.