Develocity Predictive Test Selection User Manual

Develocity develops a predictive model by observing code changes and test outcomes from your Build Scan data. It combines ongoing learning about your project structure and tests with training from millions of test executions across many projects in order to make accurate predictions.

A snapshot of the test suite and the code-under-test is cross-referenced with change and test result history to predict the subset of relevant tests, which is typically a small portion of the entire test suite.

As each new Build Scan is uploaded to Develocity, the predictive model is updated based on the new results.

(Develocity 2023.2+)

The selection profile influences the number of tests that will be selected for execution by Predictive Test Selection. Develocity offers three predefined selection profiles: Conservative, Standard, and Fast. The selection profile can be configured for each Gradle test task or Maven test goal.

The Standard profile is the default and provides the same experience as previous versions of Develocity, where the selection profile was not configurable. It aims to strike a good balance between saving time and selecting relevant tests.

The Conservative profile favors increasing the likelihood of selecting all tests that provide useful feedback, which often requires selecting more tests, which reduces time savings compared to the Standard profile.

The Fast profile favors saving time by using a higher selection threshold, which often selects fewer tests and saves more time than the Standard profile.

The Predictive Test Selection Simulator shows the simulated results of the different selection profiles, making it easy to assess the impact of the different profiles.

As flaky tests are inherently unpredictable, Predictive Test Selection will always select recently flaky tests.

To maximize the effectiveness of Predictive Test Selection, it is recommended that Develocity Test Analytics be used to identify and prioritize the stabilization of flaky tests in your project.

While testing frameworks may skip a test for given inputs in one build, they might not skip the test again in a different build. For example, JUnit’s assumptions may evaluate differently for the same inputs, causing the test to be run in one build and skipped in another.

Therefore, Predictive Test Selection always selects tests that were skipped for the same inputs.

(Develocity 2023.2+)

By default, Predictive Test Selection enabled builds will select the most relevant subset of tests for execution for the given changeset and prior test execution/change history to provide the fastest feedback possible. However, it’s advisable to run the remaining tests in a subsequent stage of the CI pipeline to ensure that the changeset is tested comprehensively. Therefore, the Develocity Gradle plugin and Maven extension provide a “remaining tests” selection mode (see instructions below for Gradle and Maven). A typical usage pattern is to use the “relevant tests” selection mode for early and frequent QA stages, such as local pre-commit checks and pre-merge CI jobs, and the “remaining tests” selection mode for later and less frequent extended coverage QA stages, such as nightly or pre-release builds, to ensure that all tests are ultimately executed. This avoids executing any given test more than once for a given change.

An essential aspect of modern software development is testing early and often during the development process instead of at the end, known as “shift-left testing”. However, a common approach is to have fast tests, such as unit tests, execute early in the lifecycle of a change and defer all long-running tests such as integration or end-to-end tests. Despite such tests being very good at detecting defects, they are often delayed until later in the development process due to being slower than unit tests. Predictive Test Selection allows a more productive approach where the most valuable subset of integration and end-to-end tests are also “shifted left”.

By executing the most relevant subset of long-running integration and end-to-end tests run early during the lifecycle of a change, Predictive Test Selection gives developers the high-quality feedback such tests provide quickly.

Tests must be run via JUnit Platform, which is part of JUnit 5. Many popular test frameworks are compatible with JUnit Platform and are supported by Predictive Test Selection, such as JUnit (including JUnit 3 and JUnit 4), Spock, Kotest (version 5.6.0 and later), TestNG, jqwik, ArchUnit and more. Cucumber tests are also supported, but they do require some extra configuration via the Cucumber-Companion plugin. Please see Gradle Testing in Java & JVM projects guide or Maven Surefire’s documentation for guidance on configuring your build to use JUnit Platform. When in doubt, please contact Develocity support to assess whether your setup is compatible with Predictive Test Selection.

Tests and sources authored in popular JVM languages are supported, including Java, Kotlin, Groovy, and Scala.

Some JUnit Platform test engines, such as Spek, are currently not supported. Please contact Develocity support if you are interested in using this or other incompatible test engines with Predictive Test Selection.

When Predictive Test Selection is enabled, only a subset of tests may be executed, resulting in a lower code coverage than usual. If you rely on code coverage measurement, we recommend enabling it only for builds with Predictive Test Selection disabled to get consistent values.

Develocity Predictive Test Selection is compatible with Gradle 5.4 and later, and works with Gradle’s built-in Test task and its subclasses. This includes Android unit test tasks, but does not include most Android device tests.

Tests must be run with Java 8 or later.

Tests from the buildSrc project and from included builds are currently not supported.

Develocity Predictive Test Selection is compatible with the Maven Surefire plugin 2.22.2 and later, and Maven Failsafe plugin 2.22.2 and later.

Tests must be run with Java 8 or later.

The Predictive Test Selection Simulator requires Develocity 2022.1 or later, and the Develocity Predictive Test Selection extension. Please contact your customer representative to enable this extension.

It requires that builds are configured to capture task/goal input files as part of Build Scans. For Gradle, see Capturing task input files for Gradle. For Maven, see Capturing goal input files for Maven.

The overview metrics and charts help you quickly estimate the expected benefits and risks of enabling Predictive Test Selection for selected builds.

Predictive Test Selection requires Develocity 2022.2 or later, and the Develocity Predictive Test Selection extension. Please contact your customer representative to enable this extension.

After enabling the extension, recent build data must be processed to construct the prediction model. Depending on the amount of build data available, this process may take up to 3 days. For most installations, this processing is able to complete overnight.

The “Use Predictive Test Selection” permission is required to use Predictive Test Selection in a build. If anonymous access is disabled for viewing Build Scans, builds must authenticate with Develocity via an access key that has the "Use Predictive Test Selection" permission enabled. See below for build tool-specific instructions.

Builds must be configured to publish Build Scan data with task/goal input files capturing enabled. See below for build tool-specific instructions.

Each Build Scan contains a summary and full details for test runs in a single build.

The count and estimated serial time savings is reported on the Summary view.

Navigate to the “Tests” view to see a structured and searchable list of tests. You can open the tasks/goals inspectors and the test class inspectors to view additional information about Predictive Test Selection usage.

The details show estimated saved serial time based on recent executions, as well as a logical prediction explanation for selected and not selected test classes. If your Develocity instance has the Test Analytics extension enabled, you can view the test history to inspect recent test outcomes, including “NOT SELECTED” results.

Test selection results from all test runs are stored in Develocity, and can be aggregated, filtered, and analyzed using the Predictive Test Selection Dashboard.

The Usage view of the dashboard visualizes the usage of Predictive Test Selection and estimated savings trends, as well as a summary for matching tasks/goals. Here you can see exactly which tasks/goals enabled test selection, how often, and the impact of doing so. This allows you to understand how test selection is being used and the resulting impact.

You can use the search or click on any task/goal to view trends for the specific task or goal, and to see details for individual test runs which enabled test selection.

Clicking any test run navigates to the associated Build Scan, where you can inspect full details of each test, including prediction explanations and estimated savings for each test.

You can also see test runs where Predictive Test Selection was unavailable, that is, selection was attempted but could not be completed for some reason, resulting in all tests being executed. Each list item shows the reason test selection was unavailable so you can quickly identify and respond to problems.

Predictive Test Selection will rarely skip a test which would have failed, which may be caught when all tests are run. The process for investigating such occurrences is similar to git bisect:

Step 1: Analyze the build and test logs in the Build Scan to identify the nature of the failure. Many test failures are caused by an unstable environment. It is helpful to quickly rule out flakiness if possible.

Step 2: Analyze test execution history to identify last known good state. You can view the test history for a specific test by clicking “View test history” when viewing test details in a Build Scan, or via searching the Tests Dashboard. Using custom metadata you can quickly isolate a specific stream of builds (e.g. test runs on a specific branch).

Step 3: Identify the changes from the last passing execution leading up to the failed test. Build comparison between the last passing and failing test builds is a useful way to identify what changed. It provides snapshots at a different granularity, between builds instead of version control commits, which may allow you to more quickly identify the breaking change.

If you identify that Predictive Test Selection did “miss” a test failure, you have the option to add the test class to must-run criteria to be certain it does not happen again. Predictive Test Selection will learn from every test failure, thus making rigorous “correcting” of mistakes unnecessary.

Predictive Test Selection is an extension of Develocity and may not be enabled for your installation. Please contact your customer representative to arrange a trial of the extension if you wish to try it.

After the Predictive Test Selection extension is enabled initially, the Develocity server starts to analyze existing builds and perform simulations. Depending on the amount and size of existing Build Scans, this process can take several hours or days. You can track its progress by adjusting the “Start time” filter on the Simulator. When in doubt, please contact Develocity support.

If your test task/goal fails with an error message similar to the following, you have probably not been executing your tests via the JUnit Platform before enabling Predictive Test Selection.

As indicated by JUnit’s exception message, there needs to be at least one "test engine" (a JUnit Platform concept) registered in order to run tests. If your tests are written in JUnit 3/4, you should add a test runtime dependency on the JUnit Vintage engine. If they use TestNG, you need an additional dependency to the TestNG engine.

If you use JUnit Jupiter, Spock 2.x, or any other supported test framework that runs natively on the JUnit Platform, you should not have to add any dependency.

If some tests start failing when Predictive Test Selection is enabled but they pass when it is disabled, your tests may have dependencies on each other.

Since Predictive Test Selection does not execute all tests, a test which depends on another test that is not selected will fail. Furthermore, tests may be executed in a different order when Predictive Test Selection is enabled. There’s no way to force Predictive Test Selection to execute test in alphabetical order, for example.

If your Gradle invocation filters tests using --tests (or Test.setTestNameIncludePatterns, the programmatic equivalent) or your Maven goal filters tests using -Dtest=…​ or -Dit.test=…​ (or uses the <test> property in the POM), Predictive Test Selection will always select all matching tests to ensure a consistent CLI and IDE user experience.

Take one of the following actions to allow Predictive Test Selection to avoid tests normally:

For Gradle, register a custom task with a TestFilter to split a test suite.

For Maven, use test exclusions or inclusions to split a test suite.

When using Predictive Test Selection, before tests are executed, the Develocity server is contacted to decide which tests should be selected. If the server is unreachable, Predictive Test Selection is deactivated and all tests are executed.

In order to select tests for a build, Develocity needs to have processed enough recent builds. If processing lags behind, Predictive Test Selection is temporarily deactivated and all tests are executed to avoid selecting the wrong set of tests.

If the selection requests to the Develocity server take too long, test selection is disabled and all tests are executed. If you frequently encounter timeouts, please contact Develocity support.

Predictive Test Selection will not attempt to make predictions for test tasks or goals for which fewer than 14 days of code and dependency changes are available. In this case, all tests except those having previously passed for identical inputs are selected and executed. It is possible that file fingerprint capture is not enabled and must be configured for Gradle or Maven.

If anonymous access is disabled for viewing Build Scans, builds must authenticate with Develocity via an access key that has the “Use Predictive Test Selection” permission enabled to use Predictive Test Selection.

In order to make a prediction, Predictive Test Selection needs to capture test task/goal inputs. Such failures can be caused, for example, by Gradle work validation problems.

If task/goal input file capturing is disabled, Predictive Test Selection will not be able to use its outcome for predictions of future builds. This, however, does not affect the prediction for the current build.

For Gradle, see Capturing task input files for Gradle.

For Maven, see Capturing goal input files for Maven.

In order to make predictions, Predictive Test Selection requires data about as many builds as possible. Therefore, we recommend publishing a Build Scan for all builds regardless whether they are running locally or on CI.

See Controlling when Build Scans are published for Gradle and Controlling when Build Scans are published for Maven, respectively.

Tracking additional input files or directories by absolute paths convolutes the change history used by Predictive Test Selection and negatively impacts its predictions. Consider moving or copying such files or directories to a location that can be referenced by relative paths.

If you encounter builds where Predictive Test Selection was disabled due to “Build plugin encountered an internal error” or “Predictive Test Selection encountered an internal error”, please report it to Develocity support.

The machine learning model takes many variables into account when making a decision. For example, which code changes affect tests in the module, how sensitive the test is to changes, the nature of recent changes made, how flaky the test is, and so on.

Tests will always be chosen if they are recently new, recently changed, recently failed, or recently flaky.

Tests which have already passed against the same sources and dependencies are less likely to be selected.

The machine learning model was trained with millions of test executions and dozens of projects from Gradle’s data partners. The training and test sets cover a variety of JVM-based projects such as libraries, API services, Android apps, server apps, and browser-based apps.

Once enabled, Predictive Test Selection will learn from historical and new code changes and test outcomes for your projects.

Yes, you can specify test patterns and annotations (for Gradle and Maven) which force matching tests to be selected and executed.

You can also choose to use a different selection profile to adjust the desired level of trade-off between the PTS system’s confidence (i.e. percentage of correctly predicted failures) vs. savings (i.e. percentage of testing time avoided).

Other inference aspects are not customizable.

A test outcome of “NOT SELECTED” indicates that the test class was skipped because Predictive Test Selection’s machine learning system determined that it is unlikely to provide a useful test signal.

This means that Predictive Test Selection does not have sufficient change history or sufficient test history to make a prediction.

The most common reasons the simulator may report that all test runs have “insufficient data” are:

The Simulator indicates the number of tests and amount of serial time where there was insufficient inference data, and the system was forced to select all impacted tests.

Many builds use JaCoCo or other tools to measure code coverage. Some builds might even fail if the code coverage is below a configured target threshold. When running only a subset of tests as it is the case when Predictive Test Selection is enabled, code coverage might be lower than expected. Therefore, we recommend to only enable code coverage measurement and checks for builds when Predictive Test Selection is disabled.