Develocity 2019.3 introduced a failures dashboard that classifies build failures as non-verification or verification.

This is to help identify and measure unexpected and disruptive failures such as build configuration or infrastructure failures, separately from failures that merely indicate a fault with the software being built such as compilation or test failures.

How failure classification works

Build failures are classified based on the exception message, stack trace, and identifying build information, using a classification model. Although this model was trained using thousands of different build failures, it may make classifications that don’t serve your needs perfectly.

What to do to improve invalid classifications

Develocity 2019.4 improved the classification model to support definitive error message prefix phrases for failure classification, as listed below. The presence of such a prefix overrides all other heuristics and forces the failure to be classified accordingly. Failure exceptions are analyzed starting with the root exception and then each of the encapsulating exceptions in turn. That is, the exception chain is analyzed from the bottom up.

For your custom build logic, where you are in control of the failure exceptions thrown when something goes wrong, you can use one of the below prefixes to force classification.

Forced failure classification is not available in Develocity 2019.3 and earlier, therefore the following prefixes are not guaranteed to produce the desired classification in those versions of Develocity.

Verification failures

This classification is meant for failures that are expected within a standard application development lifecycle. They typically represent a problem with the developer’s inputs to the build such as the source code.

To force classification of a build failure as a verification failure, prefix any exception message with any of the following phrases (case-insensitive) for verification failures.

  • Analysis failed

  • Check failed

  • Compilation failed

  • Code generation failed

  • Generation failed

  • Lint failed

  • Processing failed

  • Test failed

  • Tests failed

  • Testing failed

  • Verification failed

Using any of these will guarantee that Develocity will classify your failure as verification.

For example, the following exception stack would normally be classified as non-verification:

[org.gradle.api.tasks.TaskExecutionException] Execution failed for task ':example-project:build'.
└── [com.example.custom.tasks.ClassDocGenerationException] Could not load the class documentation for class 'com.example.MyClass'.
    └── [java.lang.RuntimeException] Expected 2 td elements in tr, found none

The RuntimeException is thrown by the custom Gradle task logic. If the exception message is changed to “Generation failed: Expected 2 td elements in tr, found none”, it will be classified as verification.

Gradle

Gradle users also have the option of extending org.gradle.api.tasks.VerificationException.

Non-verification failures

This classification is meant for failures that are typically not expected within a standard application development lifecycle, such as build configuration failures, dependency resolution failures, infrastructure failures, and so on.

To force classification as a non-verification failure, prefix any exception message with any of the following phrases (case-insensitive) for non-verification failures.

  • Unexpected error

  • Unexpected failure

Using any of these will guarantee that Develocity will classify your failure as non-verification.

For example, the following exception stack would normally be classified as verification:

[org.gradle.api.tasks.TaskExecutionException] Execution failed for task ':example-project:setupMockServer'.
└── [org.gradle.api.GradleException] Mock server failed to bootstrap correctly

The GradleException is thrown by a custom Gradle task. If the exception message is changed to “Unexpected error - Mock server failed to bootstrap correctly”, it will be classified as non-verification.

Third party failures

Currently, there is no way to customize failure classification in Develocity for third party failures. If you are unable to change the failure messages emitted by your build logic, you are reliant on the built-in classification model in Develocity. If you believe the failure classifier is making a wrong determination, please raise a support issue so that we can improve the classification for a future release.