Diagnosing Build Cache misses with Develocity task inputs comparison

About task inputs and cache misses

Gradle’s Build Cache functionality works by re-using outputs from tasks executed with the same set of inputs. Before a task executes, each input is hashed in order to calculate an overall cache key. Gradle then checks whether outputs are available from the local or remote cache for this key and reuses them if so, which is referred to as a cache hit. If there are no outputs available for the cache key, the task executes, which is referred to as a cache miss.

A significant aspect of optimizing a build for Build Cache acceleration is reducing the scenarios in which cache misses occur. However, as builds become more complex it is not always immediately obvious what caused a cache miss. Develocity task inputs comparison visualizes differences in inputs between tasks of two builds which can be used to identify the cause of a cache miss.

Gradle relies on tasks declaring the nature of their input properties, which is typically done via annotations on fields or methods of the task’s implementation class. An understanding of how this is done and how Gradle interprets different kinds of inputs can be very useful in diagnosing more complex Build Cache misses that are due to misconfiguration of a task. The Gradle User Manual provides detail on this topic.

Creating a build comparison

Within Develocity, a build comparison can be created from any two Build Scans. A comparison can be created from the Build Scan search page by selecting the scans to compare by clicking the and icons that appear when hovering over search results.

Alternatively, from an individual Build Scan, click the icon in the top right to return to the scan search page with the current scan selected as part of the comparison to create.

If you wish to compare with another build that was executed soon before or after from the same location (i.e. same directory on the same machine), you can use the button in the top toolbar to select it for comparison.

The build comparison user interface uses the concept of an and build. The colors associated with each build are used throughout to visualize the nature of differences.

Several aspects of the build are compared, such as the dependencies used and the environment of the build. For the remainder of this tutorial, the focus will be on the “Task inputs” comparison.

Determining the builds to compare

In order to diagnose a Build Cache miss, it is necessary to create a comparison between the build where the unexpected miss occurred, and a build that either produced the cache entry that was expected to be used or that had a cache hit for the task in which the unexpected miss occurred. In determining the build to compare against, it is often very helpful to be able to find related builds in terms of version control revision (e.g. Git commit ID) or CI build types. It is strongly recommended to customize your Build Scans to include such identifying attributes.

These attributes can then be used as search criteria to find the relevant build to compare against, as in the following example:

Comparing task inputs

The task inputs comparison works by identifying tasks executed in both builds, with matching task paths (e.g. :compileJava is compared to :compileJava). Out of the matched tasks, those with differences in inputs are displayed. Matched tasks with the same inputs are omitted. Tasks only executed in one of the builds are also omitted. The tasks are listed in the order that they were executed in the build, with tasks executed earlier listed before tasks that executed later.

For each task listed, the inputs are shown along with the computed cache key and task outcome. The cache key is shown regardless of whether build caching was used for the build or whether the task is cacheable. Inputs that are unchanged are omitted, except for the task’s class name.

As previously stated, there are logically two parts to a task’s inputs: its implementation and its properties. Changes to properties are more often the cause of unexpected cache misses. There are two types of properties: values (e.g. compiler settings) and files (e.g. source files).

Value properties

A “value property” is a task input property that is not a file or set of files. Task inputs comparison displays any value inputs whose value changed between the two builds (i.e. unchanged value inputs are not shown). Build Scans capture a hash of the value for comparison, not the value itself, so therefore do not show the actual values of the value input in either builds.

The following example indicates that the value of the sourceCompatibility value property changed, indicated by the and indicators next to the property.

Hover over the

and

indicators in a comparison for confirmation of what the indicators mean if unsure.

Some properties have names such as options.encoding, which effectively refers to the encoding property of the options property. In this case options is a complex object used as an input with properties of its own, that was annotated with @Nested as part of the task implementation. See the discussion of nested inputs in the Gradle User Manual for more information.

Less frequently, the change is that the property is only present in one of the builds. This can occur when inputs are registered dynamically using the runtime API for declaring inputs.

The following example shows a task inputs comparison where three value inputs have changed, with one unique to each build and one whose value changed:

File properties

Most tasks within a build operate on files, and these files are the most frequent source of changes. Moreover, many tasks operate on files produced by other tasks which makes manually reasoning about differences in order to diagnose Build Cache misses more difficult. Task inputs comparison helps here by showing added, removed and changed files.

Enabling per-file comparison

By default, Build Scans do not capture the information necessary to compare file inputs on a per file level. Capturing the hash of each input file increases the amount of Build Scan data transmitted to and stored by Develocity, so is disabled by default. This increased cost however is negligible for all but very large builds. Please see the Develocity Gradle Plugin User Manual for information on how to enable capture of task input files for comparison, and for more information on the implications.

Comparing file properties per-file requires both builds to have capture of task input files enabled. If either build did not have it enabled, only whether there was any change at all will be displayed for a file property. The following example shows a change to a file property where capture of input files has not been enabled for one of the builds:

Whereas in the following example it has been enabled for both:

When using the Develocity Build Cache, it is strongly recommended to enable capture of task input files so that per-file comparison is available for builds when unexpected cache misses occur.

Understanding file changes

At a property level, changes in file properties are visualised in the same manner as value properties. Properties that have both the and indicators are present in both builds but have changed, and properties that have not changed are not shown.

Build Scans capture a hash of each file’s content and not the content itself. Task inputs comparison shows which files have changed, but not the change(s) within files. Identifying the changes within a file requires having access to the files used by the builds and comparing them via some other means such as a text diffing tool.

File properties are expandable. If the property is present in both builds but has changed, and if per-file comparison is available, the file-level changes are shown.

At the top level under each property, the top level files or directories that were configured for the property are shown. If it is a directory, and that directory was used in both builds, the changes to its files are shown. Unchanged files are not shown.

A file or directory that exists in both builds but that has changed is displayed with both the and indicators. Files or directories that only exist in one of the builds are shown with only the indicator for that build. Hovering over the indicator(s) reveals a tooltip confirming the status of the file or directory.

Files and directories existing under the root project’s directory are displayed with paths relative to this directory. Note that this is not relative to the task’s project, but to the build’s root project. Files that are used from Gradle’s dependency cache are visualised specially as their dependency coordinates (i.e. group:name:version) and the filename. For any file or directory not shown as an absolute path, hover on the displayed name or path to see the absolute path in the builds it exists in.

You may encounter property names such as $1 or $2. These are properties that were registered without a name, via the runtime API for registering inputs. Where possible, the code registering such inputs should be updated to provide a meaningful name for the property.

Normalization

The concept of “normalization” refers to different strategies for how file inputs should be hashed. For example, is the absolute path of the file significant or only the relative path? Which files, if any, should be ignored? How should the contents of files be hashed? The purpose of normalization is to refine the hashing strategy so that aspects of the input files that do not affect the task’s outcome are ignored, leading to more cache hits.

When expanded, the normalization strategy used for each file property is shown. The help icon next to the stated normalization for each property provides a brief description of the strategy as a tooltip when hovered on.

Normalization is a complex topic which typically is not necessary to understand deeply when debugging cache misses with task inputs comparison. The most common issue related to normalization is the incorrect use of “absolute path” normalization, which is the default. This means that paths to files must match exactly between two builds for there to be a cache hit, which is usually not necessary as most tasks do not care about the absolute path to a file. It is often the case that “relative path” normalization should be used instead.

Details about the different forms of normalization and their implications can be found in the Gradle Build Cache Guide.

Ordering changes

The “compile classpath” and “runtime classpath” normalizations are sensitive to ordering, whereas all of the other normalizations are not. This means that for these normalization strategies, changes in only the order of the files results in a cache miss, as classpaths are in practice ordered (classes earlier in the classpath take precedence over classes with the same name later in the classpath). When the change is that the order of files is different, the visualization changes to show all of the files in both builds and their effective order.

Ordering changes are most frequently due to changes to the order in which dependencies are declared in build scripts.

If your build assembles classpaths via means other than dependency management, ordering changes may occur due to non-determinism in how the classpaths are assembled. For example, the order in which files are listed by the operating system when building a classpath from all of the JAR files in a directory may not always be the same.

Implementation

The implementation of the task is itself also an input to the task, as all of the code that is executed when the task is executed can potentially affect the output of the task. Cache misses due to changes in task implementation are less frequent than changes in properties, but do occur.

If the task implementation changed between the two builds, the task class name is highlighted as in the following example:

Gradle considers the task implementation to be the task class itself and all classes visible to the task class. Gradle is unable to, ahead-of-time, determine precisely the exact set of classes or code that a task uses. Therefore, all of the code that the task could potentially use must be considered, which is all of the code visible to the task’s classloader.

The significance of all visible classes being considered may not be immediately obvious. The implication is that changes to seemingly unrelated classes to a task’s implementation affect the cache key of the task. For example, if a task comes from a plugin that is loaded using the plugins {} block, its cache key will change if another plugin is added, removed, or upgraded. Similarly, for a task being loaded from a buildSrc project, its cache key will change if any change is made to any code in buildSrc.

Task inputs comparison does not indicate precisely how the task implementation changed, only that it did change. Such changes are usually easy to reason about and find the root cause of with some knowledge of the build structure, that can be gleaned from a Build Scan.

Task actions

Task actions are also considered as part of the implementation, and are considered in the same manner. That is, a hash is calculated from all classes visible to the task action.

The following example shows a comparison in which the implementation of a task’s action changed:

This kind of change most commonly occurs when a build script that adds an extra action to a task has been changed.

Rarer still is the case in which one build has an action that the other build does not have at all. This can occur if the action is added conditionally, for example to only augment the behavior for CI builds. The following example visualises this:

Conditionally adding actions to a task should be avoided. Instead, the action should always be added and the condition be internalized within the action with any data it uses in evaluating the condition be declared as an input. This makes the task inputs comparison much more accurate and descriptive.

Detecting changes from task dependencies

Many tasks use the output of other tasks as input. For example, test tasks use the output of compile tasks. Often a cache miss is noticed for such a task, where the change that caused the miss was a change in the outputs of another task. In such a case, diagnosing the miss requires determining the root cause of the miss. Or in other words, determining the upstream change that propagated downstream.

Task inputs comparison does not directly identify inputs that are the outputs of other tasks. This can often be inferred however based on the names and paths of the files, and some knowledge of the build structure. Additionally, the task inputs comparison interface orders tasks in execution order. This means that depended on, or upstream, tasks are always shown before dependent, or downstream, tasks.

The following example shows an upstream change affecting a downstream task:

In this example, a cache hit may have been expected for the :test task. The task inputs comparison tells us that the change between these two builds for this task was the submodule.jar file. Based on the path to this file, and that the :submodule:compileJava task is also shown to have had changed inputs, it can be reasoned that the root cause change is within :submodule.

Additional resources

The Using the Build Cache guide is very strongly recommended to anyone working with the Build Cache and in particular anyone debugging cache issues and optimizing cacheability. It discusses some common problems that lead to Build Cache misses and strategies for dealing with them.