Using the Build Cache for sbt

The outputs of a task are the results it produces when executed. Its inputs are the results of the tasks and settings it depends on. For example, for the compile task of sbt, all Scala and Java source files are inputs as well as all configuration options (such as compiler flags) that influence the resulting class files, i.e. its outputs.

Artifacts in the Build Cache are uniquely identified by a Build Cache key. A Build Cache key is assigned to each cacheable task execution when running with the Build Cache enabled and is used for both loading and storing outputs of task executions to the Build Cache. The inputs that contribute to the Build Cache key of a task execution are the outputs of the tasks and settings that the task depends on. Two task executions can reuse their outputs by using the Build Cache if their associated Build Cache keys are the same.

A task execution is said to have reproducible outputs if it will always generate the same outputs given the same inputs. Some tasks add extra information to their output that doesn’t depend on their inputs, e.g. a code generator might add a timestamp to the generated files. In such a case, re-executing the task will result in different outputs. Consequently, tasks that use these outputs as their inputs will need to be re-executed.

When a task is cacheable the very nature of task output caching ensures that its executions will have the same outputs for a given set of inputs. Therefore, cacheable tasks should have reproducible outputs. Otherwise, the result of executing the task and loading its outputs from cache may be different, which can lead to hard-to-diagnose cache misses.

The outputs of a task can only be loaded from cache if it has stable inputs. Unstable inputs result in frequent, unnecessary cache misses. For example, compiling tests depends on the result of compiling production code. Thus, in order for tasks to have stable inputs, the tasks they depend on should have reproducible outputs.

While we acknowledge that creating outputs that contain volatility (such as build timestamps) is a common practice for sbt builds, we see this as an antipattern because they drastically reduce the probability of cache hits. If, for example, one project in a multi-project build generates a build timestamp, the Build Cache has to assume that this timestamp is used by downstream projects. Therefore, all of them have to be rebuilt even if the timestamp is not actually used.

Timestamps in particular are of dubious significance. What does a timestamp tell us about the origin of the artifact? Does it help us to track it back to the CI job that created it? The answer is almost always "no", because the time the CI job was started is usually not identical with the timestamp the artifact was generated. Instead, you should use something that uniquely identifies the code that was used to produce the artifact. A good candidate for this is the SCM revision number or commit ID.

The Develocity sbt plugin tracks all known inputs of the supported tasks. However, sometimes, a task may read additional inputs. For example, your tests might read external files that aren’t in the classpath. To make caching work correctly, you need to specify these additional inputs.

Let’s start off by running the build for the first time:

When a task is executed, the Develocity sbt plugin first checks the local Build Cache for stored build results that may be reused. If no result is found in the local Build Cache, the remote Build Cache is queried. If neither provides a result, the task is executed and the output is stored in the local Build Cache. Since we just activated the Build Cache for the project, the local Build Cache as well as the remote Build Cache are empty and all tasks are executed.

When we run the build a second time with a populated local cache, the build results of cacheable tasks should be retrieved from the cache.

When we run the build another time, the build results of cacheable tasks should be retrieved from the cache:

This time, all results were loaded from the local Build Cache and no compilation task was executed.

In order to identify which inputs changed between builds, we can enable verbose build cache logging. This additional logging will make it easy to identify the changing inputs.

To enable verbose build cache logging, set the system property: develocity.internal.cache.verbose=true.

To make the build cache log messages more visible, you can also set the system property develocity.internal.cache.defaultLogLevel to the log level you want to see, such as info for instance.

Combining the two system properties, the command to run the build with verbose build cache logging would look like this:

With these additional system properties, Develocity will log the inputs of each task that attempts to retrieve its result from the Build Cache:

By comparing the hashes between two executions, you should be able to identify which inputs changed between the two builds. Modifying these inputs so that they produce a stable hash will fix the cache misses.

Last but not least, we need to ensure that the cache works even across machine boundaries. First, we’ll need to push some outputs to the remote cache, so we can use them from another machine:

Now we can log into another machine, maybe even one using another operating system, and check out the same commit of our project. Building it should retrieve all task outputs from the remote cache:

If your project does not retrieve its outputs from the remote cache, follow the steps above to find the changing inputs.

Once you’ve verified cache effectiveness on your own machine, you’ll probably want to allow a few other colleagues to try it out, without affecting everyone else on the team. You can do this by disabling the cache in the project’s build.sbt, unless the environment variable DEVELOCITY_CACHE_ENABLED is set.

and then letting your early adopters re-configure their shell in ~/.profile or ~/.zprofile:

Make sure that your early adopters are seeing the same local Build Cache hit ratio that you had in your own experiments.

For your CI builds, changing settings in the user home is probably not an option, as that may affect other projects. Instead, you can modify the above snippet to enable the remote cache if the CI environment variable is set (or another environment variable that your CI system sets):

At first, you may want to do this for a dedicated test pipeline, until you have convinced yourself that caching works well enough to roll it out to your main pipeline.

You can also use this configuration file to enable storing in the remote Build Cache, so that later builds can benefit from the outputs that your CI agents created.

Your CI builds will now populate the remote Build Cache. Your local builds should now get cache hits whenever they execute a task that has already been executed on CI with the same inputs. Make sure this works well for all your developers.

Many projects have a pipeline with multiple stages, with many steps running in parallel. In order to get the most out of the Build Cache, we recommend running

as your first pipeline stage, so that all subsequent stages can reuse the compiled production and test code.

If you are using ephemeral CI agents, the local Build Cache will not give you any benefit, since it disappears together with the build agent. You can disable it to save some build time in this case.

The effectiveness of using a remote Build Cache is largely dictated by the network latency between the build and the cache. Develocity provides a built-in cache node at https://develocity.mycompany.com/cache. This is the node where outputs will be stored and loaded by default. You can install additional nodes and connect them with Develocity. See the Build Cache Node User Manual for more details. Using a Build Cache node that is closer to where the builds are run can significantly reduce build times.

Each team member should configure the closest node in their shell ~/.profile or ~/.zprofile:

And modify you build to use this value:

Once you have convinced yourself that caching is working well for both your CI and local builds, you can remove the disable-by-default configuration from your project, so the cache is used by everyone.