Caching for faster builds | Develocity Docs

Introduction: Build avoidance

Gradle logically has 3 layers of reuse that allow potentially expensive tasks from being executed:

Up-to-date checks for task outputs in the workspace.
A lookup for task outputs in the local cache.
A lookup for task outputs in a remote cache, such as the remote cache provided out of the box in Develocity.

These 3 levels support making your builds faster in 3 different target scenarios:

Between consecutive runs of a Gradle build by developers, usually not many things change. The Gradle incremental build feature will only execute the tasks that are not up-to-date anymore since they were last executed.
Developers typically maintain many workspaces on many branches to perform logically distinct tasks. The local cache allows outputs to be quickly reused across workspaces and branches without having to transit any networks.
Oftentimes, CI nodes and developers run the same tasks with the same set of changes. The remote cache allows outputs to be reused across users and build agents, ensuring your team never has to build the same thing twice.

Tour: Develocity Build Cache

Let’s dive right into how remote caching can speed up your builds.

Consider a very simple example Java project. Here is the timeline page of a build scan (a complete record of what happened in a Gradle build) for a clean build of this project that rebuilt all tasks in the workspace with no benefit of caching. We can tell this by looking at the task timeline and seeing that none of the tasks say FROM-CACHE. The build, a simple contrived example, takes about 8 seconds to execute tasks, mostly to compile Java sources.

Click on the task :compileJava to see that the Build Cache result was a miss, followed by a store. We recompiled, and then stored the output in the remote cache.

We can also look at the Build Cache performance page of the build scan to see that this build enjoyed no cache hits, but it did store 3 outputs in the remote cache to the benefit of subsequent builds.

We can also see that since almost all of the time taken for this build was spent on compiling Java sources, and we stored that output, we would expect builds that can use this output via a cache hit to be much faster, saving developers time.

Clean build with outputs taken from the remote cache

Having stored the output of the compileJava tasks, we would expect that subsequent builds that use the Build Cache would not have to recompile the source.

Here is the timeline page of a build scan for a second build, run after the first one and configured to pull from the Build Cache. It could have been run by the same developer in a different or clean workspace, a different developer, or a CI build that has pulling from the Build Cache turned on.

We can immediately see that all the compilation and tests tasks are annotated FROM-CACHE, and that the build only took about a half second to complete. Using cached outputs has saved most of the time of the build.

We can also examine the Build Cache specifically on the Build Cache performance page, which shows 3 hits to the remote cache along with detailed information about the cache lookups and transfers.

Real builds often take many minutes or even hours to run. Often builds have to do the same work over and over. This is true of complex CI pipelines down to individual developer builds. Using the Develocity Build Cache saves time, both in developer hours and required CI infrastructure. When we multiply the savings by the number of developer and CI builds per day, the savings are immense.

Faster builds mean developers can run more builds per day, find issues more quickly, and deliver changes more efficiently.

The result is an exceptionally productive team that can deliver more, faster, at lower cost.

Configuration example

We can achieve a high cache hit rate by having CI builds early in the build pipeline push task outputs to the remote cache, and downstream CI builds as well as developer builds pull task outputs from the remote cache.

In the diagram below, you can see the flow of CI agents pushing to the remote cache, and developers pulling from the remote cache.

A task runs on a CI server. The build is configured to push to a configured remote cache, so that task outputs can be reused by other CI pipeline builds or developer builds.
A developer runs the same task with a local change to a file. Gradle tries to load the task output from the local then the remote cache, misses, and the task executes. The task generates the task output which is stored both in the workspace and in the configured local cache. Outputs stored in the local cache can be reused in other workspaces local to that developer’s machine, or in the same workspace, even after running the clean task.
A second developer runs that task without any local changes from the commit that CI built. This time the remote cache lookup is a hit, the cached task output is downloaded and directly copied to the workspace and the local cache, and the task does not need to be executed.

Hands-on Lab: Develocity Build Cache

Read on to go one level deeper and run builds to recreate the cache examples you viewed above.

Prerequisites

To follow these steps, you will need:

A zip file that includes the source code to recreate the build scans we discussed previously.
You can get this from here.
Unzip this anywhere, we will refer to that location as LAB_HOME in what follows.
A local JVM installed.
JVM requirements are specified at https://gradle.org/install. Note that a local Gradle Build Tool install is optional and not required. All hands-on labs will execute the Gradle Build Tool using the Gradle Wrapper.
An instance of a Develocity server.
You can request a Develocity trial here.

For the rest of the document, we will assume you have a Develocity instance, available at https://develocity.mycompany.com.

Open a terminal window in LAB_HOME/01-using-the-gradle-enterprise-build-cache.

Using the text editor of your choice, modify the build scan configuration in settings.gradle to point to your Develocity server instance:

Replace

buildScan {
  server = '<<your Develocity instance>>'
  publishAlways()
  // Remove below if you don't use a self-signed or untrusted certificate
  allowUntrustedServer = true
}

with

buildScan {
  // Your personal Develocity instance
  server = 'https://develocity.mycompany.com'
  publishAlways()
  // Remove below if you don't use a self-signed or untrusted certificate
  allowUntrustedServer = true
}

You will also need to modify the remote Build Cache configuration in the settings.gradle file, with your Develocity Build Cache service location:

Replace

buildCache {
  local {
    enabled = false
  }
  remote(HttpBuildCache) {
    push = true
    url = '<<your Develocity instance>>/cache/'
    // Remove below if you don't use a self-signed or untrusted certificate
    allowUntrustedServer = true
  }
}

with

buildCache {
  local {
    enabled = false
  }
  remote(HttpBuildCache) {
    push = true
    // Your personal Develocity instance's Build Cache access point
    url = 'https://develocity.mycompany.com/cache/'
    // Remove below if you don't use a self-signed or untrusted certificate
    allowUntrustedServer = true
  }
}

Clean build

At this point, the build is fully configured and ready to run. We will run this build with a clean, which will remove all task outputs from the workspace, execute all tasks, and then push all cacheable task outputs to the remote cache in your Develocity instance.

Here is how this standard Gradle build will now interact with the configured Develocity instance:

The local cache is disabled (according to settings.gradle).
Each cacheable task does a lookup in the Develocity server to see if there is a match between the cache key computed as a hash of task inputs with a matching element in the remote cache, at task execution time.
For a cache hit, task outputs are copied to the workspace, instead of rerunning the task.
For a cache miss, task outputs are rebuilt, and a cache key is generated and copied into the remote cache (since push is set to true for this cache in the settings.gradle file).
At the end of the build, Gradle sends the captured build data to the Develocity instance. A unique URL pointing to the generated build scan is printed on the command line. This build scan contains very fine-grained details of all aspects of cache operations, and more.

The Develocity Build Cache has an administrative page that shows cache statistics, that can be viewed from https://develocity.mycompany.com/cache-admin. At this point, assuming your Develocity instance is fresh, you should see all zeros on this page.

We will now go ahead and run our first clean build which will remove all task outputs, rebuild them all, and then view the build scan and cache admin pages. Perform the clean build with

$ ./gradlew clean check --build-cache

Your output should look something like:

$ ./gradlew clean check --build-cache

BUILD SUCCESSFUL in 7s
5 actionable tasks: 5 executed

Publishing build scan...
https://develocity.mycompany.com/s/annggkicyjzro

If you follow the build scan link, you will see an identical build scan to the first build scan discussed in the earlier section, and you will be able to see that all the tasks rebuilt, that the build had 3 cache misses, and that 3 artifacts were written to cache.

At this point, if you revisit the cache admin page at https://develocity.mycompany.com/cache-admin, you will see 3 artifacts now stored, along with the 3 cache misses.

Clean build with outputs taken from the remote cache

We will now go ahead and run our second clean build which will remove all task outputs from the workspace, but now experience cache hits, and then view the build scan and cache admin pages. Perform the second clean build with

$ ./gradlew clean check --build-cache

Your output should look something like:

$ ./gradlew clean check --build-cache

BUILD SUCCESSFUL in 1s
5 actionable tasks: 1 executed, 3 from cache, 1 up-to-date

Publishing build scan...
https://develocity.mycompany.com/s/juoqa37xwy2tw

If you follow the build scan link, you will see an identical build scan to the second build scan discussed in the earlier section, and you will be able to see that 3 tasks reused outputs that were cache hits. Also note that this second build completed in 1 second (versus 7 for the first).

At this point, if you revisit the cache admin page, you will see a record of the cache hits.

Conclusion

To summarize, utilizing Develocity’s Build Cache can result in dramatic improvements in build times for all your builds, returning time to your team to do more work and deliver more valuable features to your users more quickly and efficiently, which saves you time and your organization money.