Problems with flaky networks, build scripts, and multiple environmental factors can make your builds fail in mysterious and unrepeatable ways, slowing or stopping your application deployment production line. Develocity allows teams to find and fix these problems quickly.

You can complete this tutorial in:

  • 1 minute (read the Introduction)

  • 5-10 minutes (read the Introduction and Tour)

  • 15-20 minutes (read the Introduction and Tour and perform the Hands-on Lab)

Introduction

At scale, teams often struggle with mysterious build failures and wonder if certain types of build failures are intermittent, repeatable, or if some unknown environmental factors are changing and those are the root causes of build failures.

Develocity helps teams to find and fix failed tests and builds much more quickly than with other approaches. Instead of having to reproduce the build, tinker with logging levels, hope that the failure will happen again and then copy and paste lots of logs over chat, tickets, email, or log aggregators, build scans provide a fundamentally better way to find the cause of failed builds.

The best practice for deploying Develocity is that all builds send build scans to the centralized Develocity server. CI servers do preserve logs of failed CI builds, but the team is almost entirely blind to failures that occur in developer builds typically. With all builds collected centrally, teams can quickly find specific failed dev or CI builds, compare those builds with a recent or similar passing one, and see what changed. The user interface is optimized for finding failures and rapid analysis by a build/release team member or by the developers themselves.

Teams can aggressively configure all their builds to publish build scans, because the Develocity server (which includes a build scan service and a Build Cache service) is built to scale to the largest teams with ease and does not introduce a single point of failure. If the Develocity build scan service (or Develocity remote Build Cache service) is down, interrupted, or slow, builds proceed and do not fail. Gradle Build Tool supports later sending the last build scan to the server when service is restored.

By instituting a regular and systematic process of monitoring failures and using build scans to measure the results of changing builds to incorporate hardening code, teams will experience fewer unexplained failures, fewer flaky tests, and spend far less time finding and fixing those mysterious failures that used to sap so much of their time and attention away from value adding activities.

Tour: Reduce build failures and increase build reliability

Pull in help for an unexpectedly failing test

Let’s imagine we are a developer, looking at a test failure in our Gradle Build Tool console output, and we don’t know the cause. We want to share the test failure with a colleague.

Let’s first look at the build scan that was posted to our Develocity server by clicking here.

Note that some of the links in this tutorial refer to logically-named build scan ID’s. Typically the build scan ID will look more like a machine-generated ID.

You can see from the summary information that the build failed (the red "x" at the top), that it is the :test task that failed. In the left-hand side navigation, click Tests.

We can see that there are several tests (25 in fact), all from a single project, and that one failed. We can click on the 1 failed link at the top and the display will change to only display a row for tests that failed. Go ahead and do that.

Now click on the row for the single failing test, and you can see the details for that particular failure. The failure is a Java AssertionError and you can also open the stacktrace.

Click to expand the stack trace and examine the URL shown in your browser. You can see it is https://enterprise-samples.gradle.com/example/s/pull-in-help-for-an-unexpectedly-failing-test/tests/a5a6sydx5wc2o?openStackTraces=WzBd

This is a deep link and after the ID of the build scan, we can see that we are viewing the tests section, that we are viewing a particular test by ID, and that we are viewing the stack trace.

We can now share this deep link with the relevant people. Perhaps the developer who checked in the last commit to this file and added the test?

See all locally failing tests across all projects

This example builds upon the last. There we saw how a set of tests from a single project could be displayed and filtered for failures. The Develocity build scan service also aggregates all tests across a multi-project build to be easily handled in the same way.

Let’s look at the build scan here.

Again, click on the left-hand navigation section Tests.

This time we can see that there are 100 tests, across 5 different projects. If you scroll down and view all 100 rows of test entries, you will see several failures, and the red text 14 failed at the top tells us how many failed.

Click on the magnifying glass icon at the top of the table rows.

A search box appears. We can now both filter to show only failed tests, as we did in the previous example, or we can also now search for a specific pattern to filter the rows. Type in get in the search box.

We have found 12 tests of the 100 that match that criteria. Clear the search to return to all 100 tests. We can click on the 5 projects link and see that there are 2 sub-projects without failures, and 3 that do have failures.

And as we did in the last example, we can finish by clicking on 14 failed to see only failing tests and again drill down to details for a single row to see the failure detail and the stacktrace.

Determine if changed dynamic dependencies broke the build

One of the most common root causes for build failures is a changed upstream binary dependency that is specified to resolve dynamically.

Using Develocity’s comparison feature is especially helpful in finding such failures.

Bring up the Develocity scan list by clicking here.

We have recently noticed that the project 08-determine-if-changed-dynamic-dependency-broke-the-build has suffered a failure.

Let’s search for builds of that project, type the first characters of that project name, 08 in the Project field that is in the header of the build scan list display and then click Search.

Now we see two results for this search, one failed, one succeeded. We can see that both took about the same time to complete, and that both were built at almost the same time, so we think it would be profitable to compare these two to see if we can see a change that might explain the failure.

With your mouse all the way to the left side of the first row, click to select the A side of the comparison, then do that again with the mouse on the far-left side of the second row to select the B side of the comparison.

You should see both listed at the top of the list view with a grey background, and with a Compare visual cue now at the far-right side of the rows.

Click now on that Compare cue.

You should now see the build scan differences view. The left-hand navigation side now shows the sections of the build scans with a blue dot annotation for those sections that have a difference.

We notice that dependencies are different, so click on Dependencies on the left-hand navigation bar.

We see that there is a difference, the build that failed resolved the dependency com.acme.utils to version 1.6, while the build that succeeded resolved that same dependency to version 1.5.

This now gives us reasonable suspicion that this is the root cause of the build failure.

Investigate why your project does not compile on your colleague’s machine

One of the most common, and frustrating, and time-consuming types of failures are the "it built for me and didn’t build for you, but everything looks the same" types of failures.

Often such situations result in a lot of time spent but ultimately wasted, because no root cause can be found.

Let’s take a quick look at the failing build scan by clicking here.

Now let’s take a look at the succeeding build scan by clicking here.

It isn’t at all obvious how they differ, we will again use Develocity’s build comparison feature to find the root cause of this failure.

Open the scan list as we did in the previous example by clicking here.

Type in the string ratpack in the project filter at the top of the display to only show scans for the ratpack project.

At the bottom of this list we see two that were built by users Mark and Rene. Click on both on the far-left side of the list view as we did in the last example to set up a comparison, then again click on the Compare cue on the far-right side in the grey area at the top to display the comparison view.

This time we see a blue dot change indicator on the left-hand side area Infrastructure and we think that might be an interesting thing to look at.

What we see there is that these two builds are using different JDK versions, and more so, they are using different JDK vendors. We know that we only run and support the Oracle JDK, yet this one is failing. We can take appropriate steps to ensure the other vendor is not used, and that the issue with this version of the Oracle JDK is fixed.

Reach out for help when local build fails to succeed

Have you ever struggled to determine the root cause of a failed build, only to discover that some intermediate source code change that is uncommitted and untested is the root cause?

We will look at an example of Develocity build scan extensibility to see how elaborating the build scans with custom data can help provide additional context to making root cause determination faster and more effective.

We will look at the mechanics of authoring this type of extension in the customization tutorial.

Here is a build scan for a failure that we suppose someone has asked our help to diagnose. Go ahead and open that.

We could spend a lot of time taking the bait and diving into the test failures we see, however we notice immediately that there is a new link Diff all the way at the top in the build scan header, and instead of diving in, we first will take a look at what that shows and we click the link, and end up in a Github gist.

What we can tell from this is that there were local, un-committed changes to the same tests that are showing as failed, and we can possibly suggest to the submitter that this is why the tests are failing, and perhaps even point to lines of code if we can.

This sort of extension can dramatically reduce debug time and allow teams to more quickly find and fix the root cause of local modifications causing build failures.

Hands-on Lab: Reduce build failures and increase build reliability

Read on to go one level deeper.

Prerequisites

To follow these steps, you will need:

  1. A zip file that includes the source code to recreate the build scans we discussed previously.
    You can get this from here.
    Unzip this anywhere, we will refer to that location as LAB_HOME in what follows.

  2. A local JVM installed.
    JVM requirements are specified at https://gradle.org/install. Note that a local Gradle Build Tool install is optional and not required. All hands-on labs will execute the Gradle Build Tool using the Gradle Wrapper.

  3. An instance of a Develocity server.
    You can request a Develocity trial here.

For the rest of the document, we will assume you have a Develocity instance, available at https://develocity.mycompany.com.

Lab: Add new tests that both pass and fail

Open a terminal window in LAB_HOME/04-see-all-builds-that-ran-tests.

Using the text editor of your choice, modify the build scan configuration in settings.gradle to point to your Develocity server instance:

Replace 
buildScan {
  server = '<<your Develocity instance>>'
  publishAlways()
  // Remove below if you don't use a self-signed or untrusted certificate
  allowUntrustedServer = true
}
with 
buildScan {
  // Your personal Develocity instance
  server = 'https://develocity.mycompany.com'
  publishAlways()
  // Remove below if you don't use a self-signed or untrusted certificate
  allowUntrustedServer = true
}

You will then follow the instructions as written in the file README.md

The README will lead you through running a build, first without executing test tasks, and then a second time, when you do run the test tasks.

You observe this by using a convenience TAG extension that indicates very prominently at the build scan header whether tests were run or not. We will look at how such tags are authored in the extensibility tutorial.

We observe by viewing the second build scan, navigating to the Tests left-hand side navigation section, that we only have one test, and that it passed.

Add 4 more tests by editing the test source code, for example

vi src/test/java/HelloTest.java

And replace the file content with the following:

import org.junit.Test;
import static org.junit.Assert.fail;

public class HelloTest {
	@Test
	public void hello() {
		// Pass
	}
	@Test
	public void foo() {
		// Pass
	}
	@Test
	public void bar() {
		// Pass
	}
	@Test
	public void baz() {
		// Fail
		fail("this is the first simulated test failure");
	}
	@Test
	public void fred() {
		// Fail
		fail("this is the second simulated test failure");
	}
}

View the Tests section of the build scan produced, and you will both see the 4 tests that we added, as well as being able to view the details of the two simulated failures we introduced into the code.

Conclusion

Develocity makes it far easier for teams to find and fix the root causes of all of the complex reasons that builds can fail.

By systematically collecting the details of all build activity for a team, both CI and developer builds, a systematic practice can be put in place that will allow teams to identify trends and emerging issues in code as well as in the environment in which builds run to increase build reliability and reduce the time and frequency of build failures overall.