This guide explains how to configure Bazel builds to create Build Scans with Develocity.

Develocity Build Scans provide observability of build and test results in your web browser. They help collaboratively debug build problems and optimize build performance.

Develocity 2024.1 or later is required to use Build Scans for Bazel builds. To get the full benefits of a Build Scan, your Bazel builds need to be configured to use the remote cache and also have 'write' permission to the remote cache. See Cache Permissions for details.

Bazel Build Scan functionality is not enabled by default with Develocity. If you wish to use Bazel with Develocity, please contact your customer success representative.

Getting set up

To publish Build Scans to your Develocity server, you need to pass a few additional command line options to Bazel. The simplest way to do so is to add the following to the .bazelrc file in your build workspace.

common:develocity --remote_cache=grpcs://«develocity-server»
common:develocity --bes_results_url=https://«develocity-server»/build/
common:develocity --bes_backend=grpcs://«develocity-server»

build --config=develocity

Replacing «develocity-server» with the hostname of your server. If you use a non-standard port for HTTPS traffic, this must also include the port number. The last line makes Bazel publish a build scan for every build. See Controlling when build scans are published for more details.

While not strictly needed, consider adding the following options as well, which increase the level of detail you will see in your builds.

common:develocity --build_event_publish_all_actions=true
common:develocity --noslim_profile

Authenticating

Develocity installations may be configured to require build scan publishing to be authenticated. Additionally, installations may be configured to only allow certain users to publish build scans. If your instance doesn’t require authentication, you can skip to Configuring build scans.

Develocity access keys should be treated with the same secrecy as passwords. They are used to authorize access to Develocity from a build.

  1. Log into your Develocity server, from the upper-right corner click your user icon, then "My Settings".

  2. Ensure that your user has at least "Publish build scans" and either "Read and write build cache data" or "Read build cache data and write Bazel CAS data" permissions. See Cache Permissions.

  3. Select "Access keys" on the left and then generate a new access key.

  4. In a .bazelrc file in your user’s home directory, add the following:

common:develocity --remote_cache_header=Authorization="Bearer «the access key you just generated»"
common:develocity --bes_header=Authorization="Bearer «the access key you just generated»"
If you have an existing access key, it can be reused. Generally, it is considered best practice to use each access key for a single purpose, so that if it is revoked there is no unexpected collateral damage.

This configuration can also be specified on the command line as part of the build invocation, or via other mechanisms that Bazel provides. Please see the Bazel documentation for more information.

At the beginning and end of your build, you will see output similar to:

INFO: Streaming build results to: https://«develocity-server»/build/66e6805b-7ce1-4e32-bc26-c0fa129c76a7

The link shown is your Build Scan® link and can be used to observe the results of the build.

Short-lived access tokens

Develocity access keys are long-lived, creating risks if they are leaked. To avoid this, users can use short-lived access tokens to authenticate with Develocity. Access tokens can be used wherever an access key would be used. Access tokens are only valid for the Develocity instance that created them.

Develocity server version 2024.1+ supports access tokens.
Changing a Develocity instance’s hostname will cause all existing access tokens to become invalid.

To create an access token:

  1. Get an access key or access token for the user you wish to create a token for.

  2. Decide which permissions the new access token should have.

  3. If project-level access control is enabled, decide which projects the new access token should be able to access.

  4. Decide how long the new access token should live.

  5. Make a POST request to /api/auth/token, optionally with the following parameters. The response will be the access token.

    1. A permissions= query parameter with the config values of each permission you wish to grant. By default, all permissions for the credential used to authenticate the token request are granted to the created token.

    2. If project-level access control is enabled, a projectIds= query parameter with the ID of each project you wish to grant access to. By default, all projects for the credential used to authenticate the token request are granted to the created token.

    3. An expiresInHours= query parameter with the token’s intended lifetime in hours, with a maximum of 24. The default is two hours, or the remaining lifetime of the credential used to authenticate the request, whichever is smaller.

The requested permissions and project ids can be specified as comma-seperated lists or repeated parameters. For example, ?projectIds=a,b&projectIds=c is valid and will request projects a, b, and c.

If project-level access control is not enabled, all access tokens will be granted the “Access all data without an associated project” permission even if it is not explicitly requested.

If the user creating the token does not have one of the requested permissions or projects, Develocity will respond with a 403 Forbidden error. If an access token is used to authenticate the creation request, its permissions and projects will be used for this check instead of the user’s. The request will also error if the requested lifetime would cause the new access token to expire after the one used to authenticate the request. Together, this means you cannot create an access token with more access or a later expiration than the credentials used to authenticate the request.

See the API documentation for more details on the /api/auth/token endpoint.

Here is an example using CURL to create an access token:

$ curl -X POST https://ge.mycompany.com/api/auth/token?permissions=publishScan,writeCache,accessDataWithoutAssociatedProject&projectIds=project-a,project-b&expiresInHours=1 \
    -H "Authorization: Bearer 7asejatf24zun43yshqufp7qi4ovcefxpykbwzqbzilcpwzb52ja"

eyJraWQiOiJ0ZXN0LWtleSIsImFsZyI6IlJTMjU2IiwidHlwIjoiSldUIn0.eyJpc19hbm9ueW1vdXMiOmZhbHNlLCJwZXJtaXNzaW9ucyI6WyJSRUFEX1ZFUlNJT04iLCJFWFBPUlRfREFUQSIsIkFDQ0VTU19EQVRBX1dJVEhPVVRfQVNTT0NJQVRFRF9QUk9KRUNUIl0sInByb2plY3RzIjp7ImEiOjEsImIiOjJ9LCJ1c2VyX2lkIjoic29tZS1pZCIsInVzZXJuYW1lIjoidGVzdCIsImZpcnN0X25hbWUiOiJhIiwibGFzdF9uYW1lIjoidXNlciIsImVtYWlsIjoiYkBncmFkbGUuY29tIiwic3ViIjoidGVzdCIsImV4cCI6NzIwMCwibmJmIjowLCJpYXQiOjAsImF1ZCI6ImV4YW1wbGUuZ3JhZGxlLmNvbSIsImlzcyI6ImV4YW1wbGUuZ3JhZGxlLmNvbSIsInRva2VuX3R5cGUiOiJhY2Nlc3NfdG9rZW4ifQ.H1_NEG1xuleP-WIAY_uvSmdd2o7i_-Ko3qhlo04zvCgrElJe7_F5jNuqsyDfnb5hvKlOe5UKG_7QPTgY9-3pFQ

The resulting token would have the following permissions:

  • “Publish build scans”

  • “Read and write build cache data”

  • “Access all data without an associated project”

And it would have access to these projects:

  • “project-a”

  • “project-b”

The token would only be usable for one hour.

Using a Bazel remote cache

Bazel Build Scans provide more information when the build uses a remote cache that is also accessible by the Develocity server.

Bazel uploads build artifacts, test results and diagnostic information to the remote cache. When Build Scans are published to Develocity, such diagnostic information is read from the remote cache and included in Build Scans. This allows Build Scans to provide more information about the build. In particular, action profile information and detailed test results are included.

When viewing Build Scans, the build artifacts and diagnostic logs of targets and failed actions can be downloaded from the user interface if the remote cache is accessible.

If you’re using the remote cache that is part of Develocity, this should function automatically, though note that network connectivity from Develocity to the cache is required. If an external remote cache is used, it must support the gRPC protocol and allow anonymous read access.

Cache permissions

It is generally considered a best practice to not allow all users read/write access to the build cache due to concerns of cache poisoning. Because Bazel build scans are enriched by allowing users to upload data to the remote cache, read-only access is not appropriate. We therefore add a more granular permission - "Read build cache data and write Bazel CAS data". CAS here means "content-addressable storage", one of the two types of data a Bazel cache exposes. All files linked from a Bazel build scan are CAS data. Writes to content-addressable storage are also more secure than arbitrary writes - all keys in content-addressable storage are digests of the value they point to, making cache poisoning less of a concern. Consider granting this permission by default to developers using Bazel.

Configuring build scans

Controlling when build scans are published

Once you’ve gone through the initial setup of the previous section, you are ready to start publishing build scans. To control when build scans are published, you need to enable or disable the default configuration in your .bazelrc file. If you followed the setting up steps, you should find a line like the one below:

build --config=develocity

You can remove this line and instead add --config=develocity on the command line when you want to publish a build scan.

Background uploading

By default, Bazel will wait for all the build events and corresponding cache entries to be uploaded. This can be changed by adding the following to .bazelrc

common:develocity --bes_upload_mode=fully_async

Setting the project name

Develocity allows analyzing sets of builds based on search criteria. One such criterion is the project name, which is useful to set to a well known name for the project to allow for more effective searching.

By default, the project name is the same as the directory name of the build workspace. To configure it to a different value, specify the following configuration options:

common:develocity --bes_instance_name=«project-name»
common:develocity --remote_instance_name=«project-name»
If Project-level Access Control is enabled, setting the project name to a custom value will also set that value as the project identifier.

Project-level access control

(Develocity 2023.4+)

To leverage Develocity’s project-level access control, builds must specify the following configuration options with the desired Develocity project ID:

common:develocity --bes_instance_name=«project-name»
common:develocity --remote_instance_name=«project-name»

These are typically set in the workspace .bazelrc file and checked in to version control.

Extending Build Scans

You can easily include extra information in your Build Scans in the form of tags, links and values. This is a very powerful mechanism for capturing and sharing information that is important to your build and development process.

Such information is included by using Bazel’s workspace status and/or build metadata mechanisms, with specific value name prefixes.

The workspace status mechanism is most appropriate when a tag, value or link needs to be computed. The build metadata mechanism can be used when it is static, or is specified as part of the command line invocation.

Tags, values and links use the following prefixes respectively:

  • SCAN_TAG_

  • SCAN_VALUE_

  • SCAN_LINK_

The following are examples of creating a tag, value and link using build metadata at build invocation time:

bazel build ... \
--build_metadata=SCAN_TAG_LOCAL_BUILD= \
--build_metadata=SCAN_VALUE_GIT_COMMIT_ID=$(git rev-parse HEAD) \
--build_metadata=SCAN_LINK_CI_BUILD=https://ci-server.company.com/build/${BUILD_ID}

The workspace status mechanism works by running a program that prints status item values to stdout. The following is an example of using a shell script named scan_data.sh in the workspace directory to create a tag, value and link:

scan_data.sh 
#!/bin/sh
echo SCAN_TAG_LOCAL_BUILD ""
echo "SCAN_VALUE_GIT_COMMIT_ID $(git rev-parse HEAD)"
echo "SCAN_LINK_CI_BUILD https://ci-server.company.com/build/${BUILD_ID}"

This script must be specified as a workspace status command, which is typically achieved by adding the following to the workspace .bazelrc file.

.bazelrc 
build --workspace_status_command=$(pwd)/scan_data.sh

If your workspace status command issues a SCAN_TAG as the very last line of output, Bazel will truncate it unless it also has a dummy value. See this section for more details

Troubleshooting

Failed to query remote execution capabilities

ERROR: Failed to query remote execution capabilities: CANCELLED: Bazel compatibility has not been enabled for your Develocity installation

If you notice an error like the above, it means that Bazel support was not enabled on your Develocity instance. Contact your administrator to enable Bazel support.

A tag is missing from the Build Scan

If your workspace status command issues a SCAN_TAG as the very last line of output, Bazel will truncate it unless it also has a dummy value, leading to missing the tag in the Build Scan. The value is not going to be used, but it needs to be a non-empty string to ensure the tag is picked up correctly, like below

scan_data.sh 
#!/bin/sh
# ..
echo SCAN_TAG_LOCAL_BUILD "dummy"

Appendix A: Captured information

Build Scans for Bazel work differently from those for Gradle, Maven, or sbt: those build tools rely on a plugin developed by Gradle to determine what information is captured by Develocity, but in Bazel this is all controlled by client-side settings, and built-in Bazel functionality. Because of this, we cannot directly control which information will be captured, but this document will cover the most typical cases.

In general, assume that all data sent to Develocity by way of either the Bazel JSON Trace Profile or the Build Event Protocol (BEP) will be captured by Develocity.

Enumerated below is the data that will be sent based on a few common Bazel options being set for a Java build. The exact information captured will vary based on which Bazel rules, configuration options, and client version are used.

--noslim_profile Generating a profile is the default behavior, but by default a slim profile is generated, which contains less information. This is the information contained in a fat profile:

--build_event_publish_all_actions By default, Bazel summarizes short actions, which makes it hard for Develocity to construct a complete build timeline.

--remote_cache The Build Events will frequently reference files from the local filesystem (test.log, test.xml, failed actions, etc.) and if they are not uploaded to the remote cache, they will be unavailable for analysis.

  • Environment

    • All client environment variables are captured

  • Build

    • Build invocation options

    • Bazel version

    • Date & timestamps

    • Output directory on the client Machine

    • CPU Usage for both client overhead & build

    • Memory usage for both client overhead & build

    • Target names

    • Source file names

    • Remote output downloads

    • Platform information

    • Target configuration

    • Action inputs and outputs

    • Client environment variables

    • Actions

Redacted information

In order to protect secrets which may likely appear in environment variables or Bazel flags, Develocity, by default, displays the following fields as "<REDACTED>" when displaying a Bazel Build Scan®:

Environment Variables

ALL environment variables except for:

  • BITRISE_GIT_BRANCH

  • BITRISE_GIT_COMMIT

  • BUILDKITE_BRANCH

  • BUILDKITE_BUILD_URL

  • BUILDKITE_COMMIT

  • BUILDKITE_JOB_ID

  • BUILDKITE_REPO

  • CIRCLE_BRANCH

  • CIRCLE_REPOSITORY_URL

  • CIRCLE_SHA1

  • CI_COMMIT_BRANCH

  • CI_COMMIT_SHA

  • CI_REPOSITORY_URL

  • CI_RUNNER

  • CI

  • COMMIT_SHA

  • GITHUB_ACTOR

  • GITHUB_HEAD_REF

  • GITHUB_REF

  • GITHUB_REPOSITORY

  • GITHUB_RUN_ID

  • GITHUB_SHA

  • GIT_BRANCH

  • GIT_COMMIT

  • GIT_REPOSITORY_URL

  • GIT_URL

  • REPO_URL

  • TRAVIS_BRANCH

  • TRAVIS_COMMIT

  • TRAVIS_REPO_SLUG

  • USER

*_header Flags

There are several Bazel flags that designate headers of requests to remote servers. As some of these headers may contain sensitive information, the values of certain header names will be redacted.

The flags with redacted headers are:

  • --bes_header

  • --remote_cache_header

  • --remote_downloader_header

  • --remote_exec_header

  • --remote_header

The header names for which values are redacted are:

  • Authorization

  • X-Buildbuddy-Api-Key

  • X-Engflow-Auth-Token

Header names are checked in a case-insensitive fashion.