The Gradle Enterprise test distribution agent executes tests triggered from the Gradle Enterprise test distribution Gradle plugin or Gradle Enterprise Maven extension on remote machines.

Use of the Gradle Enterprise test distribution agent software is subject to the Gradle Enterprise Software Agreement.

Agents need to be connected to a Gradle Enterprise server.

The Gradle Enterprise test distribution agent is distributed as a Docker image via Docker Hub, and as an executable JAR. Both distributions offer the same functionality.

Requirements

CPU & memory

The agent itself needs about 128 MiB of memory and does not require significant CPU resources. However, you need to determine the total memory and CPU resources depending on the types of tests you intend to run on the agent. Two CPU cores and 2 GiB of memory are a good starting point for tests that are not particularly resource-intensive. If your tests start additional external processes, you’ll likely need more. Thus, you should monitor CPU and memory consumption of an agent while it’s executing a typical test workload.

Maximum heap size

The maximum heap size (and other memory options) of the forked Java VM that executes the tests can be configured by developers.

For Gradle builds, the maximum heap size can be configured via the maxHeapSize property and is limited to 512 MiB by default.

For Maven builds, Surefire/Failsafe lets the forked JVM determine the maximum heap size based on the machine’s physical memory by default. The outcome depends on the used JVM version and implementation and may not take into account memory restrictions when the agent runs in a Docker container on older versions of Java (see JDK-8146115). Therefore, it’s usually a good idea to configure -Xmx explicitly via the argLine configuration parameter.

Disk usage

The agent uses a local file cache to store JAR files and other artifacts that are required to execute tests. The cache directory is cleaned up periodically, so it does not grow indefinitely. How much space you’ll need depends on your tests and how often you’ve configured the cache directory to be cleaned up. Usually, a few hundred MiB should be sufficient. We recommend monitoring the directory size over the course of the cache’s retention period.

Network

The agent requires a reasonably fast and stable connection to the Gradle Enterprise server in order to operate efficiently. While running, the agent maintains a persistent WebSocket connection over default HTTP/HTTPS ports to the Gradle Enterprise server.

In case your agents cannot connect to the Gradle Enterprise server, please ensure all load balancers and proxies that are used between both ends support WebSocket connections.

If the connection is lost, e.g. when the server is restarted, the agent will periodically attempt to reconnect. Since all communication goes through the Gradle Enterprise server, the agent does not need a direct connection to the machine that initiated test execution, e.g. the developer workstation or build server running the Gradle build, or vice versa. Moreover, since the connection to the Gradle Enterprise server is initiated by the agent, it can be deployed to an internal network and does not need to be addressable from machines running builds.

Security

Test distribution allows remote code execution inside your infrastructure for authorized Gradle Enterprise users. This is comparable to CI jobs which also execute tests that could theoretically run any command on the remote host. In addition, test code will be able to access the local file system including the local file cache which might include artifacts of other projects that have previously had their tests executed on the same agent. Thus, you should treat the infrastructure you run test distribution agents on similarly to those of regular CI agents.

Installation

JAR

You can download version 1.6 of the JAR here. Older versions of the JAR can be found in the appendix below.

The agent requires at least Java 11 to run and supports Linux, macOS, and Windows. Moreover, you need to specify the server and API key parameters that are required to connect and register with the Gradle Enterprise server.

java -Xms64m -Xmx64m -XX:MaxMetaspaceSize=64m \
    -jar gradle-enterprise-test-distribution-agent-1.6.jar \
    --server https://«gradle-enterprise-hostname» \
    --api-key «api-key» (1)
1 API key generated via Gradle Enterprise administration

You will see output similar to the following:

[…] Skipping cleanup of local file cache (next due on […])
[…] Successfully registered with https://«gradle-enterprise-hostname» (will be refreshed in 1h)
[…] Agent (v1.6) started and connected to https://«gradle-enterprise-hostname»

Docker

With Docker installed, you can start an agent using the following command:

docker run --detach \
    gradle/gradle-enterprise-test-distribution-agent:1.6 \
    --server https://«gradle-enterprise-hostname» \
    --api-key «api-key» (1)
1 API key generated via Gradle Enterprise administration

Alternatively, you can use environment variables to configure the agent:

docker run --detach \
    --env TEST_DISTRIBUTION_AGENT_SERVER=https://«gradle-enterprise-hostname» \
    --env TEST_DISTRIBUTION_AGENT_API_KEY=«api-key» \
    gradle/gradle-enterprise-test-distribution-agent:1.6

The default Docker image has AdoptOpenJDK 11 installed. You can extend the image should you wish to install additional JDKs or other software.

By default, the Docker image’s entrypoint passes -Xms64m -Xmx64m -XX:MaxMetaspaceSize=64m as JVM arguments to the agent JVM. You can override these JVM arguments by passing the TEST_DISTRIBUTION_AGENT_OPTS environment variable to the Docker container.

Airgapped hosts

In order to install the agent Docker image on a host that is not connected to the Internet (i.e. airgapped), you will need to first obtain the image on an Internet connected host, then transfer it to the airgapped destination.

  1. On the non-airgapped system, pull down the latest docker image for the node and export it to a file

    docker pull gradle/gradle-enterprise-test-distribution-agent:1.6
    docker save gradle/gradle-enterprise-test-distribution-agent:1.6 \
        --output gradle-enterprise-test-distribution-agent.tar
  2. Copy this file across to the airgapped host, and then import the image into docker

    docker load --input gradle-enterprise-test-distribution-agent.tar

The agent can then be started and configured on the airgapped host in the same manner as a non-airgapped install.

Kubernetes

Below is a sample Kubernetes manifest for deploying agents to a Kubernetes cluster. You may need to adjust its configuration, e.g. the configured CPU and memory requirements, based on the needs of your tests.

---
apiVersion: v1
kind: Secret
metadata:
  name: gradle-enterprise-test-distribution-api-key-secret
  namespace: default
stringData:
  apiKey: «api-key» (1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gradle-enterprise-test-distribution-agent
  labels:
    app: gradle-enterprise
    component: test-distribution-agent
spec:
  selector:
    matchLabels:
      app: gradle-enterprise
      component: test-distribution-agent
  template:
    metadata:
      labels:
        app: gradle-enterprise
        component: test-distribution-agent
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 0
        fsGroup: 0
      terminationGracePeriodSeconds: 600 (2)
      containers:
      - name: gradle-enterprise-test-distribution-agent
        image: gradle/gradle-enterprise-test-distribution-agent:1.6
        resources:
          requests:
            memory: 1Gi
            cpu: 2
          limits:
            memory: 4Gi
            cpu: 2
        env:
        - name: TEST_DISTRIBUTION_AGENT_SERVER
          value: https://«gradle-enterprise-hostname»
        - name: TEST_DISTRIBUTION_AGENT_API_KEY
          valueFrom:
            secretKeyRef:
              name: gradle-enterprise-test-distribution-api-key-secret
              key: apiKey
        - name: TEST_DISTRIBUTION_AGENT_POOL
          value: «pool-id» (3)
1 API key generated via Gradle Enterprise administration
2 The amount of time Kubernetes should wait for the agent to gracefully terminate.
3 Agent pool configured via Gradle Enterprise administration for auto scaling (optional)
Prior to Kubernetes v1.9, the deployment apiVersion: apps/v1 should be apiVersion: apps/v1beta1.

Configuration

All configuration options can be specified via command-line arguments or environment variables whereby the former take precedence over the latter.

Registration

There are two mandatory parameters (environment variables) that are used to register with the Gradle Enterprise server on startup:

-s/--server https://«gradle-enterprise-hostname» (env: TEST_DISTRIBUTION_AGENT_SERVER)

The URL of the Gradle Enterprise server the agent should connect to

-r/--api-key «api-key» (env: TEST_DISTRIBUTION_AGENT_API_KEY)

The API key to use for authenticating with the Gradle Enterprise server

The agent sends the API key to the Gradle Enterprise server when starting up and once every hour to renew its registration. If the API key is invalid or has been revoked by a Gradle Enterprise administrator, the agent will print an error message and exit with a non-zero status code.

By default, the agent will not allow connections to Gradle Enterprise if the server presents an untrusted certificate (e.g. self-signed). In order to allow such connections, you’ll have to use the following parameter (environment variable):

--allow-untrusted true/false (env: TEST_DISTRIBUTION_AGENT_ALLOW_UNTRUSTED)

Allow/forbid connecting to servers with untrusted (e.g. self-signed) SSL certificates (defaults to false)

Enabling this option is a security risk as it makes it easier for a third party to intercept communication between the agent and Gradle Enterprise. It should only be used as a short term workaround until the server can be configured with a trusted certificate.

If your agent should be part of a previously configured agent pool, you should specify the following parameter (environment variable):

-p/--pool «pool-id» (env: TEST_DISTRIBUTION_AGENT_POOL)

The identifier of the pool to which the agent should be assigned

Finally, you may specify the agent name that will be shown on the Gradle Enterprise Administration pages:

-n/--name «agent-name» (env: TEST_DISTRIBUTION_AGENT_NAME)

Descriptive agent name (defaults to hostname)

Capabilities

When registering with Gradle Enterprise upon startup, the agents report a set of capabilities that will be matched against the requirements of test tasks. Please refer to the Gradle plugin and Maven extension user manuals for details on how to configure requirements.

The agent auto-detects the operating system it is running on (currently supported are Linux, macOS, and Windows) and adds os=linux, os=macos, or os=windows to the set of capabilities.

Moreover, the agent infers installed JDKs from environment variables that match the JDK\d\d? regex and adds a capability for each of them. For example, if JDK8 and JDK11 are set to the paths of their corresponding Java home directories, the agent will automatically add jdk=8 and jdk=11 to the set of capabilities.

If you want to install additional software on a subset of your agents and use that as a criterion when selecting agents for test execution, you can use the following parameter (environment variable):

-c/--capabilities «capability»(,«capability»)* (env: TEST_DISTRIBUTION_AGENT_CAPABILITIES)

Comma-separated list of agent capabilities to match against build requirements

Capabilities may only contain alphanumeric characters, dashes, underscores, periods, and a single equals sign.

For example, the following invocation would make the agent register with the additional docker and postgres=12 capabilities:

java -jar gradle-enterprise-test-distribution-agent-1.6.jar […] \
     --capabilities docker,postgres=12

While the exact semantics are up to you, this might indicate that the tests on that agent can use Docker and PostgreSQL in version 12.

Local file cache

The following parameters (environment variables) can be used to configure the local file cache:

--cache-dir «dir» (env: TEST_DISTRIBUTION_AGENT_CACHE_DIR)

Directory to use for the local file cache (defaults to «user-home»/.gradle-enterprise-test-distribution-agent/file-cache)

--cache-cleanup true/false (env: TEST_DISTRIBUTION_AGENT_CACHE_CLEANUP)

Enable/disable cleanup of the local file cache (defaults to true)

--cache-cleanup-interval «duration» (env: TEST_DISTRIBUTION_AGENT_CACHE_CLEANUP_INTERVAL)

Interval in which the local file cache should be cleaned up (defaults to PT24H, i.e. 24 hours)

--cache-retention-period «duration» (env: TEST_DISTRIBUTION_AGENT_CACHE_RETENTION_PERIOD)

Period for which entries in the local file cache should be retained (defaults to PT168H, i.e. 7 days)

Sharing the local file cache among several agents running on the same host is currently not supported. If you are running more than one agent on the same host make sure to configure individual cache directories for each agent or use the Docker image.

Auto scaling

General setup

In order to scale the number of agents based on demand, i.e. how many tests need to be executed at any given moment, Gradle Enterprise uses the concept of an agent pool. In general, the steps to set up an auto scaling pool of agents are as follows:

  1. Create and configure an agent pool on the administration pages of Gradle Enterprise.

  2. Configure your agents to register with the ID of the pool via the --pool option.

  3. Configure your compute platform to query the pool’s status at regular intervals.

  4. Start/stop agents in accordance with the value of the desiredAgents property in the JSON response.

Stopping agents gracefully

When scaling down, compute platforms may stop an agent that is running tests. Care should be taken in how the agents are stopped to allow agents to finish any in-flight tests.

To do this, first send the TERM signal to an agent you wish to stop and wait for a certain grace period before sending it the KILL signal. When receiving a TERM signal, the agent notifies the build that it needs to stop and terminates gracefully after finishing the current set of tests. The duration of the grace period depends on the nature of your tests — it should allow your slowest test classes to finish.

The docker stop command provides a --time option with this behaviour. The TERM signal will be sent immediately, with a KILL signal being sent if the agent has not stopped within the given --time setting. Kubernetes provides a terminationGracePeriodSeconds parameter that provides the same behaviour.

If an agent is killed before it is allowed to stop gracefully, the set of tests that it was assigned will be rescheduled for execution by another agent. This may cause some tests that had already been executed by the stopped agent to be executed again.

Kubernetes

In order to configure your agents to register with the ID of a previously configured agent pool, the simplest option is to pass the TEST_DISTRIBUTION_AGENT_POOL environment variable as demonstrated in the deployment manifest in the installation section.

For scaling the number of replicas in the deployment based on demand, using the lightweight and open source KEDA component is recommended.

First, deploy KEDA to your Kubernetes cluster (e.g. using the provided Helm chart). Then, create a KEDA ScaledObject in your cluster. Behind the scenes, KEDA sets up a standard Horizontal Pod Autoscaler (HPA), queries the pool’s status in Gradle Enterprise according to the configured pollingInterval, and configures the HPA to scale according to the value of the desiredAgents property in the JSON response.

The following manifest is all you need to set up auto scaling of agents in an agent pool in your Kubernetes cluster:

---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: gradle-enterprise-test-distribution-agent-credentials
spec:
  secretTargetRef:
  - parameter: apiKey
    name: gradle-enterprise-test-distribution-api-key-secret (1)
    key: apiKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gradle-enterprise-test-distribution-agent-scaler
spec:
  scaleTargetRef:
    name: gradle-enterprise-test-distribution-agent
  minReplicaCount: 1 (2)
  maxReplicaCount: 100 (2)
  pollingInterval: 15 (3)
  triggers:
  - type: metrics-api
    metadata:
      url: https://«gradle-enterprise-hostname»/distribution/agent-pools/«pool-id» (4)
      authMode: apiKey
      valueLocation: desiredAgents
      targetValue: '1'
    authenticationRef:
      name: gradle-enterprise-test-distribution-agent-credentials
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60 (5)
1 Reference to the API key created for agent deployment (cf. installation)
2 The minimum/maximum number of replicas should match the configuration of the agent pool in Gradle Enterprise
3 How frequently KEDA should query the agent pool’s status
4 The URL of the agent pool status endpoint in Gradle Enterprise
5 How long agents should stay connected after a drop in demand before being stopped. You may want to customize this setting depending on how long it takes to spin up nodes in your cluster.

Usage

The “Usage” tab in Gradle Enterprise visualizes the historical usage of agent pools, including the number of utilized (i.e. actually in use), connected (i.e. available to be utilized) and desired (i.e. that could be utilized if available) agents.

agent pool usage

Appendix A: Release history

1.6

15th September 2021
  • Change communication protocol to avoid illegal reflective access

Compatible with Gradle Enterprise 2021.3 or later.

1.5

1st June 2021
  • Update base Docker image to adoptopenjdk/openjdk11:jdk-11.0.10_9-ubuntu

  • Time out and retry when upgrading a connection to WebSockets hangs

  • Add --version option

  • Add --pool option

  • Deprecate --registration-key/-r option in favor of --api-key/-k

  • Perform version compatibility checks during registration with Gradle Enterprise

  • Handle SIGTERM by signalling the current job and waiting for it to finish

Compatible with Gradle Enterprise 2021.2 or later.

1.4.1

7th April 2021
  • Added basic self-healing functionality to local file cache by quarantining corrupt cache entries and re-downloading them when they are next requested

  • Fixed race between re-connection and registration to Gradle Enterprise that could cause the agent to exit due to an authorization error

  • Reduced logging severity of connection errors

Compatible with Gradle Enterprise 2021.1 or later.

1.4

15th March 2021
  • Fix race condition when agent is released shortly after being assigned

Compatible with Gradle Enterprise 2021.1 or later.

1.3.2

3rd February 2021
  • Agent keeps reconnecting to Gradle Enterprise for one hour in case the server url cannot be resolved

  • Input file archives are now sanity checked before unpacking to ensure only regular files and directories are created inside the target directory (CVE-2021-26719)

Compatible with Gradle Enterprise 2020.5 or later.

1.3.1

17th December 2020
  • Deliberately cancelled downloads are no longer logged as errors

  • Illegal reflective access warnings no longer pollute the agent log

  • Memory options are applied by default when using the Docker image

  • Regression in local file cache cleanup is fixed

  • Agent keeps reconnecting to Gradle Enterprise in case of a temporary DNS lookup failure

  • Fix sporadic ArithmeticException when reconnecting to the Gradle Enterprise server

Compatible with Gradle Enterprise 2020.5 or later.

1.3

8th December 2020
  • Retry file transfer on more intermittent failure types

Compatible with Gradle Enterprise 2020.5 or later.

1.2

27th October 2020
  • Unresponsive network connections are now detected and closed proactively

Compatible with Gradle Enterprise 2020.4 or later.

1.1.2

31st July 2020
  • Downloads that fail due to unexpectedly closed connections are now retried

  • Download failures are reported to the build in all cases

Compatible with Gradle Enterprise 2020.3 or later.

1.1.1

29th July 2020
  • Downloaded files are now always closed before storing them in the local file cache

Compatible with Gradle Enterprise 2020.3 or later.

1.1

27th July 2020
  • Input files are now downloaded in parallel from the file cache on the Gradle Enterprise server instead of being requested from connected builds directly

  • Agent jar is published alongside its signature

  • Improved retry handling in case of connection errors

  • Docker image no longer uses the JAVA_OPTS environment variable to avoid inheriting it to test VMs and their child processes

Compatible with Gradle Enterprise 2020.3 or later.

1.0

5th May 2020
  • Initial release

Compatible with Gradle Enterprise 2020.2 or later.

Appendix B: JAR downloads

Appendix C: Compatibility with Gradle Enterprise

Compatibility between versions of Gradle Enterprise and the Gradle Enterprise test distribution agent can be found here.

Appendix D: Extending the Docker image

The Docker image is based on adoptopenjdk/openjdk11 which is based on Ubuntu so installing additional dependencies is easy. A common use case is to install additional JDKs or other software packages. In the example below, OpenJDK 8 is added using a multistage build along with the zip and unzip Ubuntu packages.

FROM adoptopenjdk/openjdk8:jdk8u302-b08-ubuntu-slim AS openjdk8

FROM gradle/gradle-enterprise-test-distribution-agent:1.6

ENV JDK8=/opt/java/openjdk-8
COPY --from=openjdk8 ${JAVA_HOME} ${JDK8}

# Switch to root user for installing packages
USER root

# Install additional packages
ENV DEBIAN_FRONTEND noninteractive
RUN apt update \
 && apt install --assume-yes --no-install-recommends zip unzip

# Switch back to application user
USER gradle

Appendix E: Dealing with tests that start Docker containers

Developers nowadays rely on Docker containers being started from inside their tests in order to test integration with external services. Frameworks like Testcontainers provide Java APIs that make this even easier. Depending on the deployment model you chose, you need to take different steps in order to support executing Docker-based tests on test distribution agents.

In case you decided to deploy the JAR directly you need to make sure that:

  • Docker is installed on the host executing the agent application.

  • The user executing the agent application has sufficient permissions to use Docker.

In case you decided for a Docker-based deployment you need to make Docker available inside the test distribution agent Docker container. The Testcontainers documentation provides more information about patterns that can be used to do this.

Appendix F: Verifying the signature of the agent jar

(agent 1.0.1+)

The agent jar is published to docs.gradle.com alongside its signature. The public key is published to https://keys.openpgp.org. You can verify the signature as follows:

curl -OL https://docs.gradle.com/enterprise/test-distribution-agent/jar/gradle-enterprise-test-distribution-agent-1.6.jar
curl -OL https://docs.gradle.com/enterprise/test-distribution-agent/jar/gradle-enterprise-test-distribution-agent-1.6.jar.asc
gpg --keyserver keys.openpgp.org --recv-key 314FE82E5A4C5377BCA2EDEC5208812E1E4A6DB0
gpg --verify gradle-enterprise-test-distribution-agent-1.6.jar.asc gradle-enterprise-test-distribution-agent-1.6.jar

The output of the last command should look similar to the following:

gpg: Signature made Tue May  5 08:36:01 2020 UTC
gpg:                using RSA key 5208812E1E4A6DB0
gpg: Good signature from "Gradle Inc. <info@gradle.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 314F E82E 5A4C 5377 BCA2  EDEC 5208 812E 1E4A 6DB0

This verifies that the artifact was signed with the private key that corresponds to the imported public key. The warning is emitted because you haven’t explicitly trusted the imported key (hence [unknown]). One way of establishing trust is to verify the fingerprint over a secure channel. Please contact technical support should you wish to do so.