Develocity Test Distribution Agent User Manual

Requirements

CPU & memory

The agent itself needs about 128 MiB of memory and does not require significant CPU resources. However, you need to determine the total memory and CPU resources depending on the types of tests you intend to run on the agent. Two CPU cores and 2 GiB of memory are a good starting point for tests that are not particularly resource-intensive. If your tests start additional external processes, you’ll likely need more. Thus, you should monitor CPU and memory consumption of an agent while it’s executing a typical test workload.

Maximum heap size

The maximum heap size (and other memory options) of the forked Java VM that executes the tests can be configured by developers.

For Gradle builds, the maximum heap size can be configured via the maxHeapSize property and is limited to 512 MiB by default.

For Maven builds, Surefire/Failsafe lets the forked JVM determine the maximum heap size based on the machine’s physical memory by default. The outcome depends on the used JVM version and implementation and may not take into account memory restrictions when the agent runs in a Docker container on older versions of Java (see JDK-8146115). Therefore, it’s usually a good idea to configure -Xmx explicitly via the argLine configuration parameter.

Disk usage

The agent uses a local file cache to store JAR files and other artifacts that are required to execute tests. The cache directory is cleaned up periodically, so it does not grow indefinitely. How much space you’ll need depends on your tests and how often you’ve configured the cache directory to be cleaned up. Usually, a few hundred MiB should be sufficient. We recommend monitoring the directory size over the course of the cache’s retention period.

Network

The agent requires a reasonably fast and stable connection to the Develocity server in order to operate efficiently. While running, the agent maintains a persistent WebSocket connection over default HTTP/HTTPS ports to the Develocity server.

In case your agents cannot connect to the Develocity server, please ensure all load balancers and proxies that are used between both ends support WebSocket connections.

If the connection is lost, e.g. when the server is restarted, the agent will periodically attempt to reconnect. Since all communication goes through the Develocity server, the agent does not need a direct connection to the machine that initiated test execution, e.g. the developer workstation or build server running the Gradle build, or vice versa. Moreover, since the connection to the Develocity server is initiated by the agent, it can be deployed to an internal network and does not need to be addressable from machines running builds.

Security

Test Distribution allows remote code execution inside your infrastructure for authorized Develocity users. This is comparable to CI jobs which also execute tests that could theoretically run any command on the remote host. In addition, test code will be able to access the local file system including the local file cache which might include artifacts of other projects that have previously had their tests executed on the same agent. Thus, you should treat the infrastructure you run Test Distribution agents on similarly to those of regular CI agents.

Installation

JAR

You can download version 3.0 of the JAR here. Older versions of the JAR can be found in the appendix below.

The agent requires at least Java 21 to run and supports Linux, macOS, and Windows. Moreover, you need to specify the server, registration key, and pool parameters that are required to connect and register with the Develocity server.

java -Xms64m -Xmx64m -XX:MaxMetaspaceSize=64m \
    -jar develocity-test-distribution-agent-3.0.jar \
    --server https://«develocity-hostname» \
    --registration-key «registration-key» (1)
    --pool «pool-id» (2)

1	Registration key generated via Develocity administration
2	Agent pool configured via Develocity administration

You will see output similar to the following:

[…] Skipping cleanup of local file cache (next due on […])
[…] Successfully registered with https://«develocity-hostname» (will be refreshed in 1h)
[…] Agent (v3.0) started and connected to https://«develocity-hostname»

Docker

With Docker installed, you can start an agent using the following command:

docker run --detach \
    gradle/develocity-test-distribution-agent:3.0 \
    --server https://«develocity-hostname» \
    --registration-key «registration-key» (1)
    --pool «pool-id» (2)

1	Registration key generated via Develocity administration
2	Agent pool configured via Develocity administration

Alternatively, you can use environment variables to configure the agent:

docker run --detach \
    --env TEST_DISTRIBUTION_AGENT_SERVER=https://«develocity-hostname» \
    --env TEST_DISTRIBUTION_AGENT_API_KEY=«api-key» \
    gradle/develocity-test-distribution-agent:3.0

The default Docker image has JDK 21 installed. You can extend the image should you wish to install additional JDKs or other software.

By default, the Docker image’s entrypoint passes -Xms64m -Xmx64m -XX:MaxMetaspaceSize=64m as JVM arguments to the agent JVM. You can override these JVM arguments by passing the TEST_DISTRIBUTION_AGENT_OPTS environment variable to the Docker container.

Airgapped hosts

In order to install the agent Docker image on a host that is not connected to the Internet (i.e. airgapped), you will need to first obtain the image on an Internet connected host, then transfer it to the airgapped destination.

On the non-airgapped system, pull down the latest docker image for the node and export it to a file

docker pull gradle/develocity-test-distribution-agent:3.0
docker save gradle/develocity-test-distribution-agent:3.0 \
    --output develocity-test-distribution-agent.tar

Copy this file across to the airgapped host, and then import the image into docker
```
docker load --input develocity-test-distribution-agent.tar
```

The agent can then be started and configured on the airgapped host in the same manner as a non-airgapped install.

Kubernetes

Below is a sample manifest for deploying agents to a Kubernetes cluster using the official agent Docker image. You may need to adjust its configuration, e.g. the configured CPU and memory requirements, based on the needs of your tests.

---
apiVersion: v1
kind: Secret
metadata:
  name: develocity-test-distribution-api-key-secret
  namespace: default
stringData:
  apiKey: «api-key» (1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: develocity-test-distribution-agent
  labels:
    app.kubernetes.io/instance: test-distribution-agent-«pool-id» (2)
    app.kubernetes.io/part-of: gradle-enterprise
    app.kubernetes.io/component: test-distribution-agent
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: test-distribution-agent-«pool-id» (2)
      app.kubernetes.io/part-of: gradle-enterprise
      app.kubernetes.io/component: test-distribution-agent
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: test-distribution-agent-«pool-id» (2)
        app.kubernetes.io/part-of: gradle-enterprise
        app.kubernetes.io/component: test-distribution-agent
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 0
        fsGroup: 0
      terminationGracePeriodSeconds: 600 (3)
      containers:
      - name: develocity-test-distribution-agent
        image: gradle/develocity-test-distribution-agent:3.0 (4)
        resources:
          requests:
            memory: 1Gi
            cpu: 2
          limits:
            memory: 4Gi
            cpu: 2
        env:
        - name: TEST_DISTRIBUTION_AGENT_SERVER
          value: https://«develocity-hostname»
        - name: TEST_DISTRIBUTION_AGENT_API_KEY
          valueFrom:
            secretKeyRef:
              name: develocity-test-distribution-api-key-secret
              key: apiKey
        - name: TEST_DISTRIBUTION_AGENT_POOL
          value: «pool-id» (2)

1	Registration key generated via Develocity administration
2	Agent pool configured via Develocity administration
3	The amount of time Kubernetes should wait for the agent to gracefully terminate.
4	The official or extended agent Docker image

Prior to Kubernetes v1.9, the deployment apiVersion: apps/v1 should be apiVersion: apps/v1beta1.

Configuration

All configuration options can be specified via command-line arguments or environment variables whereby the former take precedence over the latter.

Registration

There are three mandatory parameters (environment variables) that are used to register with the Develocity server on startup:

-s/--server https://«develocity-hostname» (env: TEST_DISTRIBUTION_AGENT_SERVER): The URL of the Develocity server the agent should connect to
-p/--pool «pool-id» (env: TEST_DISTRIBUTION_AGENT_POOL): The identifier of a previously configured agent pool to which the agent should be assigned
-k/--registration-key «registration-key» (env: TEST_DISTRIBUTION_AGENT_REGISTRATION_KEY): The pool-specific registration key to use for authenticating with the Develocity server

The agent sends the API key to the Develocity server when starting up and once every hour to renew its registration. If the API key is invalid or has been revoked by a Develocity administrator, the agent will print an error message and exit with a non-zero status code.

By default, the agent will not allow connections to Develocity if the server presents an untrusted certificate (e.g. self-signed). In order to allow such connections, you’ll have to use the following parameter (environment variable):

--allow-untrusted true/false (env: TEST_DISTRIBUTION_AGENT_ALLOW_UNTRUSTED): Allow/forbid connecting to servers with untrusted (e.g. self-signed) SSL certificates (defaults to false)

Enabling this option is a security risk as it makes it easier for a third party to intercept communication between the agent and Develocity. It should only be used as a short term workaround until the server can be configured with a trusted certificate.

Finally, you may specify the agent name that will be shown on the Develocity Administration pages:

-n/--name «agent-name» (env: TEST_DISTRIBUTION_AGENT_NAME): Descriptive agent name (defaults to hostname)

Capabilities

When registering with Develocity upon startup, the agents report a set of capabilities that will be matched against the requirements of test tasks. Please refer to the Test Distribution user manual for details on how to configure requirements for Gradle or Maven builds.

The agent auto-detects the operating system it is running on (currently supported are Linux, macOS, and Windows) and adds os=linux, os=macos, or os=windows to the set of capabilities.

Moreover, the agent infers installed JDKs from environment variables that match the JDK\d\d? regex and adds a capability for each of them. For example, if JDK8 and JDK11 are set to the paths of their corresponding Java home directories, the agent will automatically add jdk=8 and jdk=11 to the set of capabilities.

If you want to install additional software on a subset of your agents and use that as a criterion when selecting agents for test execution, you can use the following parameter (environment variable):

-c/--capabilities «capability»(,«capability»)* (env: TEST_DISTRIBUTION_AGENT_CAPABILITIES): Comma-separated list of agent capabilities to match against build requirements

Capabilities may only contain alphanumeric characters, dashes, underscores, periods, and a single equals sign.

For example, the following invocation would make the agent register with the additional docker and postgres=12 capabilities:

java -jar develocity-test-distribution-agent-3.0.jar […] \
     --capabilities docker,postgres=12

While the exact semantics are up to you, this might indicate that the tests on that agent can use Docker and PostgreSQL in version 12.

Local file cache

The following parameters (environment variables) can be used to configure the local file cache:

--cache-dir «dir» (env: TEST_DISTRIBUTION_AGENT_CACHE_DIR): Directory to use for the local file cache (defaults to «user-home»/.develocity-test-distribution-agent/file-cache)
--cache-cleanup true/false (env: TEST_DISTRIBUTION_AGENT_CACHE_CLEANUP): Enable/disable cleanup of the local file cache (defaults to true)
--cache-cleanup-interval «duration» (env: TEST_DISTRIBUTION_AGENT_CACHE_CLEANUP_INTERVAL): Interval in which the local file cache should be cleaned up (defaults to PT24H, i.e. 24 hours)
--cache-retention-period «duration» (env: TEST_DISTRIBUTION_AGENT_CACHE_RETENTION_PERIOD): Period for which entries in the local file cache should be retained (defaults to PT168H, i.e. 7 days)

Sharing the local file cache among several agents running on the same host is currently not supported. If you are running more than one agent on the same host make sure to configure individual cache directories for each agent or use the Docker image.

Auto scaling

General setup

In order to scale the number of agents based on demand, i.e. how many tests need to be executed at any given moment, Develocity uses the concept of an agent pool. In general, the steps to set up an auto scaling pool of agents are as follows:

Create and configure an agent pool on the administration pages of Develocity.
Configure your agents to register with the ID of the pool via the --pool option.
Configure your compute platform to query the pool’s status at regular intervals.
Start/stop agents in accordance with the value of the desiredAgents property in the JSON response.

Stopping agents gracefully

When scaling down, compute platforms may stop an agent that is running tests. Care should be taken in how the agents are stopped to allow agents to finish any in-flight tests.

To do this, first send the TERM signal to an agent you wish to stop and wait for a certain grace period before sending it the KILL signal. When receiving a TERM signal, the agent notifies the build that it needs to stop and terminates gracefully after finishing the current set of tests. The duration of the grace period depends on the nature of your tests — it should allow your slowest test classes to finish.

The docker stop command provides a --time option with this behaviour. The TERM signal will be sent immediately, with a KILL signal being sent if the agent has not stopped within the given --time setting. Kubernetes provides a terminationGracePeriodSeconds parameter that provides the same behaviour.

If an agent is killed before it is allowed to stop gracefully, the set of tests that it was assigned will be rescheduled for execution by another agent. This may cause some tests that had already been executed by the stopped agent to be executed again.

Kubernetes

In order to configure your agents to register with the ID of a previously configured agent pool, the simplest option is to pass the TEST_DISTRIBUTION_AGENT_POOL environment variable as demonstrated in the deployment manifest in the installation section.

For scaling the number of replicas in the deployment based on demand, using the lightweight and open source KEDA (minimum version 2.5) component is recommended.

First, deploy KEDA to your Kubernetes cluster (e.g. using the provided Helm chart). Then, create a KEDA ScaledObject in your cluster. Behind the scenes, KEDA sets up a standard Horizontal Pod Autoscaler (HPA), queries the pool’s status in Develocity according to the configured pollingInterval, and configures the HPA to scale according to the value of the desiredAgents property in the JSON response.

The following manifest is all you need to set up auto scaling of agents in an agent pool in your Kubernetes cluster:

---
apiVersion: v1
kind: Secret
metadata:
  name: develocity-test-distribution-access-key-secret
  namespace: default
stringData:
  accessKey: «access-key» (1)
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: develocity-test-distribution-agent-credentials
spec:
  secretTargetRef:
  - parameter: token
    name: develocity-test-distribution-access-key-secret
    key: accessKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: develocity-test-distribution-agent-scaler
spec:
  scaleTargetRef:
    name: develocity-test-distribution-agent
  minReplicaCount: 1 (2)
  maxReplicaCount: 100 (2)
  pollingInterval: 15 (3)
  triggers:
  - type: metrics-api
    metadata:
      url: https://«develocity-hostname»/api/test-distribution/agent-pools/«pool-id»/status (4)
      authMode: "bearer"
      valueLocation: desiredAgents
      targetValue: '1'
    authenticationRef:
      name: develocity-test-distribution-agent-credentials
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60 (5)

1	Access key of a user with the “Test Distribution“ access control permission
2	The minimum/maximum number of replicas should match the configuration of the agent pool in Develocity
3	How frequently KEDA should query the agent pool’s status
4	The URL of the agent pool status endpoint in Develocity
5	How long agents should stay connected after a drop in demand before being stopped. You may want to customize this setting depending on how long it takes to spin up nodes in your cluster.

Usage

The “Usage” tab in Develocity visualizes the historical usage of agent pools, including the number of utilized (i.e. actually in use), connected (i.e. available to be utilized) and desired (i.e. that could be utilized if available) agents.

Appendix A: Release history

3.0

2nd April 2024

[NEW] Test Distribution Agent Image provides JDK 21
[NEW] Rename Test Distribution agent from Gradle Enterprise to Develocity
[NEW] Develocity Test Distribution agent requires a Java 21 runtime environment
[NEW] Update the base Docker image to ubuntu:jammy-20240125

Compatible with Develocity 2024.1 or later.

2.1

5th December 2023

[NEW] The official Docker image is now based on ubuntu:jammy to include the latest security updates

Compatible with Develocity 2023.4 or later.

2.0.3

14th September 2023

[FIX] Update to Java 17.0.8-7 for security updates and bug fixes

Compatible with Develocity 2023.1 or later.

2.0.2

22nd June 2023

[FIX] Upgrade libssl to address CVE-2023-2650

Compatible with Develocity 2023.1 or later.

2.0.1

3rd May 2023

[FIX] Update JDK in Docker image to 17.0.7
[FIX] Update Ubuntu packages in Docker image

Compatible with Develocity 2023.1 or later.

2.0

11th April 2023

[NEW] Test Distribution agent JAR is compiled for and requires Java 17
[NEW] Official Docker image includes JDK 17 instead of JDK 11
[NEW] Reconnect to Develocity on 404 responses after prior successful connection

Compatible with Develocity 2023.1 or later.