Test Distribution
Test Distribution is a feature that enables the parallel execution of tests across multiple agents or machines, significantly reducing the overall test execution time.
Develocity Test Distribution takes your existing test suites and distributes them across multiple compute resources to execute them faster. It requires connecting one or more test execution “agents” to Develocity. Builds that have enabled Test Distribution connect to Develocity, which then distributes the tests to be executed on connected agents.
Connecting Builds
Consult the Develocity Test Distribution User Manual for information on build configuration.
Develocity users must be granted the “Test Distribution” access control role to be able to use Test Distribution.
| Builds and agents connect to the Develocity server using a WebSocket connection. Thus, you need to ensure WebSocket support is enabled on every load balancer or proxy that’s used in between the build/agent and the Develocity server. |
The Jobs tab provides an overview of the currently active jobs, where each row represents a running test task or goal in a Gradle or Maven build, respectively.
The first part of the label indicates the root project name of the build, while the second part identifies the particular task or goal. The count of assigned/desired agents shows how many agents are currently assigned to/desired by the job. More agents may be assigned as they become available. The count of connected agents shows how many currently connected agents have capabilities that satisfy the requirements of the job. Similarly, the count of compatible agents shows the theoretical number of compatible agents that could be provided by agent pools.
Agents
Utilizing Test Distribution requires connecting additional compute resources, called agents, to Develocity. The Test Distribution agent is available as a Docker image or standalone JAR and is easy to deploy and operate via Kubernetes or any modern compute platform. Consult the Develocity Test Distribution Agent User Manual for more on deploying and operating agents.
The Agents tab shows all currently connected agents.
Agent Pools
An agent pool is a group of agents that provide the same capabilities.
Auto Scaling
Agents in a pool can be scaled up and down based on-demand. Each pool has a generated unique ID that’s specified by an agent when connecting to indicate its pool, and can be used by compute platform autoscalers to determine the desired number of agents at any time for that pool.
| Refer to the Develocity Test Distribution Agent User Manual for step-by-step instructions for auto scaling with Kubernetes. |
Minimum and Maximum Size
The minimum and maximum size of an agent pool is used to determine the desired number of agents at a given moment.
Ultimately, the actual number of agents is determined by the compute platform, which may impose incompatible limits. Where possible, ensure that the limits imposed by the compute platform are either identical to or compatible with those configured in Develocity.
To temporarily disable a pool, set both minimum and maximum to zero.
Capabilities
Agents in a pool are expected to provide a certain set of capabilities. While an agent may provide additional capabilities, it will be rejected when connecting without at least the mandatory capabilities configured on its agent pool.
While computing the number of desired agents for a pool its capabilities are matched against the requirements of the current jobs.
A pool is only utilized if it satisfies all of a job’s requirements.
Therefore, you should at least add a jdk=… capability to each agent pool since such a requirement is added implicitly to each job.
Moreover, each agent has an implicit os=… capability which should be added to the agent pool for the sake of consistency and in case it’s an explicit requirement of any job.
Already connected agents that don’t provide all mandatory capabilities will stay connected for up to 4 hours after adding capabilities to an agent pool. Any agent in the pool that has capabilities that don’t exactly match the ones of the pool is shown with a warning indicator on the Agents tab.
To add a new capability to an existing pool, it’s recommended to first add the capability to all connected agents and then edit the agent pool. This way agents connecting won’t be rejected while the change is being applied (e.g. due to a rolling update).
To remove a capability from an agent pool, we recommend to first remove the capability from the pool before removing it from connected agents. Agents in the pool that still provide the capability will no longer match the corresponding build requirement and not be assigned.
Project Groups
If project-level access control is enabled, usage of an agent pool can be restricted to specific project groups. In that case, only builds for projects in the specified groups can run tests on agents in this pool for users with access to those projects.
Agents from unrestricted pools can be used by any project. This is useful when Test Distribution agents provide access to secured resources, such as databases or other services, that shouldn’t be accessible to all projects. To prevent misconfiguration, such agent pools require using pool-specific registration keys for authentication.
Registration Keys
Registration keys are a pool-specific means to authorize agents with Develocity and the pool they’re connecting to. They can be generated by editing an agent pool on the Configuration tab.
A single registration key may be used my multiple agents in the same pool, but not by an agent in another pool. Multiple registration keys can be created to facilitate a rolling update if required. Revoking a registration key prevents it from being used for future registrations. Connected agents will be forcibly disconnected one hour after the registration key they used to connect is revoked.
Priority
Pools with higher priority are utilized before pools with lower priority. If tests to be executed can be executed by agents from different pools, agents of the pool with the highest priority are requested. When scaling to meet demand, higher priority pools are scaled to their configured maximum before scaling lower priority pools.
You can change the priority of agent pools by dragging them on the Configuration tab. Pools that are higher on the list are prioritized over those below.
Deletion
Agent pools can be deleted at will when they’re no longer needed. It’s recommended to stop all agents in the agent pool before deleting it. Already connected agents will stay connected for up to 4 hours after deleting an agent pool. During that time they will be shown with a warning indicator on the Agents tab.
Pool Status
Agent auto scaling relies on auto scaling machinery of a compute platform regularly querying Develocity for a pool’s desired size and starting/stopping agents to match the desired size.
The https://«hostname»/api/test-distribution/agent-pools/«pool-id»/status endpoint conveys a pool’s desired size at the moment, along with other information about the pool, as a JSON document.
Accessing the endpoint requires an access key provided via an Authorization: Bearer «access-key» HTTP header, with the user having the Test Distribution permission.
It can also be accessed in a browser when signed in to Develocity and having the Test Distribution permission.
To create an access key, log in to Develocity and go to https://«hostname»/settings/access-keys and click on the Generate button.
The following demonstrates using curl to access the endpoint:
curl -H "Authorization: Bearer «access-key»" https://«hostname»/api/test-distribution/agent-pools/«pool-id»/statusWhich will produce output of the following form:
{
"id": "«pool-id»",
"name": "Linux",
"capabilities": [
"jdk=11",
"os=linux"
],
"minimumAgents": 1,
"maximumAgents": 100,
"connectedAgents": 25,
"idleAgents": 9,
"desiredAgents": 16
}
Autoscalers need only consider the desiredAgents field and adjust the number of running agents to match this number.
Historical Usage
The Usage tab provides a historical overview of workload in terms of number of active jobs, agent utilization, and agent pool metrics, for the last hour, 24 hours, or 7 days.
By default, all jobs, agents, and agent pools are shown. Adding requirement/capability criteria shows only the jobs with at least the specified requirements, only the agents with at least the specified capabilities, and only the agent pools with at least the specified capabilities.