Develocity Python Agent User Manual (beta)

Getting set up

Setup

The Develocity Python agent is a Python package, but it is not publicly available on PyPI. To install it in your virtual environment, you can use the command line below.

pip install https://develocity-python-pkgs.gradle.com/develocity_agent-0.9-py3-none-any.whl

Alternatively, you can add the Develocity Python agent directly in your requirements.txt file (or equivalent):

requirements.txt

... your other requirements ...
https://develocity-python-pkgs.gradle.com/develocity_agent-0.9-py3-none-any.whl

Python support needs to be enabled in your license to use the Develocity Python agent. Reach out to us to enable Python support.

After installing the agent, you can use it in your Python runs.

Publishing a Build Scan to scans.gradle.com is not supported. The minimum configuration required is the Develocity server URL.

The agent is configured through a file named .develocity.py which should be located at root of your Python project. The configuration file must define a function called develocity_configuration which takes as parameter the current configuration and returns the updated configuration. Following is an example of minimal configuration:

develocity.py

def develocity_configuration(configuration):
    configuration.develocity_url = 'https://develocity.mycompany.com'
    return configuration

Authenticating with Develocity

Develocity installations may be configured to require Build Scan publishing to be authenticated. Additionally, installations may be configured to only allow certain users to publish Build Scans.

Develocity access keys should be treated with the same secrecy as passwords. They are used to authorize access to Develocity from a build.

Access key configuration

Creating access keys

To create a new access key, sign in to Develocity and access “My settings” via the user menu at the top right of the page. From there, use the “Access keys” section to generate an access key.

The access key may be specified in the .develocity.py configuration file or via the DEVELOCITY_ACCESS_KEY environment variable.

Configuring through `.develocity.py`

The access key may be specified in the .develocity.py configuration file. Below is an example of configuration. Please refer to the Configuration reference for more information.

develocity.py

def develocity_configuration(configuration):
    configuration.develocity_url = 'https://develocity.mycompany.com'
    configuration.access_key = '7w5kbqq<...>jdudtq'
    return configuration

Configuring through environment variables

The environment variable value format is «server host name»=«access key».

$ export DEVELOCITY_ACCESS_KEY=develocity.mycompany.com=7w5kbqqjea4vonghohvuyra5bnvszop4asbqee3m3sm6dbjdudtq && \
  pip list

The server host name is specified in order to prevent the access key being transmitted to a different server than intended. In the rare case that you require access keys for multiple servers, you can specify multiple entries separated by semicolons.

$ export DEVELOCITY_ACCESS_KEY=develocity1.mycompany.com=7w5kbqqjea4vonghohvuyra5bnvszop4asbqee3m3sm6dbjdudtq;develocity2.mycompany.com=9y4agfiubqqjea4vonghohvuyra5bnvszop4asbqee3m3sm67w5k && \
  pip list

Short-lived access tokens

Develocity access keys are long-lived, creating risks if they are leaked. To avoid this, users can use short-lived access tokens to authenticate with Develocity. Access tokens can be used wherever an access key would be used. Access tokens are only valid for the Develocity instance that created them.

Develocity server version 2024.1+ supports access tokens.

Changing a Develocity instance’s hostname will cause all existing access tokens to become invalid.

To create an access token:

Get an access key or access token for the user you wish to create a token for.
Decide which permissions the new access token should have.
If project-level access control is enabled, decide which projects the new access token should be able to access.
Decide how long the new access token should live.
Make a POST request to /api/auth/token, optionally with the following parameters. The response will be the access token.
1. A permissions= query parameter with the config values of each permission you wish to grant. By default, all permissions for the credential used to authenticate the token request are granted to the created token.
2. If project-level access control is enabled, a projectIds= query parameter with the ID of each project you wish to grant access to. By default, all projects for the credential used to authenticate the token request are granted to the created token.
3. An expiresInHours= query parameter with the token’s intended lifetime in hours, with a maximum of 24. The default is two hours, or the remaining lifetime of the credential used to authenticate the request, whichever is smaller.

The requested permissions and project ids can be specified as comma-seperated lists or repeated parameters. For example, ?projectIds=a,b&projectIds=c is valid and will request projects a, b, and c.

If project-level access control is not enabled, all access tokens will be granted the “Access all data without an associated project” permission even if it is not explicitly requested.

If the user creating the token does not have one of the requested permissions or projects, Develocity will respond with a 403 Forbidden error. If an access token is used to authenticate the creation request, its permissions and projects will be used for this check instead of the user’s. The request will also error if the requested lifetime would cause the new access token to expire after the one used to authenticate the request. Together, this means you cannot create an access token with more access or a later expiration than the credentials used to authenticate the request.

See the API documentation for more details on the /api/auth/token endpoint.

Here is an example using CURL to create an access token:

$ curl -X POST https://ge.mycompany.com/api/auth/token?permissions=publishScan,writeCache,accessDataWithoutAssociatedProject&projectIds=project-a,project-b&expiresInHours=1 \
    -H "Authorization: Bearer 7asejatf24zun43yshqufp7qi4ovcefxpykbwzqbzilcpwzb52ja"

eyJraWQiOiJ0ZXN0LWtleSIsImFsZyI6IlJTMjU2IiwidHlwIjoiSldUIn0.eyJpc19hbm9ueW1vdXMiOmZhbHNlLCJwZXJtaXNzaW9ucyI6WyJSRUFEX1ZFUlNJT04iLCJFWFBPUlRfREFUQSIsIkFDQ0VTU19EQVRBX1dJVEhPVVRfQVNTT0NJQVRFRF9QUk9KRUNUIl0sInByb2plY3RzIjp7ImEiOjEsImIiOjJ9LCJ1c2VyX2lkIjoic29tZS1pZCIsInVzZXJuYW1lIjoidGVzdCIsImZpcnN0X25hbWUiOiJhIiwibGFzdF9uYW1lIjoidXNlciIsImVtYWlsIjoiYkBncmFkbGUuY29tIiwic3ViIjoidGVzdCIsImV4cCI6NzIwMCwibmJmIjowLCJpYXQiOjAsImF1ZCI6ImV4YW1wbGUuZ3JhZGxlLmNvbSIsImlzcyI6ImV4YW1wbGUuZ3JhZGxlLmNvbSIsInRva2VuX3R5cGUiOiJhY2Nlc3NfdG9rZW4ifQ.H1_NEG1xuleP-WIAY_uvSmdd2o7i_-Ko3qhlo04zvCgrElJe7_F5jNuqsyDfnb5hvKlOe5UKG_7QPTgY9-3pFQ

The resulting token would have the following permissions:

“Publish Build Scans”
“Read and write Build Cache data”
“Access all data without an associated project”

And it would have access to these projects:

“project-a”
“project-b”

The token would only be usable for one hour.

Using Build Scans

Build Scans are a record of what happened during a build, captured and visualized by Develocity.

Build Scans are an important tool for developing and maintaining Python projects. They provide insights into exactly what your builds are doing, helping you identify problems with the build environment, performance, and more. They can also help you understand and improve the build more generally, and make collaborating with others easier.

Figure 1. Build Scans can be published to Develocity

Capturing scripts output

The output generated during the Python run, excluding the output produced by forked processes, is captured and displayed in Build Scans.

Extending Build Scans

You can include extra custom information in your Build Scans through tags, links, and values. This is a powerful mechanism for capturing and sharing information important to your build and development process.

This information can be anything you like. You can tag all builds run by your continuous integration tool with a CI tag. You can capture the name of the environment that the build published to as a value. You can link to the source revision for the build in an online tool such as GitHub. The possibilities are endless.

You can see how the custom data appears in the following figures:

Figure 2. A Build Scan containing tags and links

Develocity allows listing and searching across all the Build Scans in the system. You can find and filter Build Scans by tags and custom values, in addition to project name, outcome and other properties. In the following figure, for example, we are filtering for all Python Build Scans that have the tag "CI" and a git branch name of "release":

Figure 3. A filtered list of Build Scans in Develocity

Adding tags

Tags are typically used to indicate the type or class of a build, or a key characteristic. They are prominent in the user interface and quickly inform a user about the nature of a build. A build can have zero or more tags.

Tags can be added to a build scan by configuring them in .develocity.py or by setting environment variables.

Note that the order in which you declare the tags doesn’t affect the Build Scan view. They are displayed in alphabetical order, with any all-caps labels displayed before the rest.

Configuring through `.develocity.py`

The Develocity configuration object contains an attribute named tags, a set of strings, which contains the tags to add to the build scan.

develocity.py

def develocity_configuration(configuration):
    configuration.tags.add('CI')
    return configuration

Configuring through environment variables

To add a tag, add an environment variable prefixed with DEVELOCITY_TAG_ and assign the string true to it.

Add the DEVELOCITY_TAG_<tag> environment variable

export DEVELOCITY_TAG_CI=true

You can see the effect of a custom tag in figure 2.

The Develocity Python agent imposes limits on captured tags:

maximum tag count: 50
maximum tag length: 200 characters

Adding links

Builds rarely live in isolation. Where does the project source live? Is there online documentation for the project? Where can you find the project’s issue tracker? If these exist and have a URL, you can add them to the Build Scan.

Tags can be added to a build scan by configuring them in .develocity.py or by setting environment variables.

Configuring through `.develocity.py`

The Develocity configuration object contains an attribute named links, a dictionary of strings to strings, which contains the links to add to the build scan.

develocity.py

def develocity_configuration(configuration):
    configuration.links['VCS'] = 'https://github.com/myorg/my-super-project/tree/my-new-feature'
    return configuration

Configuring through environment variables

To add a link, add an environment variable prefixed with DEVELOCITY_LINK_. What comes after the prefix will be the name associated with the link, while the value of the environment variable will determine the URL it points to.

Add the DEVELOCITY_LINK_<name>=<URL> environment variable

export DEVELOCITY_LINK_VCS=https://github.com/myorg/my-super-project/tree/my-new-feature

The <name> is a string identifier that you choose, and that means something to you.

You can see the effect of a custom link in figure 2, which shows how a label for a given VCS (here GitHub) becomes a hyperlink that anyone viewing the Build Scan can follow.

The Develocity Python agent imposes limits on captured links:

The link must use the http, https, or mailto scheme
Maximum link count: 20
Maximum link label length: 100 characters
Maximum link URL length: 100,000 characters

Adding custom values

Some information just isn’t useful without context. What does "1G" mean? You might guess that it represents 1 gigabyte, but of what? It’s only when you attach the label "Max_heap_size_for_build" that it makes sense. The same applies to git commit IDs, for example, which could be interpreted as some other checksum without a suitable label.

Custom values are designed for these cases that require context. They’re standard key-value pairs, in which the key is a string label of your choosing and the values are also strings, often evaluated from the build environment.

Custom values can be added to a build scan by configuring them in .develocity.py or by setting environment variables.

Configuring through `.develocity.py`

The Develocity configuration object contains an attribute named custom_values, a dictionary of strings to strings, which contains the custom values to add to the build scan.

develocity.py

def develocity_configuration(configuration):
    configuration.custom_values['CIBuildType'] = 'QA_Build'
    return configuration

Configuring through environment variables

To add a custom value, add an environment variable prefixed with DEVELOCITY_VALUE_. What comes after the prefix will be the key, while the value of the environment variable will determine what the custom value is.

Add the DEVELOCITY_VALUE_<name>=<value> environment variable

export DEVELOCITY_VALUE_CIBuildType=QA_Build

As with tags, you can filter Build Scans by custom values in Develocity.

The Develocity Python agent imposes limits on captured custom values:

maximum custom value count: 1,000
maximum custom value key length: 1,000 characters
maximum custom value value length: 100,000 characters

Appendix A: Captured information

The Develocity Python agent captures information while the Python interpreter is running and transmits it to Develocity after it has completed.

Most of the information captured can be considered to be build data. This includes the name of the projects in your build and more general environmental information, including your Python version, operating system, hardware, country, timezone and other things of this nature.

Notably, the actual source code being built and the output artifacts are not captured. However, error messages emitted by compilers or errors in tests may reveal aspects of the source code.

Listing

The list below details the notable information captured by the Develocity Python agent and transmitted in a Build Scan.

Environment
- Username (https://docs.python.org/3/library/getpass.html#getpass.getuser)
- Public hostname (https://docs.python.org/3/library/socket.html#socket.gethostname)
- Process
  - Python version
  - Locale
    
    Language
    
    Country
    
    Variant
    
    Timezone
- Operating System (https://docs.python.org/3/library/platform.html#platform.uname)
  - Platform
  - Release
  - Architecture
- Hardware
  - Number of processors (https://docs.python.org/3/library/os.html#os.cpu_count)
Build
- Requested command
- Console output (including exception messages and stack traces)

Access

Build Scans published to a Develocity installation are viewable by all users that can reach the server and have the required roles, should Identity Access Management (IAM) be turned on. Develocity provides a search interface for discovering and finding individual Build Scans.

Appendix B: Configuration reference

Configuration file

All aspects of the Develocity Python agent can be configured with a configuration file, named .develocity.py and located at the root of your Python project. The configuration file must define a function called develocity_configuration which takes as parameter the current configuration and returns the updated configuration. The configuration object has the following attributes:

develocity_url (required): The URL of the Develocity server that the Build Scan will be uploaded to.
access_key (optional): The access key for authenticating with the Develocity server.
project_name (optional): The project name to associate with the Build Scan. If not provided, the project name will be extracted from pyproject.toml (if available), or from the name of the directory containing the configuration file (if any), or left blank.
custom_values (optional): A dictionary of string to string mappings that will be added to the Build Scan.
tags (optional): A set of strings that will be added as tags to the build scan.
links (optional): A dictionary of string (labels) to string (URLs) mappings that will be added as hyperlinks to the Build Scan.
project_id (optional): The project identifier.

If the configuration file is not present and the Develocity server URL is not defined through an environment variable either (see below), the agent will not upload a Build Scan. If the configuration file is present, but does not define a develocity_configuration function, or the function is defined but does not have the expected signature, the agent will not upload a Build Scan.

Example

Below is an example of configuration file.

develocity.py

def develocity_configuration(configuration):
    configuration.develocity_url = 'https://develocity.mycompany.com'
    configuration.access_key = '7w5kbqq<...>jdudtq'
    return configuration

Determining the project name

The project name is determined as follows: - If a project name is configured in .develocity.py under project_name, this value is used. - Otherwise, if a pyproject.toml is found in the project directory, the value project.name is used if available, otherwise the value tool.poetry.name is used. - If no project name is found in the configuration file or pyproject.toml, the name of the directory containing the configuration file is used. - Otherwise, the project name is left blank and will be displayed as (N/A) in Develocity.

Environment variables

Both the Develocity server URL and Develocity access key can be configured through environment variables. When the Develocity server URL or Develocity access key are configured through both environment variables and the configuration file, the values set through environment variables take precedence.

DEVELOCITY_ACCESS_KEY: A semi-colon-separated list of <host>=<key>. When connecting to <host>, <key> will be provided as an authentication token to the configured Develocity server.
DEVELOCITY_URL: The URL of the Develocity server that the Build Scan will be uploaded to.

Appendix C: Release history

0.9

12th December 2024

[NEW] Initial beta release compatible with Develocity 2024.3

Compatible with Develocity 2024.3 or later.

Appendix D: Compatibility with Python and Develocity

Compatibility between versions of Python, Develocity, and the Develocity Python agent can be found in the compatibility matrix.

Appendix E: Known issues

Windows compatibility

Build scans generated on Windows machines are not supported. While a build scan can be produced on a Windows machine, it may be missing information and console output may not look as expected.

Appendix F: Python tools supported by Develocity

The Develocity agent for Python supports the following tools:

autopep8
black
django
flake8
isort
mypy
pip
poetry
prospector
pycodestyle
pylint
pyright
pytest
pytype
venv

Appendix G: Verifying the signature of the agent package

The agent package is published alongside its signature. The public key is published to https://keys.openpgp.org. You can verify the signature as follows:

$ curl -OL https://develocity-python-pkgs.gradle.com/develocity_agent-0.9-py3-none-any.whl && \
  curl -OL https://develocity-python-pkgs.gradle.com/develocity_agent-0.9-py3-none-any.whl.asc && \
  gpg --keyserver keys.openpgp.org --recv-key  7B79ADD11F8A779FE90FD3D0893A028475557671 && \
  gpg --verify develocity_agent-0.9-py3-none-any.whl.asc develocity_agent-0.9-py3-none-any.whl

The output of the last command should look similar to the following:

gpg: Signature made Thu Sep 28 16:17:46 2023 CEST
gpg:                using RSA key 893A028475557671
gpg: Good signature from "Gradle Inc. <info@gradle.com>" [unknown]
gpg:                 aka "Gradle Inc. <maven-publishing@gradle.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7B79 ADD1 1F8A 779F E90F  D3D0 893A 0284 7555 7671

This verifies that the artifact was signed with the private key that corresponds to the imported public key. The warning is emitted because you haven’t explicitly trusted the imported key (hence [unknown]). One way of establishing trust is to verify the fingerprint over a secure channel. Please contact technical support should you wish to do so.