This manual covers the installation and operation of Gradle Enterprise into an existing Kubernetes cluster. It is useful for administrators and maintainers of Gradle Enterprise installations, and build engineers that maintain or develop Gradle or Apache Maven™ builds that leverage Gradle Enterprise.

This approach to operating Gradle Enterprise is only relevant if you are already operating a Kubernetes cluster. If you are not operating a Kubernetes cluster, you can alternatively install the Gradle Enterprise appliance.

Installation requirements

This section outlines the installation resource requirements and configuration aspects that need to be resolved during installation.

Supported Kubernetes versions

Gradle Enterprise has been tested as compatible with Kubernetes API versions 1.9.x to 1.18.x. Later versions may be compatible but have not been verified to work.

There are known issues with Kubernetes versions 1.9.4 and 1.9.5 around volume subpaths that make them incompatible with Gradle Enterprise.

Supported Kubernetes platforms

Gradle Enterprise does not use any platform specific features and is expected to work on all platforms, but not all are verified to work. Gradle Enterprise is verified to work on Kubeadm, K3s, EKS, and OpenShift. It has also been tested as compatible with GKS and AKS.

Gradle Enterprise installer requirements

The Gradle Enterprise installer needs to be executed from a machine with access to the internet. The resulting installation shell script needs to be executed from a machine with kubectl, or oc in the case of an OpenShift installation, configured to communicate with your cluster. Those might be the same machine. Otherwise, the folder where the Gradle Enterprise installer was executed will have to be copied across. The user running the installer should also have permissions to create Kubernetes resources in the designated namespace.

Storage requirements

Storage Class

Gradle Enterprise uses persistent volume claims for storing logs and data. During the installation you will need to provide the name of the desired storage class to be used for provisioning persistent volumes.

Some pods are associated with multiple persistent volumes and for Kubernetes platforms with multiple availability zones, the pods and their persistent volumes must be located in the same zone. In this case we recommend using a storage class with a volumeBindingMode of WaitForFirstConsumer to ensure that all persistent volumes are provisioned in the same zone that the pod was scheduled in.

Capacity

The recommended minimum capacity to provide for Builds Scans is 250 GB, 10 GB for Build Cache, and 10 GB for Test Distribution file cache. If you are producing many build scans a day or intend to retain build scans for long periods of time you may want to consider provisioning more storage. If your storage provisioner does not allow expanding volumes you should also consider preparing for future data growth.

Performance

Disk performance has a significant impact on Gradle Enterprise. For production workloads, storage volumes should exhibit SSD-class disk performance of at least 3000 input/output operations per second (IOPS).

Networking configuration

Firewall and proxy configuration

The system running the installer will need access to the internet for retrieving application and installer images as well as validating license information. If you require a proxy for internet access this should be pre-configured on the system running the installer. You should be able to use command line tools like curl as well as issue commands like docker pull against public registries.

Application hostname

When installing Gradle Enterprise, you will be asked to provide a hostname. This should be the hostname that users of the installation use to access it and therefore should resolve within your network. This may depend on how you intend on directing external traffic into your cluster to access Gradle Enterprise. Please refer to this section for more information.

HTTPS SSL certificate

It is strongly recommended that production installations of Gradle Enterprise are configured to use HTTPS with a trusted certificate.

Gradle Enterprise natively supports serving traffic over HTTPS when configured with a certificate and key. If you intend to use an ingress controller for directing external traffic to Gradle Enterprise, you may opt to terminate HTTPS there.

Untrusted SSL certificates

By default, Gradle Enterprise uses the default trust settings of the Java runtime that it ships with when connecting to other systems using SSL. If your organization uses certificates that are not signed by a trusted certificate authority, you must perform additional configuration for this to work. This may be the case if you use self-signed certificates or an internal certificate authority.

Additional certificates to trust can be configured by during installation/configuration by enabling “Trust additional certificates for SSL communication” and uploading all required certificates as a single PEM file.

If using self-signed certificates, the certificate of each service must be uploaded. If using an internal certificate authority, its certificate must be uploaded.

Saving the installation directory

The Gradle Enterprise installer produces installation shell scripts and saves the entered configuration in the directory from which the installer is run. In order to make reconfiguring and upgrading easier, this directory should be saved and stored somewhere for later use. Reusing this directory when reconfiguring or upgrading saves having to reenter your installation configuration.

Consider dedicating a persistent storage area where this directory can be copied or archived to after installation.

Installation flavors

Gradle Enterprise supports multiple installation mechanisms.

Standard installation requires the Kubernetes cluster to have internet access for pulling Docker images from public internet registries. During operation, Gradle Enterprise does not require a connection to the internet.

Airgapped installation can be performed on a Kubernetes cluster without an internet connection. An internal Docker registry can be used either as a registry mirror or to host the Gradle Enterprise Docker images, in which case they are pushed there as part of the installation.

Unattended installation

Gradle Enterprise can be installed without user intervention, by specifying configuration options in files before starting the installer. This allows scripted installations or installation by configuration management tools. For assistance in performing an unattended installation, please submit a support ticket at support.gradle.com.

Installation

There are two steps to deploying Gradle Enterprise to a Kubernetes cluster:

  1. Running the Gradle Enterprise installer to produce an installation shell script, tailored to your environment

  2. Executing the installation shell script to install Gradle Enterprise onto your Kubernetes cluster

The Gradle Enterprise installer is a locally run web application, that can be run on any machine with access to the Internet (it does not need to be run on the cluster). After entering your configuration, the installation shell script (and supporting files) will be created on the host running the installer.

If that Internet connected machine can’t talk with the Kubernetes cluster, the generated files will have to be copied to the airgapped datacenter to execute the installation shell script from there.

It is not necessary for Gradle Enterprise to have access to the Internet once running on the cluster. See the Airgapped installation section for more details.

Subsequently, updating your Gradle Enterprise version or changing installation configuration involves regenerating the installation script and running it again.

Creating the install script

Standard installation

  1. Prior to installation you will be provided a customer ID and installation ID. You will need to provide these as parameters (cust-id resp. install-id) to the install command shown in the next step.

  2. Execute the following command from the installation machine. Supplying the password inline is not recommended to avoid unintentionally leaking the values in the shell history or the process table. You can simulate supplying a password inline by adding your password to a file, your_password.txt in the example, and piping it into STDIN.

    mkdir -p ~/gradle-enterprise && \
    cd ~/gradle-enterprise && \
    cat your_password.txt | docker login --username <your-username> --password-stdin && \
    docker run -p 8800:8800 --rm -it -v `pwd`:/out \
      -v /var/run/docker.sock:/var/run/docker.sock \
      replicated/ship:0.54.1 app \
      --customer-id=<cust-id> \
      --installation-id=<install-id>
    If you are using Docker on a non-Linux system to perform the installation then you must run the installer from a folder inside your user home directory in order for the installer to save the generated installation scripts to your local machine.
  3. Instructions on the console will ask you to open http://localhost:8800 on a browser.

    If you are using a non-Linux system and Docker Toolbox (i.e. boot2docker) to run Docker in a virtual machine then you will instead need to browse to port 8800 on the IP address of your Docker VM. You can find the IP of your VM by running docker-machine ip from a command line. Additionally, if you are using a proxy server for internet access this you must configure this IP to bypass your internal proxy.
  4. Provide the name of the existing Kubernetes namespace that Gradle Enterprise resources should be created in. If you have not yet created this namespace you should do so now. Lookup this documentation as reference.

    namespace 2021.1
  5. If using HTTPS as recommended, you’ll need to provide your SSL Private Key and SSL Certificate.

  6. For Storage see the above listed requirements.

  7. Select Save changes, then select Continue to next step to proceed with the installation.

  8. Next, the installer will ask you to execute:

    ./installer/scripts/install.sh

    If the machine doesn’t have access to the Kubernetes cluster, copy the contents of the current folder (including the hidden .ship folder) to a machine within the airgapped network. This script will install all Gradle Enterprise resources into your Kubernetes cluster.

  9. The installer will persist all configuration values and install script in the directory from which you ran the installer. Consider storing this directory somewhere safe and reusing it when reconfiguring or upgrading your installation.

Airgapped installation

If your Kubernetes cluster is not able to pull Docker images from public internet registries, then you can create an airgapped install script that will use an internal Docker registry instead.

  1. Prior to installation you will be provided a customer ID and installation ID. You will need to provide these as parameters (cust-id resp. install-id) to the install command shown in the next step.

  2. Execute the following command from the installation machine:

    mkdir -p ~/gradle-enterprise && \
    cd ~/gradle-enterprise && \
    docker run -p 8800:8800 --rm -it -v `pwd`:/out \
      -v /var/run/docker.sock:/var/run/docker.sock \
      replicated/ship:0.54.1 app \
      --customer-id=<cust-id> \
      --installation-id=<install-id>
    If you are using Docker on a non-Linux system to perform the installation then you must run the installer from a folder inside your user home directory in order for the installer to save the generated installation scripts to your local machine.
  3. Instructions on the console will ask you to open http://localhost:8800 on a browser.

    If you are using a non-Linux system and Docker Toolbox (i.e. boot2docker) to run Docker in a virtual machine then you will instead need to browse to port 8800 on the IP address of your Docker VM. You can find the IP of your VM by running docker-machine ip from a command line. Additionally, if you are using a proxy server for internet access this you must configure this IP to bypass your internal proxy.
  4. Provide the name of the existing Kubernetes namespace that Gradle Enterprise resources should be created in. If you have not yet created this namespace you should do so now. Lookup this documentation as reference.

    namespace 2021.1
  5. Select the Airgap cluster setting. You will need to provide the address of an internal Docker registry where Gradle Enterprise images will be pulled from, and select whether it is a mirror. If your registry is not a mirror, Gradle Enterprise images will be pushed to it by the installer. If your Kubernetes cluster additionally requires the use of a pull secret to pull images from this registry you may provide the name of the pull secret as well. You will need to create this pull secret in the same namespace as specified above in step 4.

    airgap 2021.1
  6. If using HTTPS as recommended, you’ll need to provide your SSL Private Key and SSL Certificate.

  7. For Storage see the above listed requirements.

  8. Select Save changes, then select Continue to next step to proceed with the installation.

  9. Docker images will be downloaded ready for pushing to the Docker registry specified above in step 5.

  10. Next, the installer will ask you to execute:

    ./installer/scripts/install.sh

    If the machine doesn’t have access to the Kubernetes cluster, copy the contents of the current folder (including the hidden .ship folder) to a machine within the airgapped network. This script will install all Gradle Enterprise resources into your Kubernetes cluster. All necessary images will also be pushed to your internal Docker registry.

  11. The installer will persist all configuration values and install script in the directory from which you ran the installer. Consider storing this directory somewhere safe and reusing it when reconfiguring or upgrading your installation.

Exposing Gradle Enterprise outside your cluster

The Gradle Enterprise Kubernetes installer creates a service named gradle-proxy exposing ports 80 and 443 for accessing the application. This service uses a ClusterIP for communication meaning it is not accessible outside of the Kubernetes cluster by default. To access Gradle Enterprise outside of your cluster you will need to create some kind of entrypoint to this service into the cluster. Potential methods for this are via an ingress, explicit node port, external IP service or a load balancer. The method you choose depends on your cluster and network configuration.

We do not recommend editing the existing gradle-proxy service to enable external access. Changes to configuration or subsequent Gradle Enterprise updates will overwrite these changes. If you want to use a node port or external IP instead, you should define a new service in the same namespace.

Operational support scripts

The Gradle Enterprise installer generates a number of support scripts in addition to the installation script in the ./installer/scripts folder in the installation directory. These scripts include:

Script Description

install.sh

Installs (or upgrades) Gradle Enterprise resources in your Kubernetes cluster

uninstall.sh

Removes all Gradle Enterprise resources from your Kubernetes cluster

support_bundle.sh

Generates a support bundle containing various log files used for troubleshooting Gradle Enterprise issues

create_backup.sh

Triggers a backup of Gradle Enterprise database

copy_backup.sh

Copies selected backup archive from Gradle Enterprise server to the local system

restore_backup.sh

Restores Gradle Enterprise data from a previously created backup archive

test_notification.sh

Sends a test email using the SMTP settings specified in the “Email notifications” section.

db_index_check.sh

Triggers integrity check of database indexes.

It is recommended that you keep these scripts available for future use. Should they be misplaced, simply run the installer again to regenerate them.

Post-installation setup

Basic installation settings for Gradle Enterprise are configured via the Gradle Enterprise installer. This includes how Gradle Enterprise is accessed via the network, and how it stores its persistent data. Other configuration aspects are configured via the Gradle Enterprise application. Only users with the administration permission can do this, and they can do so by using the user menu in the top right of the application and choosing “Administration”. Until user accounts have been created, or in the case that no user accounts will be used, the system user can be used for this purpose.

Please read each of the following sections for information on recommended configuration.

System user password

The “system user” is an ever-present local user account that can be used to bootstrap the configuration of an installation, or for emergency local access in case of a failure with an external authentication provider once configured.

It is strongly recommended to immediately change the system user password from its default after installation.

To change the system user password, visit Gradle Enterprise in a web browser via the hostname that was specified during installation. Use the “Sign in” button at the top right of the page to sign in as the system user by using the username system and password default. You will then be required to specify a new password for the system user account. The new password should be recorded and kept secret as it can be used to access Gradle Enterprise as an administrator.

After setting the system user password, you will be redirected to the administration section.

It is recommended that the system user account not be used regularly. Instead, real administrator user accounts should be created by configuring access control.

Access control

Gradle Enterprise allows locally managed user accounts and permissions, and externally managed via an LDAP service or SAML 2.0 identity provider.

Initial setup can be performed by the system user. Subsequently, any user with the administration permission can configure access control by using the user menu in the top right of the application to access “Administration” and then “Access control”.

The following permissions are available to grant access to the described function:

Permission Description Default role name

Build scan view

Allows viewing of build scans and associated build data

ge-scans-viewing

Build scan publish

Allows publishing of build scans

ge-scans-publishing

Test distribution

Allows use of test distribution

ge-distributed-testing

Export API

Allows consuming build scan data via the Export API

ge-data-export

Cache Admin

Allows administration of build cache functionality

ge-cache-administration

Admin

Allows general administration of Gradle Enterprise (e.g. access control)

ge-administration

For locally managed accounts, permissions must be assigned for each user. A default set for all new users can be specified.

For externally managed user accounts (LDAP or SAML 2.0), permissions can be managed locally or by group/role membership defined by the provider. When using provider-defined membership, each permission can be mapped to one or more roles. By default, each permission maps to a unique role with the “default role name” specified in the table above. This can be changed for each permission.

Anonymous access

By default, Gradle Enterprise allows anonymous viewing and creation of build scans, along with anonymous use of the built-in build cache. This makes it easier to get started by reducing build configuration, but may not be suitable for your environment.

Anonymous build scan viewing and publishing can be changed by going to “Administration” via the top right hand user menu, then “Access control” → “Anonymous access”.

Anonymous build cache access can be changed by going to “Build cache” via the top right hand user menu, then “Nodes” → ”View built-in node details” → “Cache access control”.

Authenticated build scan publishing

By default, build scan publishing does not require any build configuration as anonymous build scan publishing is enabled. If this is disabled, builds wanting to publish build scans will need to be configured with an access key of a user with permission to publish build scans.

Please consult the Gradle Enterprise Gradle Plugin User Manual or Gradle Enterprise Maven Extension User Manual for guidance on how to configure a build environment for authenticated build scan publishing.

For production installations, it is recommended to disable anonymous build scan publishing.

Local users

Locally managed user accounts can be created to allow users to access Gradle Enterprise. They are not mutually exclusive with externally managed user accounts and both can co-exist provided the usernames and emails are unique.

Setup
  1. As an administrator, navigate to “Access control” → “Users”

  2. Click Add.

  3. Enter details for the user and set an initial password.

  4. Assign the required roles for the user.

  5. Click Save.

SAML 2.0

A SAML 2.0 identity provider can be configured to allow users to access Gradle Enterprise using their organization credentials. User accounts for users authenticating with the SAML provider will be created on first login.

A user cannot login via a SAML provider if a locally defined account exists for the same username or email.

Sign out from Gradle Enterprise will not log users out of the SAML identity provider.

Setup
  1. As an administrator, navigate to “Access control” → “Identity provider”

  2. Check ”Enable external identity provider”

  3. Choose “SAML 2.0” from Identity provider type options

  4. Enter a name for the identity provider

  5. Create a SAML application at your identity provider using the displayed “Service provider SSO URL” and “Service provider entity ID”.

  6. Download the metadata for the SAML application from your identity provider, and select this file for the “Identity provider metadata file” field.

  7. Configure signing/encryption options

  8. Configure attribute mapping (described below)

  9. Configure role management (described below)

  10. Click “Save”.

If any signing or encryption is being used, use “Download service provider config” to obtain a configuration file that will need to provided your identity provider.

Attribute mappings

User’s “first name”, “last name” and “email” attributes can be obtained from the identity provider or prompted for on first login.

To obtain an attribute from the identity provider, select “Manage in identity provider” for the attribute and specify the name of the SAML attribute that will provide the value. Attribute changes made at the SAML identity provider will only take affect after either a user initiated logout, administrator force logout, or session expiry.

Locally managed attributes can be updated for a user by an administrator.

Roles

User role membership can be defined by the identity provider or managed locally.

To use identity provider specified role membership, select “Defined by identity provider” in the “Role membership” section. The name of the SAML attribute that defines the roles for a user must be specified, along with the values to map to Gradle Enterprise access roles.

When using “Defined by Gradle Enterprise” as the “Role membership” option, the default roles for every user can be specified. Users will be assigned the roles when they first sign in. Changing the default roles will not change role membership of any existing users. Administrators can change role membership for individual users after they have signed in for the first time.

LDAP

An LDAP identity provider can be configured to allow users to access Gradle Enterprise using their organization credentials. User accounts for users authenticating with the LDAP identity provider will be created on first login.

A user cannot login via an LDAP provider if a locally defined account exists for the same username or email.

Setup
  1. As an administrator, navigate to “Access control” → “Identity provider”

  2. Check ”Enable external identity provider”

  3. Choose “LDAP” from Identity provider type options

  4. Complete the connection details for your LDAP provider.

  5. Configure user attributes (described below)

  6. Configure role management (described below)

  7. Click “Save”.

Attribute mappings

There are five required fields in Gradle Enterprise: “username”, ”first name”, “last name”, “email“, and “UUID”. These fields are required to be mapped to fields in your LDAP identity provider.

Commonly, the “UUID” attribute is “entryUUID”, however for LDAP providers where this is missing, another sensible unique persistent ID should be used. For Active Directory, this is usually called “objectGUID”.

Roles

User role membership can be defined by the identity provider or managed locally.

To use identity provider specified role membership, select “Defined by identity provider” in the “Role membership” section. Details on where to find roles and how they are defined must be provided.

For most LDAP providers it is common for the role object class to be “groupOfNames”, however for Active Directory this is usually “group”.

When using “Defined by Gradle Enterprise” as the “Role membership” option, the default roles for every user can be specified. Users will be assigned the roles when they first sign in. Changing the default roles will not change role membership of any existing users. Administrators can change role membership for individual users after they have signed in for the first time.

Disk space management

Typically, disk usage is dominated by the storage of build scan data. How much space is used is dependent on how many build scans are being published and how much information is being recorded for each build. Build scan data is stored using compression and deduplication techniques that make data growth non-linear; storing data for twice as many builds does not mean that twice the space will be required.

Gradle Enterprise provides complimentary options for removing build scans based on their age and automatically deleting old build scans when the amount of free space drops below a specified percentage. Additionally, the system can be configured to send warning emails when free space drops below a specified percentage and/or reject incoming data when free space drops below a specified percentage.

These settings can be changed by going to “Administration” via the top right hand user menu, then “Build scans”.

A recommended configuration that maintains a predictable build scan retention period is:

  1. Specify a maximum build scan age

  2. Send a warning email when there is less than 10% of space free

  3. Reject incoming data when there is less than 5% of space free

An alternative configuration that stores as much build scan history as space permits is:

  1. Do not specify a maximum build scan age

  2. Automatically delete old build scans when there is less than 15% of space free

  3. Send a warning email when there is less than 10% of space free

  4. Reject incoming data when there is less than 5% of space free

When enabling automatic deletion of old build scans when disk is low, be mindful that the result of another process filling the volume that Gradle Enterprise is using will be that all build scan data will be deleted.

If backups are created on the same volume, make sure to leave enough room for them in your thresholds. For example, if the total space that your backups take up is 40% of the disk, the above recommended settings would be 55%, 50% and 45%. It is recommended to store backups on a separate volume to simplify space management.

Backup and disaster recovery

Gradle Enterprise provides system backup and restore capabilities to facilitate disaster recovery. Backups can be triggered manually or scheduled to be done automatically on a periodic basis, and require no system downtime.

The backup process produces a single compressed archive which can be used to restore all user data to a state at the time of the backup. It is highly recommended that this archive additionally be copied to an off-host location, in the case of a complete loss of the host system or volume used to store backup archives.

In addition to disaster recovery, backup archives can be used to migrate your Gradle Enterprise data to a new host system. This can be useful in scenarios where a trial instance is promoted to a production one.

Creating backups

The backup schedule and settings can be configured by going to “Administration” via the top right hand user menu, then “Backups”.

An administrator can be notified of the backup process outcome, which is particularly important in the case of failures. This requires administrator email details and SMTP server settings to be configured for the Gradle Enterprise instance.

In addition to configuring automatic scheduled backups, you may trigger a manual backup at any time by running the following command from the installation system.

./installer/scripts/create_backup.sh

You can download your newly created backup locally by running:

./installer/scripts/copy_backup.sh

The backup archive will be copied to the folder from which the script is executed. If multiple backups exist you will be prompted to select the backup you would like to copy.

Restoring from a backup

Once a backup archive is created you may use it to restore a Gradle Enterprise instance to the backup state at any time.

./installer/scripts/restore_backup.sh /path/to/my_backup.zip

This script will require some downtime on Gradle Enterprise until the restore has finished.

Email notifications

Email server settings can be configured by going to “Administration” via the top right hand user menu, then “Email server”.

With an email server specified, Gradle Enterprise can send email notifications on completion of backups or when disk space is low. See the Disk space management and Creating backups sections for more details.

Upgrading

Before upgrading, be sure to check the Upgrade notes section for any special considerations when upgrading from older versions of Gradle Enterprise.

To upgrade Gradle Enterprise follow the same procedure as for an initial installation. However, there are some notes to consider:

  • To avoid the need to reenter all configuration values, use the same install directory used for the previous installation/upgrade.

  • You will be able to update any configuration values at upgrade time, but keep in mind that some values are immutable after initial installation, such as persistent volume sizes for provisioners that do not support dynamic resizing.

Uninstalling

Gradle Enterprise can be uninstalled by running the following command from the installation system.

./installer/scripts/uninstall.sh
For storage classes with a 'delete' reclaim policy this will mean that all data will also be lost.

Troubleshooting

Support

If you are experiencing issues with Gradle Enterprise, the Gradle build scan plugin, or the Gradle Enterprise Maven extension please submit a support ticket at support.gradle.com, including details of the issue and an attached support bundle.

Support bundles assist our engineers in troubleshooting your issue by providing the following information about your Gradle Enterprise installation:

  • Server logs

  • System performance metrics

  • Gradle Enterprise database statistics

To generate a support bundle execute the following script created by the installer:

./installer/scripts/support_bundle.sh
Generation of the support bundle may take several minutes to complete.

If you have questions about a particular build scan, please submit a support ticket at support.gradle.com, and attach a dump of the build scan.

To generate one, replace /s/ with /scan-dump/ in the build scan url, e.g. https://your-ge-server/scan-dump/vbdei7xhyojq2

Getting started

Learn more about Build Scans, Build Cache, and Test Distribution here.

Appendix A: Upgrade notes

Temporarily degraded performance due to data reindexing

Upon upgrading, a data reindexing process will be initiated in the background with Gradle Enterprise being usable for its duration. CPU usage will be increased and performance may be slightly degraded. For large installations storing many build scans, the reindexing process may take several hours. During this time, some builds may be omitted from cross-build data visualizations. Recent builds are prioritized, making their data available sooner.

Kubernetes workload resources change

In the Kubernetes based installation, some Deployment workload resources have been migrated to StatefulSets. The ./installer/scripts/install.sh script will delete the stale Kubernetes resources when upgrading. If your deployment mechanism doesn’t use that install script, you will need to delete the following resources: deploy/gradle-admin, pvc/gradle-admin-logs-volume, deploy/gradle-server, pvc/gradle-server-logs-volume, deploy/gradle-keycloak, pvc/gradle-keycloak-logs-volume, deploy/gradle-proxy, pvc/gradle-proxy-logs-volume.

Appendix B: Gradle Enterprise Admin CLI JAR downloads

Appendix C: Gradle Enterprise config schema downloads