Disaster Recovery

This section provides instructions for restoring Develocity to a previous state using a database backup.

Restoring is a significant operational event that requires system downtime and should be performed with caution.

A restore operation overwrites current data. Before proceeding, ensure you have identified the correct backup archive and understand that any data generated between the backup time and the present will be lost.

Restoring an Embedded Database

If you are using the embedded database, you use the develocityctl CLI to perform a restore. This process involves scaling down the application pods, extracting the data, and restarting the services.

Standalone Installations

Identify the Backup: Locate the .zip backup archive in /opt/gradle/backups (or your custom off-host storage location).
Execute the Restore: Run the following command, replacing the placeholder with your specific filename:
```
develocityctl backup restore /opt/gradle/backups/backup-<DATE-TIME>.zip
```

Kubernetes Installations

Identify the Backup: If you have previously copied backups off-cluster, use one of those. If you need to copy a backup from the cluster, use develocityctl backup copy:
```
develocityctl backup copy ./backup.zip
```
Execute the Restore: Run the following command with the path to the backup archive:
```
develocityctl backup restore ./backup.zip
```

Monitoring the Restore

The CLI will scale down the monitoring, proxy, and operator deployments before restoring the data.

Found Develocity installed in the 'develocity' namespace.
Copying backup
Restarting
deployments.apps/gradle-monitoring scaled
deployments.apps/gradle-proxy scaled
deployments.apps/gradle-enterprise-operator scaled
...
Done. Backup will be restored.

If horizontal scaling is enabled, the restore command will reset the replica counts to the values currently defined in your configuration.

Restoring a User-Managed Database

For external databases (PostgreSQL, RDS, etc.), Develocity doesn’t manage the restore process directly. You must use your database provider’s native tools.

Scenario A: In-Place Restore

Use this if you are restoring data to the same database instance.

Stop Develocity: To prevent data corruption, stop the application services.
```
develocityctl system stop
```
Restore Data: Follow your provider’s instructions (e.g., pg_restore or a cloud snapshot restoration).
Verify Permissions: If you aren’t providing superuser credentials to Develocity, re-run your database setup scripts to ensure the application and migrator user permissions are correct.
Start Develocity:
```
develocityctl system start
```

Scenario B: Restore to a New Instance

Use this if you are moving data to a different database server.

Stop the Application: Ensure the old instance is no longer receiving traffic via develocityctl system stop.
Restore to New Host: Follow provider instructions to hydrate the new instance with your backup data.
Update Configuration: Configure Develocity with the new instance connection details and superuser credentials if those are being provided. This means re-running Helm, or updating the specified ConfigMap and Secret resources.
Verify Permissions: If you aren’t providing superuser credentials to Develocity, run the current database setup script used during database setup (see your installation manual) on the database and ensure that working application and migrator passwords are configured in the configuration interface or Secret resources being used.
Restart Develocity: Use develocityctl system start to initiate the connection.

Troubleshooting Restoration Failures

If the system fails to start after a restore:

Check Logs: Review the application logs for connection strings or permission errors.

kubectl --namespace=«your-namespace» logs --selector=app.kubernetes.io/component=enterprise-app

Verify Network: If you migrated to a new database instance, verify that the Develocity pods can reach the new database host and port. You can test connectivity from within the cluster by running a temporary pod:
```
kubectl --namespace=«your-namespace» run --rm -it pg-test --image=busybox --restart=Never -- nc -zv «db-host» «db-port»
```
Generate Support Bundle: If the issue persists, generate a diagnostic bundle to share with Gradle Support.
```
develocityctl system support-bundle
```