Troubleshooting

Operations

Updating Develocity Provenance Governor

To update to a new version:

  1. Review the release notes for breaking changes and new features.

  2. Update the container image in your deployment:

    # Edit the deployment to use the new image version
kubectl set image deployment/api api-http=registry.gradle.com/develocity/provenance-governor:NEW_VERSION@sha256:NEW_DIGEST \
  -n develocity-provenance-governor

# Watch the rollout
kubectl rollout status deployment/api -n develocity-provenance-governor

  3. Verify the update:

    # Check pod is running with new image
kubectl get pods -n develocity-provenance-governor -o jsonpath='{.items[0].spec.containers[0].image}'

# Check logs for successful startup
kubectl logs deployment/api -n develocity-provenance-governor --tail=50

Before updating production, test the new version in a non-production environment with your existing configurations.

Rolling Back to a Previous Version

If you encounter issues after an update:

  1. Rollback the deployment:

    kubectl rollout undo deployment/api -n develocity-provenance-governor

  2. Verify rollback succeeded:

    kubectl rollout status deployment/api -n develocity-provenance-governor
kubectl logs deployment/api -n develocity-provenance-governor --tail=50

  3. Check rollout history:

    kubectl rollout history deployment/api -n develocity-provenance-governor

Updating Configuration Without Downtime

Configuration changes (ConfigMaps, Secrets) require a pod restart to take effect:

  1. Update the ConfigMap or Secret:

    # For policies
kubectl create configmap policies \
  --from-file=policies/ \
  --dry-run=client -o yaml | kubectl apply -f -

# For application properties
kubectl create configmap properties \
  --from-file=application.properties \
  --dry-run=client -o yaml | kubectl apply -f -

# For secrets
kubectl create secret generic secrets \
  --from-file=secrets/ \
  --dry-run=client -o yaml | kubectl apply -f -

  2. Restart the deployment:

    kubectl rollout restart deployment/api -n develocity-provenance-governor
kubectl rollout status deployment/api -n develocity-provenance-governor

  3. Verify configuration loaded:

    kubectl logs deployment/api -n develocity-provenance-governor | grep "support enabled"

Managing Policies

Adding a new policy:

  1. Add the policy YAML to your policies directory.

  2. Update the policies ConfigMap.

  3. Restart the deployment.

  4. Verify the policy is loaded in the logs.

Disabling a policy temporarily:

Remove the labels that PolicyScanDefinitions use to select it. The policy will remain in the ConfigMap but won’t be evaluated.

Testing policy changes:

  1. Create a new PolicyScanDefinition with a unique name for testing.

  2. Use labels to select only the new/modified policies.

  3. Test against known packages.

  4. Once verified, update your production PolicyScanDefinition.

Backup and Restore

Backup your configuration:

# Export all configurations
kubectl get configmap policies -n develocity-provenance-governor -o yaml > policies-backup.yaml
kubectl get configmap properties -n develocity-provenance-governor -o yaml > properties-backup.yaml
kubectl get secret secrets -n develocity-provenance-governor -o yaml > secrets-backup.yaml
kubectl get secret license -n develocity-provenance-governor -o yaml > license-backup.yaml

Restore from backup:

kubectl apply -f policies-backup.yaml
kubectl apply -f properties-backup.yaml
kubectl apply -f secrets-backup.yaml
kubectl apply -f license-backup.yaml
kubectl rollout restart deployment/api -n develocity-provenance-governor

Monitoring HTTP Client Requests

Develocity Provenance Governor makes outbound HTTP requests to several external services as part of its normal operation:

  • Develocity instances - To retrieve build scan data and attestation information

  • Artifactory instances - To publish and retrieve attestations

  • OSV (Open Source Vulnerabilities) APIs - To query vulnerability information

  • External package repositories - To verify package metadata

Understanding these HTTP client request patterns is critical for operators to:

  • Monitor system health and performance

  • Identify integration issues with external services

  • Track API usage and rate limits

  • Troubleshoot connectivity problems

  • Optimize performance through caching strategies

HTTP Client Metrics

Develocity Provenance Governor exposes HTTP client metrics through the Spring Boot Actuator endpoint at http://provenance-governor.example.com:9090/actuator/prometheus.

The actuator endpoints are exposed on port 9090 and are not accessible through the public ingress by default. Configure your monitoring system to scrape metrics directly from the pod or service on port 9090.

The primary metric for monitoring HTTP client requests is:

  • Metric name: http_client_requests_seconds_*

  • Type: Timer (tracks both count and duration)

  • Availability: Available when Spring Boot Actuator and Micrometer are enabled (default configuration)

Metric Tags

Each HTTP client request metric includes the following tags to help identify and filter requests:

Tag Description

uri

The URI template pattern for the request (e.g., /api/builds/{id}, /evidence/{artifact-path}). Variables are replaced with {placeholder} to prevent cardinality explosion.

method

The HTTP method used (GET, POST, PUT, DELETE, etc.)

status

The HTTP response status code (200, 404, 500, etc.)

client.name

The target hostname of the request (e.g., develocity.example.com, artifactory.example.com)

outcome

The outcome of the request: SUCCESS (2xx), CLIENT_ERROR (4xx), SERVER_ERROR (5xx), or UNKNOWN

develocity_instance

The Develocity instance name (e.g., prod, staging). Present only on Develocity API requests. The prefix dv: is stripped from the configured instance key.

artifactory_instance

The Artifactory instance name (e.g., prod, staging). Present only on Artifactory API requests. The prefix af: is stripped from the configured instance key.

Common URI Template Patterns

Operators can expect to see the following URI patterns in their metrics:

Develocity API patterns:

  • /build-export/v2/build/{id}/events - Streaming build events for attestation generation

  • /api/builds/{id} - Retrieving individual build scan metadata

  • /api/builds - Querying build scans by criteria

Artifactory patterns:

  • /evidence/{artifact-path} - Publishing attestations for artifacts

  • /artifact/{download-path} - Retrieving attestations or artifact metadata

External API patterns (OSV, package registries):

  • /v3/systems/{packageType}/packages/{packageKeyName} - OSV package vulnerability queries

  • /v3/systems/{packageType}/packages/{packageKeyName}/versions/{version} - Version-specific vulnerability data

  • /v1/query - OSV batch vulnerability queries

  • /v1/vulns/{id} - Individual vulnerability details

Enabling Histogram Metrics

By default, Spring Boot exports only count and sum metrics for HTTP requests. To enable percentile histograms (for calculating p50, p95, p99 latencies in your monitoring system), add the following properties to your application.properties configuration:

management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.percentiles-histogram.http.client.requests=true

Add these properties to the properties ConfigMap:

kubectl edit configmap properties -n develocity-provenance-governor

Update the management.metrics.distribution.percentiles-histogram.http.* keys:

# Histogram Metrics Configuration
management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.percentiles-histogram.http.client.requests=true

After adding the configuration, restart the deployment:

kubectl rollout restart deployment/api -n develocity-provenance-governor
kubectl rollout status deployment/api -n develocity-provenance-governor

For more information on histogram configuration, see:

Example Prometheus Queries

Use these example queries in your Prometheus or Grafana dashboards:

Success rate for Develocity build API:

rate(http_client_requests_seconds_count{uri="/api/builds/\{id\}",outcome="SUCCESS"}[5m])

Average latency for attestation publishing:

rate(http_client_requests_seconds_sum{uri="/evidence/\{artifact-path\}"}[5m])
/ rate(http_client_requests_seconds_count{uri="/evidence/\{artifact-path\}"}[5m])

Error rate by endpoint:

sum by (uri) (rate(http_client_requests_seconds_count{outcome!="SUCCESS"}[5m]))

Request count by target service:

sum by (client_name) (rate(http_client_requests_seconds_count[5m]))

Request count by Develocity instance:

sum by (develocity_instance) (rate(http_client_requests_seconds_count{develocity_instance!=""}[5m]))

Compare error rates across Artifactory instances:

sum by (artifactory_instance) (rate(http_client_requests_seconds_count{artifactory_instance!="", outcome!="SUCCESS"}[5m]))
/
sum by (artifactory_instance) (rate(http_client_requests_seconds_count{artifactory_instance!=""}[5m]))

95th percentile latency for all external requests:

histogram_quantile(0.95,
  sum by (le, uri) (rate(http_client_requests_seconds_bucket[5m]))
)

The last query (95th percentile) requires histogram metrics to be enabled as described in the previous section.

Grafana Dashboard Recommendations

Consider creating the following visualizations in your Grafana dashboards:

Request Volume Panel:

  • Visualization: Time series graph

  • Query: Total requests per second grouped by client.name

  • Purpose: Identify which external services are being called most frequently

Error Rate Panel:

  • Visualization: Time series graph

  • Query: Error rate percentage by uri and status

  • Purpose: Quickly identify failing endpoints or integration issues

Latency Heatmap:

  • Visualization: Heatmap

  • Query: Request duration distribution by uri

  • Purpose: Identify slow endpoints that may need optimization or caching

Status Code Distribution:

  • Visualization: Pie chart or bar chart

  • Query: Request count by status code

  • Purpose: Understand the distribution of successful vs. failed requests

Top Slowest Endpoints:

  • Visualization: Table

  • Query: Average and p95 latency grouped by uri

  • Purpose: Prioritize performance optimization efforts

Troubleshooting with Metrics

High error rates to Develocity instances:

  • Check the http_client_requests_seconds_count metric filtered by client.name=develocity.example.com and outcome="CLIENT_ERROR"

  • Use develocity_instance tag to identify which specific instance is affected when multiple instances are configured

  • Common causes: Invalid access key, expired credentials, network connectivity issues

  • Resolution: Verify access key in the secrets Secret and check network policies

Isolating issues to specific instances:

When multiple Develocity or Artifactory instances are configured, use the instance tags to identify which specific instance is experiencing problems:

# Find which Develocity instance has the highest error rate
topk(5,
  sum by (develocity_instance) (rate(http_client_requests_seconds_count{develocity_instance!="", outcome!="SUCCESS"}[5m]))
)

Slow attestation publishing:

  • Check latency metrics for requests to /evidence/{artifact-path} endpoint

  • Common causes: Network latency to Artifactory, large attestation payloads, rate limiting

  • Resolution: Consider adjusting timeouts, enabling connection pooling, or optimizing attestation size

Unexpected 404 responses:

  • Filter metrics by status="404" to identify which endpoints are returning not found errors

  • Common causes: Incorrect API paths, missing resources, configuration errors

  • Resolution: Verify instance URIs in the properties ConfigMap and ensure resources exist

Common Issues

The following are some possible common issues, how they are recognized, and what might be needed to resolve them:

Invalid or missing Develocity license

The startup error clearly describes the action needed:

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to bind properties under 'develocity' to com.gradle.cavendish.dv.DevelocityProperties:

    Reason: java.lang.IllegalArgumentException: No develocity.license provided in configuration.

Action:

Update your application's configuration

Ensure that the Develocity Provenance Governor deployment is configured with a valid Develocity license.

Missing Configuration properties

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to bind properties under 'develocity' to com.gradle.cavendish.dv.DevelocityProperties:

    Reason: java.lang.IllegalArgumentException: Develocity Access Key `develocity.instances.main.access-key` must have text

Action:

Update your application's configuration

Double check to see that the property mentioned is spelled correctly and in the configuration.

Access control issues (403 or 401 responses)

Access control issues can be troublesome to track down as the system errors on the side of keeping sensitive information secure.

Follow these troubleshooting steps:

  1. Verify your authentication is working:

    kubectl logs deployment/api -n develocity-provenance-governor | grep 'Identity support enabled'

  2. Check that access control policies are loaded:

    kubectl logs deployment/api -n develocity-provenance-governor | grep 'AccessControl'

  3. Verify resource matchers include trailing wildcards where needed:

    • Correct: dv:my-instance/*

    • Incorrect: dv:my-instance (missing wildcard)

  4. Start with a permissive policy for testing:

    withResources:
  - "*:*/*"  # Allows access to all resources

    Once authentication is working, tighten the policy to specific resources.

  5. Enable debug logging temporarily to see policy evaluation details (consult support for instructions).

Invalid access keys for external integrations (e.g. Develocity and Artifactory)

2025-10-16T17:51:58.967-06:00 ERROR 70284 --- [ctor-http-nio-2] c.g.c.dv.DevelocityAccessTokenManager    : Error refreshing access tokens

org.springframework.web.reactive.function.client.WebClientResponseException$Unauthorized: 401 Unauthorized from POST https://develocity.grdev.net/api/auth/token
	...

Again, the logging should be fairly clear in this respect. The above, for example, illustrates Develocity Provenance Governor cannot communicate with Develocity. The corrective action here would be to examine the access key in the configuration that is defined for accessing Develocity and make sure it is correct.

Artifactory doesn’t have public key for validating signed attestations

"body": {
  "errors": [
    {
      "message": "not found"
    }
  ]
},
"status": 400,

When a private key signs an attestation with a key that hasn’t had the corresponding public key installed in Artifactory, a 400 "not found" value is returned.

It is worth mentioning this as the error behavior isn’t exactly explicit, presumably for security reasons within Artifactory.

Note that the Provenance Governor logs will show the Public Key ID that will match the Public Key ID in the Artifactory UI. For example:

2025-12-08T14:36:54.180-07:00  INFO 43275 --- [  restartedMain] c.g.d.d.p.SignatureServiceConfiguration  : Signature support enabled, for key pair [name:20250818123202:keyid:6bce0a:signing-algorithm:ed25519]

The Artifactory UI would match:

Artifactory Key ID Example

Policies don’t seem to be used

2025-10-16T17:51:58.098-06:00  INFO 70284 --- [  restartedMain] c.g.c.p.PolicyScanDefinitionEvaluator    : Policy Scan support enabled for [java-21-toolchain] including policy [none-match-oracle-java-toolchain]
2025-10-16T17:51:58.098-06:00  INFO 70284 --- [  restartedMain] c.g.c.p.PolicyScanDefinitionEvaluator    : Policy Scan support enabled for [java-21-toolchain] including policy [must-match-bellsoft-java-toolchain]
2025-10-16T17:51:58.099-06:00  INFO 70284 --- [  restartedMain] c.g.c.p.PolicyScanDefinitionEvaluator    : Policy Scan support enabled for [java-21-toolchain] including policy [purls-none-match-lombok]
2025-10-16T17:51:58.099-06:00  INFO 70284 --- [  restartedMain] c.g.c.p.PolicyScanDefinitionEvaluator    : Policy Scan support enabled for [java-21-toolchain] including policy [none-match-purl-spring-beans-6.2.7]
2025-10-16T17:51:58.100-06:00  INFO 70284 --- [  restartedMain] c.g.c.p.PolicyScanDefinitionEvaluator    : Policy Scan support enabled for [java-21-toolchain] including policy [must-match-repositories]

When evaluating policies, check startup logs to see which policies loaded. Each enabled Policy Scan™ evaluation logs its included policies. If your expected policy is missing:

  1. Check the policy name matches exactly (case-sensitive)

  2. Verify policy labels match the scan’s label selector

  3. Confirm the policy YAML is in the 'policies' ConfigMap:

    kubectl get configmap policies -n develocity-provenance-governor -o yaml

  4. Look for parsing errors in earlier log messages:

    kubectl logs deployment/api -n develocity-provenance-governor | grep -i error

Policy Writing and Debugging

My policy doesn’t match anything

Symptoms: Policy Scan shows PASSED for a none-match policy even though the forbidden item exists, or FAILED for a must-match policy even though the required item exists.

Common Causes:

  1. Incorrect pURL format - Check the exact format of the dependency in attestations:

    # View attestations to see exact pURL format
curl -H "Authorization: Basic ..." \
  "https://provenance-governor.example.com/packages/maven/com.example/my-app/1.0.0/attestations"

  2. Case sensitivity - Package names and vendors are case-sensitive:

    • Wrong: pkg:maven/Org.Example/artifact

    • Right: pkg:maven/org.example/artifact

  3. Version mismatch - If you specify a version in the pURL, it must match exactly:

    • pkg:maven/org.example/artifact@1.0.0 does NOT match 1.0.0-SNAPSHOT

    • Use wildcards if you want to match all versions: pkg:maven/org.example/artifact

  4. Namespace separator - For Maven, use / not .:

    • Wrong: pkg:maven/org.example.subpackage/artifact

    • Right: pkg:maven/org.example/subpackage-artifact

Understanding why a policy failed

To debug policy failures:

  1. Check the Policy Scan results - The failure message includes which attestation values caused the failure:

    {
  "policyName": "must-match-java-21",
  "status": "FAILED",
  "message": "Found Java vendor: Oracle Corporation, version: 17.0.8",
  "remediation": "Update to Java 21"
}

  2. Review attestation data - Compare what the policy expects vs. what’s in the attestations:

    # Get all attestations for the package
curl -H "Authorization: Basic ..." \
  "https://provenance-governor.example.com/packages/maven/com.example/my-app/1.0.0/attestations" | jq .

  3. Check predicate types - Ensure your policy operates on the correct predicate type:

Policy is not being evaluated

Symptoms: Policy exists but doesn’t appear in scan results.

Troubleshooting steps:

  1. Verify policy is loaded:

    kubectl logs deployment/api -n develocity-provenance-governor | grep "Policy Scan support enabled"

  2. Check label selectors match:

    Your policy must have labels that match the PolicyScanDefinition selector:

    Policy:

    metadata:
  name: my-policy
  labels:
    policy.my-corp.com/gate: build

    PolicyScanDefinition:

    spec:
  policy-selector:
    match-labels:
      policy.my-corp.com/gate: build  # Must match!

  3. Verify YAML syntax:

    # Check for YAML parsing errors
kubectl get configmap policies -n develocity-provenance-governor -o yaml

Testing policies before deployment

Before deploying policies to production:

  1. Start with none-match policies - These are easier to debug because they fail when unexpected items are found.

  2. Test against known packages - Run scans against packages you’ve already built and know the contents of.

  3. Use descriptive names - Name policies clearly: block-oracle-jdk is better than java-policy-1.

  4. Add detailed remediation - Include specific steps in the remediation field to help developers fix issues.

  5. Test both pass and fail cases - Verify the policy fails when it should and passes when it should.

Additional Help

If you experience issues with Develocity Provenance Governor or related components, please submit a support ticket at support.gradle.com, including details of the issue and an attached support bundle from Develocity, if applicable.