Monitoring

You can monitor the health and performance of your Develocity Provenance Governor deployment using built-in monitoring endpoints. These are accessible on a dedicated port (9090) and protected by basic authentication by default. Separation from the main application port improves security for production traffic.

Available Endpoints

Endpoint

Description

/actuator/health

Overall health status

/actuator/health/liveness

Kubernetes liveness probe

/actuator/health/readiness

Kubernetes readiness probe

/actuator/prometheus

Prometheus scrapeable metrics

/actuator/info

General application info

Accessing Endpoints

Use port-forwarding for local machine access, or direct service URLs for in-cluster.

kubectl port-forward svc/api-http-monitoring 9090:9090 -n develocity-provenance-governor

Security & Configuration

All monitoring endpoints on port 9090 require basic authentication; see application config for username/password management. For load balancer/liveness, unauthenticated /livez and /readyz are offered on app port (8080).

Prometheus Metrics

The /actuator/prometheus endpoint exposes metrics in Prometheus format, which can be scraped by a Prometheus server for monitoring and alerting.

Available Metrics

The Prometheus endpoint (/actuator/prometheus) provides metrics including:

  • JVM Metrics: Memory usage, garbage collection, thread counts

  • HTTP Metrics: Request rates, response times, error rates

  • Application Metrics: Custom business metrics for attestation publishing and policy evaluation

  • System Metrics: CPU and system resource utilization

Enabling Histogram Percentiles

By default, only basic timing metrics (_sum, _count, _max) are collected. To enable percentile calculations (p50, p95, p99), configure histogram buckets in your application properties:

# Enable histogram buckets for HTTP server requests
management.metrics.distribution.percentiles-histogram.http.server.requests=true

# Enable histogram buckets for HTTP client requests (upstream calls)
management.metrics.distribution.percentiles-histogram.http.client.requests=true

# Optionally, define specific percentiles to publish directly
management.metrics.distribution.percentiles.http.server.requests=0.5,0.95,0.99
management.metrics.distribution.percentiles.http.client.requests=0.5,0.95,0.99

# Define SLO buckets for alerting
management.metrics.distribution.slo.http.server.requests=100ms,250ms,500ms,1s,5s

Histogram buckets increase metric cardinality and storage requirements. Each histogram creates multiple time series (one per bucket boundary). Enable only for endpoints where percentile analysis is needed, or accept the additional storage cost for comprehensive monitoring.

With histogram buckets enabled, you can use histogram_quantile() in Prometheus to calculate arbitrary percentiles without pre-defining them at collection time.

Prometheus Query Examples

The following queries help monitor the health and performance of Develocity Provenance Governor.

Request Rate

Monitor the rate of incoming HTTP requests per second:

sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor"}[5m])) by (uri, status)

Filter to specific endpoints:

sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor", uri=~"/packages.*"}[5m])) by (uri, status)

Success Rate

Calculate the percentage of successful requests (2xx responses):

sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor", status=~"2.."}[5m]))
/
sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor"}[5m]))
* 100

Error Rate

Track client errors (4xx) and server errors (5xx):

# Client errors (4xx)
sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor", status=~"4.."}[5m])) by (uri)

# Server errors (5xx)
sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor", status=~"5.."}[5m])) by (uri)

Request Latency

Average latency per endpoint:

sum(rate(http_server_requests_seconds_sum{namespace="develocity-provenance-governor"}[5m])) by (uri)
/
sum(rate(http_server_requests_seconds_count{namespace="develocity-provenance-governor"}[5m])) by (uri)

Maximum latency (note: this value resets periodically):

http_server_requests_seconds_max{namespace="develocity-provenance-governor"}

Percentile latency (requires histogram configuration):

# 95th percentile (p95)
histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket{namespace="develocity-provenance-governor"}[5m])) by (le, uri))

# 99th percentile (p99)
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{namespace="develocity-provenance-governor"}[5m])) by (le, uri))

Upstream Service Latency

Monitor latency to external services (Develocity, Artifactory):

# Average latency to Develocity
sum(rate(http_client_requests_seconds_sum{namespace="develocity-provenance-governor", client_name="develocity.example.com"}[5m]))
/
sum(rate(http_client_requests_seconds_count{namespace="develocity-provenance-governor", client_name="develocity.example.com"}[5m]))

JVM Metrics

Monitor JVM health:

# Heap memory usage
jvm_memory_used_bytes{namespace="develocity-provenance-governor", area="heap"}

# Live thread count
jvm_threads_live_threads{namespace="develocity-provenance-governor"}

# CPU usage (0-1 ratio)
process_cpu_usage{namespace="develocity-provenance-governor"}