chore(deps): update prometheus-stack (prometheus 3.0.1, operator 0.79.2)

[renovate]

Note: This PR was initially generated by Renovate but due to changes needed in the operator/other places to support the latest Prometheus version this PR description has been scrubbed of the original contents.

Description

This PR contains a number of application updates to the prometheus stack:

• Prometheus: 2.54.1 -> 3.0.1
• Prometheus Operator: 0.77.1 -> 0.79.2
• Prometheus Config Reloader: 0.77.1 -> 0.79.2
• Kube State Metrics: 2.13.0 -> 2.14.0

Two helm chart updates are also included for the prometheus stack:

• Kube-Prometheus-Stack: 65.3.1 -> 67.3.1
• Prometheus-Operator-CRDs: 15.0.0 -> 17.0.2

In addition to provide a seamless upgrade experience with the new version of Prometheus 3.0 (specifically the change in behavior around scrape protocol), the following changes were made:

• Addition of support for setting fallbackScrapeProtocol in the monitor section of the Package CR spec
• New mutation added to PodMonitor and ServiceMonitor code to set the fallbackScrapeProtocol to PrometheusText004 by default (this matches the pre-3.x behavior in Prometheus)
• "One-time" onDeploy.after task to annotate all existing pod and service monitors: This forces all monitors to go through the mutation cycle again and have the fallbackScrapeProtocol added immediately, without operator intervention. Note that other options were considered (i.e. doing this via Pepr) but this was seen as the easiest and most lightweight solution.
• In order to support the above changes in the operator code the types for Prometheus CRDs were regenerated

Also note that since this update includes a major version of Prometheus there are additional notes in the upstream migration guide. In testing with UDS Core and several apps on top of core no immediate issues were identified with this upgrade.

Related Issue

N/A

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Steps to Validate

The issue with metrics and scrape protocol can be seen with GitLab. To validate the proper behavior across upgrades I ran through the below steps:

# Using Colima the below VM was a good size to handle the GitLab workloads
colima start --cpu 12 --memory 20 -t vz --disk 100

# Deploy part of core FROM `main`
uds run test-single-layer --set LAYER=monitoring --set FLAVOR=unicorn

# Deploy gitlab test bundle
uds deploy ghcr.io/defenseunicorns/packages/uds/bundles/gitlab-test:17.6.2-uds.0 --confirm

# Check metrics status using the e2e test (this is with main so all should pass, just here as a sanity check)
uds run -f src/prometheus-stack/tasks.yaml e2e-test

# Deploy core base, then monitoring FROM `renovate/prometheus-stack` branch
uds run test:single-layer --set LAYER=base --set FLAVOR=unicorn
uds run test:single-layer --set LAYER=monitoring --set FLAVOR=unicorn 
# Note that the tests may fail here due to timing, Prometheus takes upwards of 1 min to read in the new config, re-run them after a short wait
uds run -f src/prometheus-stack/tasks.yaml e2e-test
# Expected behavior here is that all metrics continue to work, simply by deploying latest core, despite upstream issues with GitLab's content type on some metrics

# Check metrics status (manual version of the e2e test) if desired
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring
# Navigate to http://localhost:9090/targets?pool= and validate all show as up/no targets

To more generally validate core:

# Test a complete install
uds run test-uds-core --set FLAVOR=unicorn
# Run the e2e tests for Prom
uds run -f src/prometheus-stack/tasks.yaml e2e-test

# Test a complete upgrade (e2e tests will run as part of this)
uds run test-uds-core-upgrade --set FLAVOR=unicorn

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide followed

[mjnagel]

Looking for 0.79.1 here to support the fallbackScrapeProtocol.

[renovate]

Edited/Blocked Notification

Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.

You can manually request rebase by checking the rebase/retry box above.

⚠️ Warning: custom changes will be lost.

[mjnagel]

Operator 0.79.2 has a fix we need for some of our endpoints, so waiting on the cgr and ib images for that.

This should be fully functional now for the upstream flavor. The below is how I validate the metrics changes in the operator code:

# Note that these steps deploy GitLab which can be resource intensive. This worked for me with a Colima VM setup with 12 cpu / 20gb memory
colima start --cpu 12 --memory 20 -t vz --disk 100

# Deploy core monitoring
uds run test-single-layer --set LAYER=monitoring --set FLAVOR=upstream

# Deploy gitlab test bundle
uds deploy ghcr.io/defenseunicorns/packages/uds/bundles/gitlab-test:17.6.2-uds.0 --confirm

# Portforward prometheus
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

# Open http://localhost:9090/targets and validate that all targets show as up (or unknown/no active targets)