Repository navigation
🌱 Add Stellar/GPU/notification observability to dashboard and alerts - #23374
Conversation
Extend the existing recommendation-only Grafana dashboard and opt-in PrometheusRule with panels/alerts for metrics that already exist in pkg/api/metrics but had no dashboard or alerting coverage: - console_stellar_* (stale-approval sweep, daily-digest, scheduler dispatch cycle rates; action outcomes and execution duration) - console_gpu_util_* (scrape cycles, DCGM/alert-send errors, reservation collection outcomes) - console_notification_* (delivery rate/latency by channel_type and outcome) New alerts (opt-in via metrics.prometheusRule.enabled, thresholds configurable via values.yaml, no magic numbers): - ConsoleStellarSchedulerStalled: dispatch cycle counter stopped incrementing - ConsoleNotificationDeliveryFailing: failure ratio over threshold - ConsoleGPUUtilDCGMScrapeErrors: DCGM scrape errors observed No new exporter, backend, or data flow — this only extends an existing, already-scoped self-metrics dashboard/alert set. Signed-off-by: kubestellar-hive[bot] <223556219+Copilot@users.noreply.github.com>
|
Important Held for human sign-off on the direction, not on the code. This PR's only tracked rationale is #23373, which the hive filed itself — issue #23373 was filed by kubestellar-hive[bot] and no human has acknowledged it. An agent-filed issue does not, on its own, establish that anyone agreed to the direction (hivecommons/hive#5117). The change may well be right; nothing here is a review of it. To release the hold, acknowledge the direction on that issue — comment on it, assign yourself, or add the |
kubestellar-prow
Bot
added
the
dco-signoff: yes
✅ Deploy Preview for kubestellarconsole canceled.
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!
This is an automated message. |
kubestellar-prow
Bot
added
the
size/L
|
hold released: self-authorization hold disabled by config (#5117) |
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
|
Post-merge build verification passed ✅ Both Go and frontend builds compiled successfully against merge commit |
console-self-metrics-dashboard.json covered 15 of 20 registered console_* self-metrics after #23374. 5 metrics still had no panel: console_stellar_stale_approval_sweep_duration_seconds, console_stellar_stale_approval_actions_reviewed_total, console_stellar_daily_digest_sent_total, console_stellar_scheduler_actions_picked_up_total, and console_gpu_util_scrape_duration_seconds. Adds 5 panels (ids 14-18) covering these, following the existing panel style. No new metric, exporter, or external data flow — this only visualizes metrics the backend already registers with fixed, bounded label sets. Verified all referenced metric names are in the pkg/api/metrics/metrics.go allow-list (scripts/lint-dashboards.sh metric-reference check) and that the JSON is well-formed. Signed-off-by: kubestellar-hive[bot] <223556219+Copilot@users.noreply.github.com> Co-authored-by: kubestellar-hive[bot] <223556219+Copilot@users.noreply.github.com>
Fixes #23373
Extends the existing recommendation-only Grafana dashboard (
deploy/grafana/console-self-metrics-dashboard.json) and opt-in PrometheusRule (deploy/helm/kubestellar-console/templates/prometheusrule.yaml) with panels/alerts for metrics that already exist inpkg/api/metricsbut had no dashboard or alerting coverage.Added dashboard panels
Added alerts (opt-in via
metrics.prometheusRule.enabled)ConsoleStellarSchedulerStalled—console_stellar_scheduler_dispatch_cycles_totalcounter stopped incrementingConsoleNotificationDeliveryFailing— failure ratio ofconsole_notification_sends_totalover a configurable thresholdConsoleGPUUtilDCGMScrapeErrors—console_gpu_util_dcgm_scrape_errors_totalincreases observedAll new thresholds/durations are parameterized in
values.yaml(no magic numbers). Labels used (outcome,channel_type) are bounded enums already emitted by existing instrumentation — no new unbounded labels.Verification
pkg/api/metrics/metrics.go/metrics_test.go.console-self-metrics-dashboard.jsonas valid JSON.Build/lint are validated by CI on this PR.
— hive: agent=telemetry backend=copilot model=claude-sonnet-4-6