Skip to content

fix(selfhost): wire alerting to reach an operator by default + close dead-letter blind spot #2533

Description

@JSONbored

Parent: #1936

Problem

Self-host ships a fairly complete Prometheus alert-rule set (target-down, queue backlog, dead-letter, Postgres health, backups, webhook failures, 5xx ratio, latency SLO), but the default Alertmanager routing sends every alert to a name-only null receiver, and every real receiver (Slack/Discord/email/webhook) ships commented-out as an example. Confirmed against a live production deployment, not just the repo template: a fresh docker compose --profile observability up -d produces zero notifications for any alert, of any severity, until an operator edits the config by hand and happens to notice the comment block.

Separately, the dead-letter alert pair has a coverage gap: the "growing" rule is a 15-minute rate window (increase(gittensory_jobs_dead_total[15m]) > 0), so it only pages during a narrow window around the transition into dead; the "backlog" rule only fires above 50 dead jobs. A single dead job — which does happen, e.g. from a one-off backend-compatibility bug during a Postgres migration — sits invisible in the queue table forever once the 15-minute window closes, with no way for an operator to know without querying the database directly.

Requirements

  • Ship a safe, working default notification path (or, if a truly generic default receiver isn't practical, make the "you must configure a receiver" step impossible to silently skip — e.g. a startup warning log, a /ready degraded flag, or a first-boot doc callout that's hard to miss).
  • Add a sustained/level dead-letter alert that fires on ANY non-zero dead-letter count persisting beyond a short grace window, independent of the existing rate-based alert.
  • Keep the existing rate-based and backlog-threshold alerts; this is additive, not a replacement.

Deliverables

  • Alertmanager default routing that either notifies somewhere real out of the box, or clearly and loudly signals "unconfigured" rather than silently discarding.
  • A new GittensoryDeadLetterBacklogPersisting-style rule (or equivalent) in prometheus/rules/alerts.yml.
  • Updated self-hosting operations docs describing the notification-setup step as a required (not optional-looking) part of first deployment.

Acceptance criteria

  • A fresh deployment either notifies an operator on a firing alert, or produces an unmistakable, hard-to-miss signal that no receiver is configured.
  • A single dead-lettered job that has not been retried/cleared within a bounded window (e.g. 1 hour) triggers an alert, regardless of when it entered the dead state relative to the 15-minute rate window.
  • No regression to existing alert rules or their thresholds.

Expected outcome

An operator running this stack finds out about a real failure from an alert, not from a maintainer manually inspecting the database weeks later.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions