Parent: #1936
Problem
Self-host ships a fairly complete Prometheus alert-rule set (target-down, queue backlog, dead-letter, Postgres health, backups, webhook failures, 5xx ratio, latency SLO), but the default Alertmanager routing sends every alert to a name-only null receiver, and every real receiver (Slack/Discord/email/webhook) ships commented-out as an example. Confirmed against a live production deployment, not just the repo template: a fresh docker compose --profile observability up -d produces zero notifications for any alert, of any severity, until an operator edits the config by hand and happens to notice the comment block.
Separately, the dead-letter alert pair has a coverage gap: the "growing" rule is a 15-minute rate window (increase(gittensory_jobs_dead_total[15m]) > 0), so it only pages during a narrow window around the transition into dead; the "backlog" rule only fires above 50 dead jobs. A single dead job — which does happen, e.g. from a one-off backend-compatibility bug during a Postgres migration — sits invisible in the queue table forever once the 15-minute window closes, with no way for an operator to know without querying the database directly.
Requirements
- Ship a safe, working default notification path (or, if a truly generic default receiver isn't practical, make the "you must configure a receiver" step impossible to silently skip — e.g. a startup warning log, a
/ready degraded flag, or a first-boot doc callout that's hard to miss).
- Add a sustained/level dead-letter alert that fires on ANY non-zero dead-letter count persisting beyond a short grace window, independent of the existing rate-based alert.
- Keep the existing rate-based and backlog-threshold alerts; this is additive, not a replacement.
Deliverables
- Alertmanager default routing that either notifies somewhere real out of the box, or clearly and loudly signals "unconfigured" rather than silently discarding.
- A new
GittensoryDeadLetterBacklogPersisting-style rule (or equivalent) in prometheus/rules/alerts.yml.
- Updated self-hosting operations docs describing the notification-setup step as a required (not optional-looking) part of first deployment.
Acceptance criteria
- A fresh deployment either notifies an operator on a firing alert, or produces an unmistakable, hard-to-miss signal that no receiver is configured.
- A single dead-lettered job that has not been retried/cleared within a bounded window (e.g. 1 hour) triggers an alert, regardless of when it entered the dead state relative to the 15-minute rate window.
- No regression to existing alert rules or their thresholds.
Expected outcome
An operator running this stack finds out about a real failure from an alert, not from a maintainer manually inspecting the database weeks later.
Parent: #1936
Problem
Self-host ships a fairly complete Prometheus alert-rule set (target-down, queue backlog, dead-letter, Postgres health, backups, webhook failures, 5xx ratio, latency SLO), but the default Alertmanager routing sends every alert to a name-only
nullreceiver, and every real receiver (Slack/Discord/email/webhook) ships commented-out as an example. Confirmed against a live production deployment, not just the repo template: a freshdocker compose --profile observability up -dproduces zero notifications for any alert, of any severity, until an operator edits the config by hand and happens to notice the comment block.Separately, the dead-letter alert pair has a coverage gap: the "growing" rule is a 15-minute rate window (
increase(gittensory_jobs_dead_total[15m]) > 0), so it only pages during a narrow window around the transition intodead; the "backlog" rule only fires above 50 dead jobs. A single dead job — which does happen, e.g. from a one-off backend-compatibility bug during a Postgres migration — sits invisible in the queue table forever once the 15-minute window closes, with no way for an operator to know without querying the database directly.Requirements
/readydegraded flag, or a first-boot doc callout that's hard to miss).Deliverables
GittensoryDeadLetterBacklogPersisting-style rule (or equivalent) inprometheus/rules/alerts.yml.Acceptance criteria
Expected outcome
An operator running this stack finds out about a real failure from an alert, not from a maintainer manually inspecting the database weeks later.