observability: instant_email_failover_total alert + tile + catalog (#35)

k8s/prometheus-rules.yaml

@@ -521,6 +521,52 @@ spec:
               coverage test that iterates eventEmailBuilders so this can't
               happen again for the same kind.
 
+    # Failover-ESP engagement. The worker's FailoverProvider
+    # (worker/internal/email/failover_provider.go) emits
+    # instant_email_failover_total{outcome} on every send when
+    # EMAIL_PROVIDER_FALLBACK is configured. Two alert-able outcomes:
+    #   fallback_ok — primary ESP (Brevo) failed/rejected but the secondary
+    #                 (SES) sent the email. Email is NOT lost, but the primary
+    #                 is degraded and the failover is carrying load → P1.
+    #   all_failed  — every ESP in the chain refused the send. Email IS being
+    #                 lost → P0. This is the failure mode the feature exists to
+    #                 surface (the standing Brevo-sender-unvalidated P0 with no
+    #                 working secondary).
+    # primary_ok is the healthy path and is deliberately NOT alerted.
+    - name: instant-worker-email-failover
+      rules:
+        - alert: EmailFailoverEngaged
+          expr: |
+            sum(rate(instant_email_failover_total{outcome="fallback_ok"}[10m])) > 0
+          for: 10m
+          labels:
+            severity: warning
+            service: worker
+          annotations:
+            summary: "Email failover engaged — primary ESP degraded, secondary carrying load"
+            description: |
+              instant_email_failover_total{outcome="fallback_ok"} rate > 0 for
+              >10m. The primary ESP (Brevo) is failing or rejecting sends and
+              the configured secondary (SES) is delivering them instead. Email
+              is still flowing, but the primary needs attention — check
+              brevo_send_errors_total by classification and the Brevo sender-
+              domain validation status.
+        - alert: EmailFailoverAllFailed
+          expr: |
+            sum(rate(instant_email_failover_total{outcome="all_failed"}[5m])) > 0
+          for: 5m
+          labels:
+            severity: critical
+            service: worker
+          annotations:
+            summary: "Email failover ALL providers failed — email is being LOST"
+            description: |
+              instant_email_failover_total{outcome="all_failed"} rate > 0 for
+              >5m. Every ESP in the failover chain (primary + all secondaries)
+              refused the send — transactional email is being dropped on the
+              floor. Check both BREVO_API_KEY/sender validation and the SES_*
+              credentials + SES sandbox/production-access state immediately.
+
     # /readyz deep-health gauge, exposed by api, worker, and provisioner via
     # readyz_check_status{service,check}. 1=ok, 0.5=degraded, 0=failed. This
     # rule mirrors the NR alert readyz-component-failed.json (which depends

newrelic/alerts/email-failover-all-failed.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-worker — email failover ALL providers failed (email is being LOST)",
+  "type": "NRQL",
+  "description": "P0 page. instant_email_failover_total{outcome=\"all_failed\"} ticks when EVERY ESP in the failover chain (primary Brevo + all configured secondaries like SES) refused a send — transactional email is being dropped on the floor. This is the failure mode the failover feature exists to make loud: the standing Brevo-sender-unvalidated P0 with no working secondary. ANY tick means at least one customer-facing email (magic-link, claim, billing, expiry warning) silently failed across all providers. Runbook: check BOTH BREVO_API_KEY / Brevo sender-domain validation AND the SES_* credentials + SES sandbox/production-access state immediately; the forwarder holds its cursor on the last error class so backlog recovers once one provider is healthy. Source: worker/internal/metrics/metrics.go (EmailFailoverTotal), emit site worker/internal/email/failover_provider.go. Inert when EMAIL_PROVIDER_FALLBACK is unset.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' AND outcome = 'all_failed'"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 300,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/email-failover-engaged.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-worker — email failover engaged (primary ESP degraded, secondary carrying load)",
+  "type": "NRQL",
+  "description": "P1 page. instant_email_failover_total{outcome=\"fallback_ok\"} ticks every time the primary ESP (Brevo) errored or hard-rejected a send and the configured secondary (SES) delivered it instead. Email is NOT being lost, but the primary is degraded and the failover (worker/internal/email/failover_provider.go) is carrying production traffic. This is the leading indicator for the standing P0 (Brevo sender-domain unvalidated → every Brevo send rejected): a sustained fallback_ok rate means Brevo is down for sends and SES is the only thing keeping transactional email flowing. Runbook: check brevo_send_errors_total by classification, confirm Brevo sender-domain validation status in the Brevo dashboard, and verify SES production-access (not sandbox) so the secondary can carry full load. Source: worker/internal/metrics/metrics.go (EmailFailoverTotal), emit site worker/internal/email/failover_provider.go. Inert when EMAIL_PROVIDER_FALLBACK is unset (single-provider default never emits this metric).",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' AND outcome = 'fallback_ok'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/dashboards/instanode-reliability.json

@@ -333,6 +333,31 @@
             }
           }
         },
+        {
+          "title": "Email failover outcomes (1h) — fallback_ok=primary degraded (P1), all_failed=email lost (P0)",
+          "layout": {
+            "column": 1,
+            "row": 62,
+            "width": 6,
+            "height": 3
+          },
+          "visualization": {
+            "id": "viz.line"
+          },
+          "rawConfiguration": {
+            "nrqlQueries": [
+              {
+                "accountIds": [
+                  0
+                ],
+                "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' FACET outcome TIMESERIES SINCE 1 hour ago"
+              }
+            ],
+            "platformOptions": {
+              "ignoreTimeRange": false
+            }
+          }
+        },
         {
           "title": "Provisioner circuit-breaker state per backend",
           "layout": {

observability/METRICS-CATALOG.md

@@ -37,6 +37,7 @@ fires. Operators need this so they don't panic when a fresh deploy looks
 | `readyz_check_status` | api + worker + provisioner | `service,check` | **eager** (GaugeVec — set at boot by every /readyz probe; value 1=ok / 0.5=degraded / 0=failed) | `readyz-component-failed.json` | `ReadyzCheckFailed`, `ReadyzCheckDegraded` | "/readyz status (api / worker / provisioner)", "/readyz failed checks", "/readyz degraded checks" |
 | `instant_provisioner_circuit_state` | provisioner | `backend` | **eager** (GaugeVec — every breaker initialised at boot at state=0 CLOSED) | `provisioner-circuit-open.json` | `ProvisionerCircuitOpen`, `ProvisionerCircuitHalfOpen` | "Provisioner circuit-breaker state per backend" |
 | `email_missing_renderer_total` | worker | `kind` | lazy (CounterVec — any tick is a bug, label series only appears on the broken kind) | `email-missing-renderer.json` | `EmailMissingRenderer` | "Email missing-renderer ticks (any > 0 == P0)" |
+| `instant_email_failover_total` | worker | `outcome` | lazy (CounterVec — ONLY emits when `EMAIL_PROVIDER_FALLBACK` is configured; inert single-provider default never observes a label. `primary_ok` on first send, `fallback_ok`/`all_failed` only on a real primary failure) | `email-failover-engaged.json`, `email-failover-all-failed.json` | `EmailFailoverEngaged` (P1), `EmailFailoverAllFailed` (P0) | "Email failover outcomes (1h)" |
 | `migration_version`, `migration_count`, `migration_status` (worker `/healthz` JSON fields, NOT Prometheus metrics) | worker | n/a (log-based) | **eager** (read live from `schema_migrations` table by `migrations.Reader`, cached 60s) | `worker-migration-mismatch.json` (log-based) | n/a (log-based) | "Worker /healthz migration_count drift" |
 | `instant_idempotency_replay_refunded_total` | api | `route` | lazy (CounterVec — first cache HIT on each route materialises the label series; a fresh deploy with no retries reports nothing until the first agent retries with the same `Idempotency-Key`) | `idempotency-replay-refund-spike.json` | `IdempotencyReplayRefundSpike` | "Idempotency replay refunds by route (1h) — FINDING API-1" |
 | `instant_auth_probe_outcome_total` | worker | `leg,result` | lazy (CounterVec — `pass`/`degraded` materialise on the first happy tick; `fail` only appears after a real regression. AUTH-004 synthetic prober: every 5 min the worker drives /auth/email/start + /auth/exchange CORS contract + /auth/me bearer against prod) | `auth-probe-fail.json` | `AuthProbeFail` | "AUTH-004 synthetic prober — outcomes per leg (1h)", "AUTH-004 synthetic prober — fails (last 1h, must be 0)" |