diff --git a/k8s/prometheus-rules.yaml b/k8s/prometheus-rules.yaml index 4c29c7e..14d1661 100644 --- a/k8s/prometheus-rules.yaml +++ b/k8s/prometheus-rules.yaml @@ -521,6 +521,52 @@ spec: coverage test that iterates eventEmailBuilders so this can't happen again for the same kind. + # Failover-ESP engagement. The worker's FailoverProvider + # (worker/internal/email/failover_provider.go) emits + # instant_email_failover_total{outcome} on every send when + # EMAIL_PROVIDER_FALLBACK is configured. Two alert-able outcomes: + # fallback_ok — primary ESP (Brevo) failed/rejected but the secondary + # (SES) sent the email. Email is NOT lost, but the primary + # is degraded and the failover is carrying load → P1. + # all_failed — every ESP in the chain refused the send. Email IS being + # lost → P0. This is the failure mode the feature exists to + # surface (the standing Brevo-sender-unvalidated P0 with no + # working secondary). + # primary_ok is the healthy path and is deliberately NOT alerted. + - name: instant-worker-email-failover + rules: + - alert: EmailFailoverEngaged + expr: | + sum(rate(instant_email_failover_total{outcome="fallback_ok"}[10m])) > 0 + for: 10m + labels: + severity: warning + service: worker + annotations: + summary: "Email failover engaged — primary ESP degraded, secondary carrying load" + description: | + instant_email_failover_total{outcome="fallback_ok"} rate > 0 for + >10m. The primary ESP (Brevo) is failing or rejecting sends and + the configured secondary (SES) is delivering them instead. Email + is still flowing, but the primary needs attention — check + brevo_send_errors_total by classification and the Brevo sender- + domain validation status. + - alert: EmailFailoverAllFailed + expr: | + sum(rate(instant_email_failover_total{outcome="all_failed"}[5m])) > 0 + for: 5m + labels: + severity: critical + service: worker + annotations: + summary: "Email failover ALL providers failed — email is being LOST" + description: | + instant_email_failover_total{outcome="all_failed"} rate > 0 for + >5m. Every ESP in the failover chain (primary + all secondaries) + refused the send — transactional email is being dropped on the + floor. Check both BREVO_API_KEY/sender validation and the SES_* + credentials + SES sandbox/production-access state immediately. + # /readyz deep-health gauge, exposed by api, worker, and provisioner via # readyz_check_status{service,check}. 1=ok, 0.5=degraded, 0=failed. This # rule mirrors the NR alert readyz-component-failed.json (which depends diff --git a/newrelic/alerts/email-failover-all-failed.json b/newrelic/alerts/email-failover-all-failed.json new file mode 100644 index 0000000..0646f25 --- /dev/null +++ b/newrelic/alerts/email-failover-all-failed.json @@ -0,0 +1,31 @@ +{ + "name": "instant-worker — email failover ALL providers failed (email is being LOST)", + "type": "NRQL", + "description": "P0 page. instant_email_failover_total{outcome=\"all_failed\"} ticks when EVERY ESP in the failover chain (primary Brevo + all configured secondaries like SES) refused a send — transactional email is being dropped on the floor. This is the failure mode the failover feature exists to make loud: the standing Brevo-sender-unvalidated P0 with no working secondary. ANY tick means at least one customer-facing email (magic-link, claim, billing, expiry warning) silently failed across all providers. Runbook: check BOTH BREVO_API_KEY / Brevo sender-domain validation AND the SES_* credentials + SES sandbox/production-access state immediately; the forwarder holds its cursor on the last error class so backlog recovers once one provider is healthy. Source: worker/internal/metrics/metrics.go (EmailFailoverTotal), emit site worker/internal/email/failover_provider.go. Inert when EMAIL_PROVIDER_FALLBACK is unset.", + "enabled": true, + "nrql": { + "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' AND outcome = 'all_failed'" + }, + "terms": [ + { + "priority": "CRITICAL", + "operator": "ABOVE", + "threshold": 0, + "thresholdDuration": 300, + "thresholdOccurrences": "AT_LEAST_ONCE" + } + ], + "signal": { + "aggregationWindow": 60, + "aggregationMethod": "EVENT_FLOW", + "aggregationDelay": 120, + "fillOption": "STATIC", + "fillValue": 0 + }, + "expiration": { + "expirationDuration": 3600, + "openViolationOnExpiration": false, + "closeViolationsOnExpiration": true + }, + "violationTimeLimitSeconds": 86400 +} diff --git a/newrelic/alerts/email-failover-engaged.json b/newrelic/alerts/email-failover-engaged.json new file mode 100644 index 0000000..fc3d30e --- /dev/null +++ b/newrelic/alerts/email-failover-engaged.json @@ -0,0 +1,31 @@ +{ + "name": "instant-worker — email failover engaged (primary ESP degraded, secondary carrying load)", + "type": "NRQL", + "description": "P1 page. instant_email_failover_total{outcome=\"fallback_ok\"} ticks every time the primary ESP (Brevo) errored or hard-rejected a send and the configured secondary (SES) delivered it instead. Email is NOT being lost, but the primary is degraded and the failover (worker/internal/email/failover_provider.go) is carrying production traffic. This is the leading indicator for the standing P0 (Brevo sender-domain unvalidated → every Brevo send rejected): a sustained fallback_ok rate means Brevo is down for sends and SES is the only thing keeping transactional email flowing. Runbook: check brevo_send_errors_total by classification, confirm Brevo sender-domain validation status in the Brevo dashboard, and verify SES production-access (not sandbox) so the secondary can carry full load. Source: worker/internal/metrics/metrics.go (EmailFailoverTotal), emit site worker/internal/email/failover_provider.go. Inert when EMAIL_PROVIDER_FALLBACK is unset (single-provider default never emits this metric).", + "enabled": true, + "nrql": { + "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' AND outcome = 'fallback_ok'" + }, + "terms": [ + { + "priority": "WARNING", + "operator": "ABOVE", + "threshold": 0, + "thresholdDuration": 600, + "thresholdOccurrences": "AT_LEAST_ONCE" + } + ], + "signal": { + "aggregationWindow": 60, + "aggregationMethod": "EVENT_FLOW", + "aggregationDelay": 120, + "fillOption": "STATIC", + "fillValue": 0 + }, + "expiration": { + "expirationDuration": 3600, + "openViolationOnExpiration": false, + "closeViolationsOnExpiration": true + }, + "violationTimeLimitSeconds": 86400 +} diff --git a/newrelic/dashboards/instanode-reliability.json b/newrelic/dashboards/instanode-reliability.json index 72adf1d..71391fd 100644 --- a/newrelic/dashboards/instanode-reliability.json +++ b/newrelic/dashboards/instanode-reliability.json @@ -333,6 +333,31 @@ } } }, + { + "title": "Email failover outcomes (1h) — fallback_ok=primary degraded (P1), all_failed=email lost (P0)", + "layout": { + "column": 1, + "row": 62, + "width": 6, + "height": 3 + }, + "visualization": { + "id": "viz.line" + }, + "rawConfiguration": { + "nrqlQueries": [ + { + "accountIds": [ + 0 + ], + "query": "SELECT sum(instant_email_failover_total) FROM Metric WHERE metricName = 'instant_email_failover_total' FACET outcome TIMESERIES SINCE 1 hour ago" + } + ], + "platformOptions": { + "ignoreTimeRange": false + } + } + }, { "title": "Provisioner circuit-breaker state per backend", "layout": { diff --git a/observability/METRICS-CATALOG.md b/observability/METRICS-CATALOG.md index a2ac0ab..7c830a9 100644 --- a/observability/METRICS-CATALOG.md +++ b/observability/METRICS-CATALOG.md @@ -37,6 +37,7 @@ fires. Operators need this so they don't panic when a fresh deploy looks | `readyz_check_status` | api + worker + provisioner | `service,check` | **eager** (GaugeVec — set at boot by every /readyz probe; value 1=ok / 0.5=degraded / 0=failed) | `readyz-component-failed.json` | `ReadyzCheckFailed`, `ReadyzCheckDegraded` | "/readyz status (api / worker / provisioner)", "/readyz failed checks", "/readyz degraded checks" | | `instant_provisioner_circuit_state` | provisioner | `backend` | **eager** (GaugeVec — every breaker initialised at boot at state=0 CLOSED) | `provisioner-circuit-open.json` | `ProvisionerCircuitOpen`, `ProvisionerCircuitHalfOpen` | "Provisioner circuit-breaker state per backend" | | `email_missing_renderer_total` | worker | `kind` | lazy (CounterVec — any tick is a bug, label series only appears on the broken kind) | `email-missing-renderer.json` | `EmailMissingRenderer` | "Email missing-renderer ticks (any > 0 == P0)" | +| `instant_email_failover_total` | worker | `outcome` | lazy (CounterVec — ONLY emits when `EMAIL_PROVIDER_FALLBACK` is configured; inert single-provider default never observes a label. `primary_ok` on first send, `fallback_ok`/`all_failed` only on a real primary failure) | `email-failover-engaged.json`, `email-failover-all-failed.json` | `EmailFailoverEngaged` (P1), `EmailFailoverAllFailed` (P0) | "Email failover outcomes (1h)" | | `migration_version`, `migration_count`, `migration_status` (worker `/healthz` JSON fields, NOT Prometheus metrics) | worker | n/a (log-based) | **eager** (read live from `schema_migrations` table by `migrations.Reader`, cached 60s) | `worker-migration-mismatch.json` (log-based) | n/a (log-based) | "Worker /healthz migration_count drift" | | `instant_idempotency_replay_refunded_total` | api | `route` | lazy (CounterVec — first cache HIT on each route materialises the label series; a fresh deploy with no retries reports nothing until the first agent retries with the same `Idempotency-Key`) | `idempotency-replay-refund-spike.json` | `IdempotencyReplayRefundSpike` | "Idempotency replay refunds by route (1h) — FINDING API-1" | | `instant_auth_probe_outcome_total` | worker | `leg,result` | lazy (CounterVec — `pass`/`degraded` materialise on the first happy tick; `fail` only appears after a real regression. AUTH-004 synthetic prober: every 5 min the worker drives /auth/email/start + /auth/exchange CORS contract + /auth/me bearer against prod) | `auth-probe-fail.json` | `AuthProbeFail` | "AUTH-004 synthetic prober — outcomes per leg (1h)", "AUTH-004 synthetic prober — fails (last 1h, must be 0)" |