Replace In-Memory Rate Limiting with Atomic Redis-Backed Distributed Limiter (Multi-Replica Correct, IETF Headers, Fail-Open/Closed Policies)

[Christopherdominic]

Summary

Rate limiting uses tower_governor/governor, which keeps its token buckets in process memory. The repo ships Helm charts and k8s manifests — i.e. it is designed to run multiple replicas — where per-instance buckets make the effective limit N_replicas × configured_limit, drifting arbitrarily as HPA scales. Limits are also lost on every restart/deploy, and there is no coordination with the Redis instance the service already maintains a connection manager for.

Task

1. Implement a Redis-backed distributed rate limiter (atomic sliding-window or GCRA via a single Lua script / EVALSHA — no read-modify-write races) keyed by client identity.
2. Layered policies, configurable via the typed config:
   • per-IP (with correct client-IP extraction behind the ingress: rightmost trusted X-Forwarded-For hop, configurable trusted-proxy depth — spoofable naive parsing is an automatic reject)
   • per-API-key / per-authenticated-user (integrating with middleware/api_key_auth.rs and middleware/auth.rs)
   • per-route-class overrides (e.g. stricter on auth endpoints, looser on public reads)
3. Emit standard RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset headers (IETF draft) plus Retry-After on 429s.
4. Fail-open vs fail-closed must be an explicit per-policy config decision when Redis is unavailable, with a metrics counter for degraded-mode decisions; auth endpoints default fail-closed, public reads fail-open.
5. Load-test evidence: run two service instances against one Redis and demonstrate the combined observed limit equals the configured limit (±5%); include the harness (e.g. k6/vegeta script) in scripts/.
6. Concurrency-safe unit tests for the Lua/GCRA math (burst, sustained, window rollover) and integration tests for header emission and 429 behavior.

Acceptance criteria

• Two replicas share one limit pool, verified by the included load-test script output
• No read-modify-write race (single atomic script; test hammers with concurrent requests)
• Standard rate-limit headers on all limited routes; Retry-After on 429
• Explicit, tested degraded-mode behavior per policy when Redis is down
• All checks pass

PR requirements (mandatory)

• ✅ Your PR must pass all checks — PRs with failing or skipped checks will not be merged.
• 📸 You must attach a screenshot in the PR description demonstrating the result (for this issue: the two-replica load-test output showing the shared limit enforced).
• 🔗 You must link this issue number in your PR description (e.g. Closes #<issue-number>). PRs without a linked issue will not be reviewed.