perf: connection pool budget accounting across relay process

[TheSentinel454]

Problem

A single relay process creates 3-4 independent Postgres connection pools hitting the same database, with inconsistent configuration:

Pool	Default max	min	acquire_timeout	Config source
Db writer	50 (relay) / 20 (DbConfig default)	2	3s	BUZZ_DB_POOL_SIZE → DbConfig
Db reader	= writer	0	150ms	BUZZ_DB_READ_POOL_SIZE
Audit	5	1	sqlx default (30s!)	hardcoded in main.rs:357
Search	sqlx default (10)	sqlx default (0)	sqlx default (30s!)	hardcoded in main.rs:412

Issues:

1. Audit and Search pools use sqlx defaults — no explicit acquire_timeout, max_lifetime, or idle_timeout. The sqlx default acquire_timeout is 30 seconds, 10x the writer pool's 3s. Under pool exhaustion, a search or audit request holds a task for 30 seconds before failing.

2. No unified connection budget — the total connection count per pod is writer_max + reader_max + audit_max + search_max but there is no single place this sum is computed or validated against PG max_connections. The comment in DbConfig::default() says "Sized for a single relay pod against PG max_connections=100" but the relay overrides to 50, and audit+search add 15 more = 115 per pod if using all pools.

3. Search pool deliberately uses the read replica when available (good) but has no replica-failure fallback — if the read replica goes down, search queries fail entirely rather than degrading to the writer.

4. Audit pool has min_connections: 1 which eagerly dials at construction, even if audit is rarely used. The writer pool uses min_connections: 2 for the same reason (pre-warming). But audit's 1 eager connection with a 30-second acquire timeout means a boot-time Postgres hiccup blocks startup for 30 seconds on audit alone.

Proposed changes

1. Centralize pool configuration: move audit and search pool creation into DbConfig or a PoolBudget struct that computes the total connection budget and validates it against a configurable PG max_connections ceiling. Log the breakdown at startup.

2. Standardize timeouts: all pools should use the same acquire_timeout (3s), max_lifetime (1800s), and idle_timeout (600s) unless there's a documented reason to diverge.

3. Add connection budget metrics: emit a startup log line and a Prometheus gauge showing total_max_connections = writer + reader + audit + search vs the PG ceiling. Alert when utilization > 80%.

4. Search pool replica fallback: when READ_DATABASE_URL is configured and the search pool's replica is down, fall back to the writer pool rather than failing search entirely.

Priority

Medium — connection budget misaccounting is a deploy-time risk when scaling replicas.

🤖 AI review update (2026-08-23)

Elevate the accounting portion and fold it into #11. Budget by pool role and deployment replica/HPA maximum, reserve database headroom, and emit declared maxima plus actual active/idle/acquire failures; querying SHOW max_connections can validate the configured ceiling. Preserve documented role-specific timeouts and bulkheads, and treat search fallback as a routed behavior rather than raw pool selection.