Make relay readiness process-local - #7341
Draft
TheSentinel454 wants to merge 3 commits into
Draft
Conversation
🔐 Codex Security Review
|
A reconnect burst exhausted the per-pod writer pools and two feedback
loops turned that into a total outage.
Readiness evaluated shared Postgres, Redis, and deletion-catalog health,
so every replica went NotReady together and the burst had nowhere to
land. The probe was also part of the load: the deletion-catalog check
acquires the writer pool, so each pod spent writer connections against
the exhausted pool every five seconds while failing. /_readiness now
answers from local process lifecycle only — shutting_down is 503,
anything else is 200 — and the dependency evaluation moves to /_status
on the same private health listener, under a `dependencies` object
carrying the fields the readiness body used to return. No startup state
is added: the health listener binds only after the database,
migrations, Redis, and pub/sub are up, so a process that can answer has
booted.
run_registered_community_connection collapsed Ok(false) and Err into
"not active", so a writer-pool timeout in is_community_active read as
confirmed archival and dropped the socket, which reconnected and
re-checked. Only a confirmed Ok(false) cancels now; a lookup failure
admits the socket with a structured warning and defers to the periodic
revalidate_live_communities backstop. Writes are unaffected and remain
fail-closed on their own per-event fence.
Telemetry keeps its existing names: buzz_readiness_checks_total narrows
to {ready, shutting_down}, the dependency families are now sampled by
/_status, dependency gauges are dropped, and one new bounded counter,
buzz_community_admission_checks_total{outcome}, counts the admission
decision. The per-pod raw-series ceiling drops from 99 to 86.
This deletes the readiness publication machinery — the mutex, probe
generations, ProbeTicket/ProbeStart, finish_probe,
finish_public_evaluation, and a second shutdown flag duplicating
AppState::shutting_down. All of it existed to order concurrent async
dependency evaluations against shutdown. Readiness is now a single
atomic load, so the one ordering guarantee still worth keeping — a
racing shutdown must win, and never leave a draining pod advertising a
ready gauge — is a post-write re-read in record_readiness_probe rather
than a generation-fenced mutex.
Co-authored-by: Claude Code <noreply@anthropic.com>
Redis had no startup gate at all. `deadpool_redis` pools dial lazily and
PubSubManager::new only allocates channels, so "Redis pub/sub connected"
was logged against a dead port and boot ran to completion. With readiness
now answering from local lifecycle alone, such a pod bound its health
listener and advertised ready for the rest of its life. state::
verify_redis_command_path acquires one connection from the command pool
and issues PING before AppState is built, and therefore before the health
listener binds, because binding is the one-way latch that makes a pod
routable. No startup_ready flag is added for the same reason. Post-start
Redis failures are unchanged: they are dependency failures and never move
readiness. Postgres startup connection behavior is untouched.
Signed-off-by: tornquist <tornquist@squareup.com>
TheSentinel454
force-pushed
the
tornquist/relay-readiness-overload
branch
from
September 4, 2026 20:04
25bc2c2 to
a8e2c48
Compare
Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Keep dependency checks on
/_status, preserve WebSocket connections when the community-active database check is inconclusive, and add bounded rollout metrics for both paths.This simplifies readiness from a serialized shared-dependency coordinator to a two-state local decision: ready or shutting down. Dependency health remains observable without controlling load-balancer membership.
Related issue
None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03.
Testing
Verified at
25bc2c29ddd2027fc38a0b0b81d0c229b31b2550on Blox with:cargo fmt --all -- --checkcargo clippy -p buzz-relay --all-targets -- -D warningscargo test -p buzz-relay --lib readiness::— 8 passedcargo test -p buzz-relay --lib router::tests— 13 passedcargo test -p buzz-relay --lib state::tests— 24 passed, 1 ignoredcargo test -p buzz-relay --test boot_lifecycle— 9 passedThe complete
cargo test -p buzz-relay --librun was not green:telemetry::tests::trace_context_lookup_does_not_enable_callsitesfailed with 1040 passed, 1 failed, and 89 ignored. Baseline characterization was inconsistent: the telemetry test passed across six baseline runs while the existing mesh-demo timeout failed intermittently. This PR does not change telemetry or mesh-demo code.Generated with Claude Code