Skip to content

Shutdown race conditions#110

Open
vnermolaev wants to merge 4 commits intomaiko-rs:mainfrom
vnermolaev:issue-25
Open

Shutdown race conditions#110
vnermolaev wants to merge 4 commits intomaiko-rs:mainfrom
vnermolaev:issue-25

Conversation

@vnermolaev
Copy link
Copy Markdown
Contributor

@vnermolaev vnermolaev commented Mar 9, 2026

Context

Addresses #25.
Before this PR, there was no global stop-gate protecting Stage-1 producers.
Shutdown relied on broker-side timeout heuristics (10ms + 100us sleep loops), which is fragile under load.

PR changes

  1. Introduces a global stop-gate (CancellationToken) shared by Supervisor and actor Context.
  2. Adds internal::gated_send for shutdown-aware Stage-1 send:
  • check stop-gate
  • reserve().await
  • re-check stop-gate
  • permit.send(...) (this is sync, so definitely nothing can get queued)
  1. Routes both send entrypoints through gated_send:
  • Context::send_envelope
  • Supervisor::send
  1. Publishes shutdown Supervisor::stop.
  2. Replaces broker timeout polling with deterministic coordination:
  • close Stage-1 receivers,
  • drain every sender until sender.next() returns None.
  1. Adds tests covering gate behavior, including the “blocked on capacity, then stop flips” case.

Why this addresses the race

Previously, lack of a stop-gate meant sends could slip through during shutdown transitions, and timeout polling tried to “wait enough.”
Now, shutdown is explicit and synchronized:

  • producers are gated at ingress,
  • blocked senders re-check after capacity wait,
  • broker closes ingress and drains remaining buffered events deterministically.

Status / caveat

@ddrcode This may not be the final ready-to-ship shutdown model, but it is a promising direction aligned with the ticket: it removes magic timing assumptions and introduces explicit coordination.

Extension path to multiple brokers

The same model scales:

  • one shared stop-gate across all producers,
  • close-and-drain per broker,
  • await all broker tasks before runtime shutdown completes.

This keeps shutdown semantics explicit and composable as broker count grows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant