Skip to content

fix(framework): Interrupt Fleet retries during teardown - #7922

Draft
tanertopal wants to merge 1 commit into
agent/shutdown-runtime-cleanupfrom
agent/shutdown-fleet-retries
Draft

fix(framework): Interrupt Fleet retries during teardown#7922
tanertopal wants to merge 1 commit into
agent/shutdown-runtime-cleanupfrom
agent/shutdown-fleet-retries

Conversation

@tanertopal

@tanertopal tanertopal commented Aug 18, 2026

Copy link
Copy Markdown
Member

What changed

  • disable the Fleet retry coordinator before node teardown RPCs
  • wake an active heartbeat backoff instead of only lowering its future attempt limit
  • verify authenticated connection cleanup invokes retry cancellation

Why

A Fleet heartbeat already waiting in backoff could survive SIGTERM cleanup and keep connection teardown blocked. Explicit cancellation wakes that wait before deactivation and unregister calls run.

Stack

Umbrella/reference: #7895 — Prevent task executor shutdown deadlock

Merge the stack in order. After the previous PR merges, retarget this PR to main.

Validation

  • pytest py/flwr/client/grpc_rere_client/node_auth_client_interceptor_test.py
  • Ruff and Black on the changed connection modules
  • git diff --check

@github-actions github-actions Bot added the Maintainer Used to determine what PRs (mainly) come from Flower maintainers. label Aug 18, 2026
@tanertopal
tanertopal force-pushed the agent/shutdown-runtime-cleanup branch from 15a2c89 to d26f847 Compare August 18, 2026 16:11
@tanertopal
tanertopal force-pushed the agent/shutdown-fleet-retries branch from 8a36848 to 17b9dbe Compare August 18, 2026 16:11
@tanertopal
tanertopal force-pushed the agent/shutdown-runtime-cleanup branch from d26f847 to cdbc04f Compare August 18, 2026 16:25
@tanertopal
tanertopal force-pushed the agent/shutdown-fleet-retries branch from 17b9dbe to fcfa48b Compare August 18, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Maintainer Used to determine what PRs (mainly) come from Flower maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant