fix(multiagent): add DISPATCHED status to prevent fan-in double-execution on resume (#3633) - #3724
Open
SuperMarioYL wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a graph node is dispatched (
asyncio.create_task) but the process crashes before the node's generator body setsStatus.EXECUTING(e.g., Lambda timeout, container restart,os._exit), the node is leftPENDING. On session resume,_compute_pending_sourcesseeds only fromEXECUTINGnodes — so a dispatched-but-not-started node with no completed ancestor is indistinguishable from a bypassed dead branch. Its fan-in children are marked ready off the completed sibling alone, and the fan-in is double-executed on resume.This adds a
DISPATCHEDstatus to theStatusenum, set at the dispatch site immediately beforeasyncio.create_task._compute_pending_sourcesnow seeds from bothEXECUTINGandDISPATCHED, closing the window between dispatch and generator-body start. This completes the partial fix from #3390, which added theEXECUTINGseed but left the dispatched-but-not-started gap.Changes:
base.py: AddDISPATCHED = "dispatched"toStatusenum (additive, non-breaking)graph.py: SetDISPATCHEDat the dispatch site in_execute_nodes_parallel; update_compute_pending_sourcesto seed fromEXECUTINGandDISPATCHEDtest_graph.py: AddTestResumeDispatchedSiblingwith two tests verifying the fan-in is not double-executed when a sibling was dispatched but not startedRelated Issues
Fixes #3633
Documentation PR
No documentation changes needed — this is an internal status enum addition with no new public API surface.
Type of Change
Bug fix
Testing
Added
TestResumeDispatchedSiblingwith two tests:test_dispatched_sibling_excluded_from_resume_frontier: focused unit test that injectsDISPATCHEDstate directly (right.execution_status = DISPATCHED) and asserts the fan-injoinis excluded fromnext_nodes_to_executewhile the dispatched siblingrightis included. It pins the resume-frontier computation; on raw master it raisesAttributeErroronStatus.DISPATCHED.test_resume_runs_fan_in_once_for_dispatched_sibling: real-dispatch end-to-end reproduction — a fast-completingleftentry point and arightentry point held in the dispatched window (dispatched via the real_execute_nodes_parallelpath, blocked before_execute_nodesetsEXECUTING). The snapshot is captured fromleft'sAfterNodeCallEvent, deserialized into a fresh graph, and resumed;joinmust run exactly once with both parents' outputs present.Both tests are red on master via distinct, valid signals: the end-to-end test fails with an assertion failure (
joinlands innext_nodes_to_execute→ double-execution on resume, becauserightstaysPENDINGand is treated as a bypassed dead branch), while the unit test raisesAttributeErroronStatus.DISPATCHED. Both are green on the branch.All 83 existing
test_graph.pytests passAll 181 broader multiagent tests pass (excluding
a2awhich has a pre-existing missing-dependency issue)ruff checkandruff format --checkpassmypypasses with no issuesI ran
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.