spawn_parallel_agents: opt out of the 120s per-tool timeout so the fan-out isn't truncated#4686
Conversation
spawn_parallel_agents is a fan-out primitive: it spawns N independent sub-agents and awaits them concurrently via spawn_parallel_graph's bounded map_reduce. It never overrode Tool::timeout_policy, so it inherited the global per-tool wall-clock deadline (default 120s) and the whole fan-out was hard-killed at 120s — truncating every worker's work and bounding the group at one worker's budget instead of ~the slowest. Override timeout_policy to Unbounded, matching the long-running scripting tools (shell, node_exec). The fan-out is already bounded internally by its max_concurrency, the run cancellation token, and each sub-agent's own iteration/turn caps, so it governs its own lifetime. Add a unit test.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthrough
ChangesSpawn Parallel Agents Timeout Override
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
|
CI + review pass — LGTM. Reviewed
Checks: |
…yhumansai#4734) ArchetypeDelegationTool synthesizes delegate_tools_agent (tools_agent) and run_code (code_executor). It defaulted to ToolTimeout::Inherit = the global 120s per-tool cap, so any delegated sub-agent run exceeding two minutes was hard-killed and truncated (Sentry TAURI-RUST-K29: 1236 events / 71 users; TAURI-RUST-8HB: 755 / 51). The child bounds its own lifetime via max_iterations, the run cancellation token, and each inner tool's timeout, so override timeout_policy -> Unbounded, matching spawn_parallel_agents (tinyhumansai#4686) and the long-running scripting tools. Adds a unit test asserting the override.
Bug #2 — 120s truncation (root-caused + fixed here)
Every OpenHuman tool is wrapped in
tokio::time::timeout(deadline, exec)(tinyagents/tools.rs), wheredeadlinefor the defaultToolTimeout::Inheritis the global per-tool wall-clock (tool_timeoutDEFAULT_TIMEOUT_SECS = 120).SpawnParallelAgentsToolnever overrodetimeout_policy, so the entire fan-out was hard-killed at 120s — the parent span's exact 120.000s — truncating every worker and bounding the group at one worker's budget instead of ~the slowest.Fix: override
timeout_policy→ToolTimeout::Unbounded, matching the long-running scripting tools (shell,node_exec,rhai_workflows). The fan-out is already bounded internally — by itsmax_concurrency, the run cancellation token, and each sub-agent's own iteration/turn caps — so it governs its own lifetime rather than the single-tool default. Adds a unit test.Bug #1 — serialization (investigated; not an OpenHuman fan-out bug)
The fan-out already runs concurrently:
run_spawn_parallel_workerscallstinyagents::graph::parallel::map_reducewithmax_concurrency = prepared.len()overbuffer_unordered(concurrent since #4249). The serial fallback only triggers for shared-workspace write workers — read-only research never hits it;run_queueisNone; and the managed (OpenAI-compatible) provider has no client-side concurrency semaphore. So OpenHuman does not request serial execution. The observed strict serialization isn't reproducible from any OpenHuman fan-out line — the likely causes are the 120s parent-timeout aborting the concurrent fan-out (removed by this PR) or server-side per-token serialization in the managed backend. I deliberately avoided a speculative concurrency change since concurrency is alreadymax_concurrency=N.Note / follow-up
Each worker's own model call and inner tools still inherit the global 120s (
tool_timeout+ inferencerequest_timeout). Making those higher or research-tier-configurable is a separate global-config/design decision, flagged rather than bundled here.Prompt-independent from #4680 (that fix was orchestrator-prompt-only and does not touch this execution path).
Closes #4685
Summary by CodeRabbit
Bug Fixes
Tests