Skip to content

spawn_parallel_agents: opt out of the 120s per-tool timeout so the fan-out isn't truncated#4686

Merged
senamakel merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/GH-4685-parallel-agents-concurrency
Jul 8, 2026
Merged

spawn_parallel_agents: opt out of the 120s per-tool timeout so the fan-out isn't truncated#4686
senamakel merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/GH-4685-parallel-agents-concurrency

Conversation

@M3gA-Mind

@M3gA-Mind M3gA-Mind commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Bug #2 — 120s truncation (root-caused + fixed here)

Every OpenHuman tool is wrapped in tokio::time::timeout(deadline, exec) (tinyagents/tools.rs), where deadline for the default ToolTimeout::Inherit is the global per-tool wall-clock (tool_timeout DEFAULT_TIMEOUT_SECS = 120). SpawnParallelAgentsTool never overrode timeout_policy, so the entire fan-out was hard-killed at 120s — the parent span's exact 120.000s — truncating every worker and bounding the group at one worker's budget instead of ~the slowest.

Fix: override timeout_policyToolTimeout::Unbounded, matching the long-running scripting tools (shell, node_exec, rhai_workflows). The fan-out is already bounded internally — by its max_concurrency, the run cancellation token, and each sub-agent's own iteration/turn caps — so it governs its own lifetime rather than the single-tool default. Adds a unit test.

Bug #1 — serialization (investigated; not an OpenHuman fan-out bug)

The fan-out already runs concurrently: run_spawn_parallel_workers calls tinyagents::graph::parallel::map_reduce with max_concurrency = prepared.len() over buffer_unordered (concurrent since #4249). The serial fallback only triggers for shared-workspace write workers — read-only research never hits it; run_queue is None; and the managed (OpenAI-compatible) provider has no client-side concurrency semaphore. So OpenHuman does not request serial execution. The observed strict serialization isn't reproducible from any OpenHuman fan-out line — the likely causes are the 120s parent-timeout aborting the concurrent fan-out (removed by this PR) or server-side per-token serialization in the managed backend. I deliberately avoided a speculative concurrency change since concurrency is already max_concurrency=N.

Note / follow-up

Each worker's own model call and inner tools still inherit the global 120s (tool_timeout + inference request_timeout). Making those higher or research-tier-configurable is a separate global-config/design decision, flagged rather than bundled here.

Prompt-independent from #4680 (that fix was orchestrator-prompt-only and does not touch this execution path).

Closes #4685

Summary by CodeRabbit

  • Bug Fixes

    • Improved reliability for parallel agent spawning by allowing this operation to run without the standard per-tool time limit, reducing unexpected timeouts during larger fan-out tasks.
  • Tests

    • Added coverage to verify the parallel agent spawning behavior is not constrained by the global tool timeout.

spawn_parallel_agents is a fan-out primitive: it spawns N independent
sub-agents and awaits them concurrently via spawn_parallel_graph's
bounded map_reduce. It never overrode Tool::timeout_policy, so it
inherited the global per-tool wall-clock deadline (default 120s) and the
whole fan-out was hard-killed at 120s — truncating every worker's work
and bounding the group at one worker's budget instead of ~the slowest.

Override timeout_policy to Unbounded, matching the long-running scripting
tools (shell, node_exec). The fan-out is already bounded internally by
its max_concurrency, the run cancellation token, and each sub-agent's own
iteration/turn caps, so it governs its own lifetime. Add a unit test.
@M3gA-Mind M3gA-Mind requested a review from a team July 7, 2026 23:33
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ed1af5f9-75cd-40aa-a89b-f9647c73b16f

📥 Commits

Reviewing files that changed from the base of the PR and between 89d601c and d8e8a6a.

📒 Files selected for processing (2)
  • src/openhuman/agent_orchestration/tools/spawn_parallel_agents.rs
  • src/openhuman/agent_orchestration/tools/spawn_parallel_agents_tests.rs

📝 Walkthrough

Walkthrough

SpawnParallelAgentsTool now implements a timeout_policy method returning ToolTimeout::Unbounded, exempting the tool from the global per-tool wall-clock timeout. A new unit test verifies this behavior.

Changes

Spawn Parallel Agents Timeout Override

Layer / File(s) Summary
Unbounded timeout policy
src/openhuman/agent_orchestration/tools/spawn_parallel_agents.rs
Imports ToolTimeout and adds a timeout_policy method returning ToolTimeout::Unbounded to disable the global timeout for this fan-out tool.
Test coverage
src/openhuman/agent_orchestration/tools/spawn_parallel_agents_tests.rs
Imports ToolTimeout and adds a test asserting timeout_policy returns ToolTimeout::Unbounded.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

  • tinyhumansai/openhuman#4262: Updates run_agent_tool_call to honor each tool's timeout_policy, directly consuming the override added here.

Suggested labels: agent

Poem

A rabbit hops with timer freed,
No 120-second cap to heed,
Parallel workers run and race,
Unbounded now, they find their pace,
Hop hop hooray, the fan-out's fast indeed! 🐇⏱️

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR fixes the timeout truncation, but it does not address the linked issue’s sequential fan-out/concurrency bug. Investigate and fix the spawn_parallel_agents execution path so workers run concurrently, then keep the timeout change if still needed.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: opting out of the 120s timeout for spawn_parallel_agents.
Out of Scope Changes check ✅ Passed The changes stay focused on spawn_parallel_agents timeout behavior and its test, with no obvious unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot added the agent Built-in agents, prompts, orchestration, and agent runtime in src/openhuman/agent/. label Jul 7, 2026
@M3gA-Mind

Copy link
Copy Markdown
Collaborator Author

CI + review pass — LGTM.

Reviewed origin/main...HEAD. The fix overrides SpawnParallelAgentsTool::timeout_policy to ToolTimeout::Unbounded, removing the global 120s per-tool wall-clock cap that this fan-out primitive was silently inheriting (default ToolTimeout::Inherit) — which truncated every worker and bounded the whole group at one worker's budget. Verified:

  • Correct/minimal: a single timeout_policy override + import + unit test.
  • Matches the established pattern: shell, node_exec, npm_exec, and plan_review all opt out the same way (-> ToolTimeout::Unbounded, documented as 'no harness-imposed deadline').
  • No safety bound removed: the fan-out is still bounded internally — the task count is capped at max_parallel_tools (enforced with an error in spawn_parallel_graph), concurrency is bounded by map_reduce/ParallelOptions, the run CancellationToken still applies, and each sub-agent keeps its own iteration/turn caps. The 120s cap was inappropriate for a group primitive, not a needed guard.
  • Test present + valid: asserts timeout_policy(&json!({})) == ToolTimeout::Unbounded (enum derives PartialEq, so it compiles) and passes in CI.

Checks: cargo check --lib ✓ clean compile, cargo fmt --check ✓; GitHub CI green — Rust Quality (fmt, clippy), Rust Core Coverage (runs the new test), and the rest all pass (10/10, 5 skipped as not-applicable). No fix needed.

@senamakel senamakel merged commit 63163ea into tinyhumansai:main Jul 8, 2026
19 of 23 checks passed
oxoxDev added a commit to oxoxDev/openhuman that referenced this pull request Jul 9, 2026
…yhumansai#4734)

ArchetypeDelegationTool synthesizes delegate_tools_agent (tools_agent) and
run_code (code_executor). It defaulted to ToolTimeout::Inherit = the global
120s per-tool cap, so any delegated sub-agent run exceeding two minutes was
hard-killed and truncated (Sentry TAURI-RUST-K29: 1236 events / 71 users;
TAURI-RUST-8HB: 755 / 51). The child bounds its own lifetime via max_iterations,
the run cancellation token, and each inner tool's timeout, so override
timeout_policy -> Unbounded, matching spawn_parallel_agents (tinyhumansai#4686) and the
long-running scripting tools. Adds a unit test asserting the override.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Built-in agents, prompts, orchestration, and agent runtime in src/openhuman/agent/.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[agent][perf] spawn_parallel_agents runs workers sequentially + truncates each at a 120s cap

2 participants