Complete picture
Issue #48488 — Lane queue has no task-level timeout — hung promises permanently block session lanes
• Filed 2 months ago by kyletabor, well-analyzed
• Their log evidence matches ours exactly
• Root cause pinpointed: pump() in src/process/command-queue.ts awaits each task with no timeout
• Currently marked P2, stale-flagged, but not-repro-on-main label is aspirational — the bug clearly still affects 2026.5.7
PR #48690 — proposed fix (author: same kyletabor)
• Open since March 17, 2026 — 2+ months
• CONFLICTING (needs rebase)
• ClawSweeper bot: "needs real behavior proof before merge"
• Last updated yesterday (May 19) — author is active
• Fix: 5-min Promise.race timeout in pump() so lanes auto-recover after 5min
• Not in any released version yet (we're on 2026.5.7, latest is 2026.5.18)
What this changes about our plan
┌──────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┐
│ Approach │ Verdict │
├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ File a new openclaw issue │ ❌ Skip — #48488 already exists with better analysis │
│ Upgrade to openclaw 2026.5.18 │ 🟡 Worth doing for unrelated improvements, but won't fix the bug (PR #48690 not merged │
│ │ yet) │
│ Wait for PR #48690 to merge │ ❌ It's been stalled 2 months on rebase + "needs proof"; even after merge it's a 5-min │
│ │ recovery, not a cure │
│ Implement Option A (plugin │ ✅ Still the right move — prevents the trigger, works today, resilient even when #48690 │
│ serialization) │ lands │
│ Provide repro evidence on #48488 │ ✅ Quick win — gives ClawSweeper the "real behavior proof" they're asking for; may │
│ │ unstick the PR │
└──────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┘
Option A — Plugin-side serialization (we own this — ~30 LOC)
Replace fire-and-forget with a per-agent promise chain so only one dispatch is in flight per agent at a time. Sidesteps
openclaw's broken queue entirely:
// channel/dispatch.ts
const _dispatchChains = new Map<string, Promise<void>>();
export function enqueueDispatch(
agentId: string,
fn: () => Promise<void>,
log: Logger,
): Promise<void> {
const prev = _dispatchChains.get(agentId) ?? Promise.resolve();
const next = prev.catch(() => {}).then(async () => {
try { await fn(); }
catch (e) { log.warn(`[mycelium-room] dispatch chain error for ${agentId}: ${e}`); }
});
_dispatchChains.set(agentId, next);
// Best-effort prune to avoid unbounded chain growth
next.finally(() => {
if (_dispatchChains.get(agentId) === next) _dispatchChains.delete(agentId);
});
return next;
}
Then in channel/index.ts:
void enqueueDispatch(action.agentId, () =>
dispatchToAgent(runtime, cfg, action.agentId, action.sender,
action.content, action.messageId, log), log);
Pros: localized to one file we own, no openclaw changes, fixes today, easy to test
Cons: doesn't fix openclaw for everyone else; slightly slower per agent (which is desired anyway)
Excellent question — here's the answer with hard data. Even though pytest runs sequentially, within a single test's
coordination session, the CFN backend generates bursts of events to the same agent faster than the LLM can process them.
Look at the actual timeline from test_41:
15:14:51.644 dispatch #1 (tick round 5) ◄─ openclaw starts processing
15:15:07.229 dispatch #2 (tick round 6) ◄─ queued behind #1
15:15:08.883 before_agent_start (round 5 picked up) ◄─ LLM call begins for #1
15:15:18.384 dispatch #3 (tick round 7) ◄─ queued behind #1 (LLM still running)
15:15:28.295 dispatch #4 (consensus) ◄─ queued behind #1 (LLM still running)
[~15:15:34] round 5's LLM call finishes ◄─ task #1 completes
── openclaw lane is now "idle" with q=3 — and never advances ──
Why this happens within ONE test:
A negotiation session goes through many rounds (round 5 → 6 → 7 → consensus). For each round, the CFN backend posts a
coordination_tick to the session room, addressed to whichever agent's turn it is. When the round completes, it posts
coordination_consensus to everyone, plus a notify-home for return-channel delivery.
Meanwhile, the LLM call for each turn takes 10-30 seconds (LLM thinking + tool calls + reply). So during that single-agent
processing window:
• 2-4 more rounds may have finished on the other agents
• Each generates a tick addressed to our agent (since they want our response)
• Plus the eventual consensus + notify-home
All of these hit void dispatchToAgent(...) (fire-and-forget) → they pile up in openclaw's per-agent lane queue → openclaw
queues them → the bug triggers.
So the trigger requires three ingredients, all present in normal coordination:
1. Slow tasks (LLM calls — naturally 10-30s)
2. Bursty events to one agent (negotiation rounds and consensus naturally do this)
3. Plugin doesn't await dispatches (void dispatchToAgent is fire-and-forget by design — it relies on openclaw's lane to
serialize)
It's actually unavoidable in any plausible coordination test. Even a simple "two-agent negotiation that takes 5 rounds"
will trigger it.
Why Option A fixes it cleanly:
With per-agent promise-chain serialization in the plugin:
• Plugin holds events #2, #3, #4 in its own internal chain
• Only one dispatch is in flight to openclaw at a time
• Openclaw's lane never has q > 1 queued items
• The buggy queue-advancement path is never exercised
• All 5 events still get delivered, just one-at-a-time
Worth noting: our wedge symptom (last=run:completed, agent idle, queue full) is slightly different from #48488's stated
cause (task promise never settles). Ours says the task did complete — but pump() didn't fire afterward. So we might
actually be hitting a related-but-distinct race in the queue's generation/draining logic (see closed PRs #33979, #27433,
#15178 which all touched that area). Worth mentioning in the comment to #48488 when we file evidence.
This also explains why last night's full e2e run wedged hard (every distributed test had multi-round negotiations) and why
today's restart-then-rerun still failed half (5 of the 10 distributed tests hit the trigger). And why simple smoke tests
(single dispatch) never reproduce it.
Want me to proceed with Option A now?
Excellent forensic data. CFN is NOT firing ticks too quickly — it's perfectly well-behaved. Here's the reconstructed timeline
for session 7477c84e:
┌───────┬──────────┬──────────┬─────────────────┬─────────────────┬─────────┐
│ Round │ Started │ Duration │ alpha response │ claire response │ Outcome │
├───────┼──────────┼──────────┼─────────────────┼─────────────────┼─────────┤
│ 0 │ 15:13:48 │ 25.9s │ 8.9s (accept) │ 25.7s (counter) │ ongoing │
│ 1 │ 15:14:14 │ 15.1s │ 14.9s (counter) │ 7.9s (accept) │ ongoing │
│ 2 │ 15:14:29 │ 11.0s │ 9.2s (accept) │ 10.9s (counter) │ ongoing │
│ 3 │ 15:14:40 │ 11.9s │ 11.7s (counter) │ 8.4s (accept) │ ongoing │
│ 4 │ 15:14:52 │ 15.6s │ 9.0s (accept) │ 15.4s (counter) │ ongoing │
│ 5 │ 15:15:07 │ 11.1s │ 11.0s (counter) │ 9.2s (accept) │ ongoing │
│ 6 │ 15:15:18 │ 9.9s │ 9.3s (accept) │ 6.7s (accept) │ agreed │
└───────┴──────────┴──────────┴─────────────────┴─────────────────┴─────────┘
Key observations:
1. CFN waits for decision_path: all_replied every round — never times out, never fires the next round prematurely. It's
gating on actual agent responses.
2. Round cadence is set by the slowest agent — averaging ~14 seconds per round. That's the agents' LLM latency, not CFN
policy.
3. Yesterday's failures were sessions stuck at round 0 with decision_path: aborted, elapsed_ms: 300000,
last_reply_received_ms: null — agents got the first tick, wedged immediately, and CFN's 5-minute budget expired. So the
bug killed sessions, not the other way around.
4. Today's session 7477c84e completed normally (agreed at round 6) — but during its lifetime, ticks were arriving in
openclaw's queue every ~10-15s. If alpha's task processing extended past the next tick's arrival, queue depth grew → bug
triggered for the next test's first message.
So why does openclaw's queue still wedge despite well-behaved CFN?
It's the collision of two natural cadences:
CFN round cadence: every ~10-15s (waits for all_replied)
Agent task cadence: every ~10-30s (LLM thinking + tool calls + reply)
When agent task duration > round duration (even briefly), the agent's local queue accumulates. CFN doesn't know openclaw is
still processing the previous turn — it just sees the reply was posted (which happens mid-task) and moves on. The next tick
arrives while openclaw's pump() is still finishing the prior task.
Round 6 specifically lined up perfectly to trigger the wedge:
• Round 5 ends at 15:15:18 → tick for round 6 queued for alpha
• Consensus reached at 15:15:28 → 🤝 → alpha queued (only ~10s after round 6's tick)
• notify-home at 15:15:29 queued (1s after consensus)
• alpha's earlier task is still finishing → 3 items queued behind it
• Earlier task completes → openclaw should pump → doesn't → wedge
Implications
• ✅ Don't change CFN — its cadence is correct. Throttling it would slow legitimate negotiations.
• ✅ Don't change agent prompts — task duration is dominated by LLM latency, not prompt structure.
• ✅ Option A is still the right fix — plugin-side serialization makes the openclaw queue depth always ≤ 1, sidestepping
the bug regardless of arrival cadence.
There's one tangential observation worth noting:
• Consensus dispatch is a separate LLM turn for the agent. When agents have already voted "accept" (round 6: both accept),
maybe they don't need to LLM-process the consensus event itself — it's just notification. Skipping the consensus dispatch
when an agent already voted accept in the final round would eliminate one of the 3 events that triggers the wedge. But
this is a perf optimization, not a fix — Option A solves it more cleanly.
Pasting Selina's agent context here for safe keeping: