[Bug]: Cumulative tool-output snapshots amplify queued memory with slow consumers (256 KiB -> 96 MiB)

[salim4n]

Summary

The direct-CLI event pump turns each ToolOutputDelta into a full cumulative output snapshot (acc.clone()) and publishes it into a broadcast channel with capacity 512 events, with no byte budget on this path.

When a subscriber falls behind, many independently allocated versions of the same growing output remain queued. A small isolated reproduction gives 256 KiB of final output -> 96 MiB of retained snapshot payload (384.25x). The queue is bounded in event count, but its byte footprint can still become very large.

We investigated this after a Linux WebUI deployment became unresponsive under cgroup memory pressure. The production evidence correlates a large command output with relay lag and a multi-GiB AionCore footprint. The amplification mechanism is reproduced; the exact allocation breakdown of the former production process is not proven, because it was restarted before collecting a heap profile.

Environment and scope

• Linux VPS, approximately 8 GiB RAM; standalone/headless WebUI, no Electron.
• AionUi 2.1.61, bundled AionCore 0.1.72, using the direct Codex app-server backend and multi-agent Teams.
• Web host, AionCore and spawned agents/tools share one systemd service cgroup: MemoryHigh=5G, MemoryMax=6G.
• Inspected v0.1.72 source commit: 57a34cc1b1a3b17bcc023de06b9e6768fceac36f.
• Also compared source tag v0.2.0, commit 409966d7b9585006a0297ab0a001a4a32bcf1d6a: the relevant cumulative-output branch and capacities remain unchanged, and the source-extracted isolated replay gives the same result. v0.2.0 was not deployed or tested end-to-end.
• Binary/source reproducible-build equivalence was not tested.

Production observations

On 2026-08-30, an agent searched generated Next.js distribution files with rg ... node_modules/next/dist ... | head -100. Limiting the number of lines did not bound output size because some generated lines were extremely long.

• Persisted tool output: 5,087,006 characters, with only 32 newline characters.
• Longest individual line: 2,280,548 characters.
• Shortly afterward, the same conversation logged Stream relay lagged, some events dropped repeatedly; sampled dropped-event counts were 17–96 per warning.
• Local HTTP requests timed out after 5 seconds, public requests after 8 seconds, while systemd still reported the service as active.
• Main AionCore RSS reached 3,737,496 KiB, later 4,027,352 KiB.
• Service memory exceeded its soft limit. A web-host thread was blocked in mem_cgroup_handle_over_high; memory pressure was approximately 100% some and 92% full.
• Captured cgroup counters included high=20783590, oom=0, oom_kill=0. This was reclaim/throttling, not an observed OOM kill.
• About 1.8 GiB remained available on the host; sampled CPU steal was only 2–3%.
• Raising the soft limit temporarily to 5.5 GiB allowed one local response, then the service reached that limit and stalled again.
• Restarting the service restored responsiveness. During subsequent inspection, main AionCore RSS was approximately 121 MiB with multiple CLI/MCP processes running, service memory approximately 757 MiB, API latency approximately 67 ms, and memory pressure zero.

The exact production chunk boundaries, number of snapshots retained, per-connection backlog and allocator overhead were not captured. These observations do not establish that every byte of the incident footprint came from this queue, nor that every long-running agent was leaked.

Relevant source path

1. The pump appends each delta and clones the whole accumulator into ToolCallEventData.output. The channel capacity is 512.
2. The relay forwards each tool event and awaits persistence. Persistence serializes and upserts the full payload, which can slow consumption as snapshots grow.
3. WebSocket broadcasts clone serialized messages for each user connection, with 64 messages per connection. This is an additional possible byte multiplier; its actual contribution during the incident is unmeasured.

Small, deterministic reproduction

This runs locally with synthetic data and at most approximately 96 MiB of retained output payload. It does not invoke an agent or contact a server.

It reproduces the equivalent cumulative-clone + Tokio broadcast allocation pattern, not the full application. We also ran an isolated harness that extracted the verbatim production branch, payload type and capacity from both source tags, with the same results. The public version below uses String directly to remove application dependencies; production wraps that string in ToolCallEventData.output.

Create a standalone Rust project with these files and run cargo run --quiet.

Cargo.toml

[package]
name = "cumulative-output-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "=1.52.3", features = ["sync"] }

src/main.rs

use tokio::sync::broadcast::{self, error::TryRecvError};

fn main() {
    for chunk_bytes in [64_usize, 256] {
        let chunks = 1024;
        let capacity = 512;
        let (tx, mut rx) = broadcast::channel::<String>(capacity);
        let mut accumulated = String::new();

        // Equivalent allocation pattern; the receiver deliberately falls behind.
        // Production wraps each string in ToolCallEventData.output.
        for _ in 0..chunks {
            accumulated.push_str(&"x".repeat(chunk_bytes));
            tx.send(accumulated.clone()).unwrap();
        }
        drop(tx);

        let (mut retained_bytes, mut frames, mut dropped) = (0, 0, 0);
        loop {
            match rx.try_recv() {
                Ok(snapshot) => {
                    retained_bytes += snapshot.len();
                    frames += 1;
                }
                Err(TryRecvError::Lagged(n)) => dropped += n,
                Err(TryRecvError::Closed | TryRecvError::Empty) => break,
            }
        }
        assert_eq!(frames, capacity);
        assert_eq!(dropped, 512);
        assert_eq!(retained_bytes, ((513..=1024).sum::<usize>()) * chunk_bytes);
        println!(
            "output_bytes={} retained_frames={} retained_payload_bytes={} amplification={:.2}x",
            accumulated.len(), frames, retained_bytes,
            retained_bytes as f64 / accumulated.len() as f64
        );
    }
}

Observed output (Tokio 1.52.3, matching the v0.1.72 lockfile):

output_bytes=65536 retained_frames=512 retained_payload_bytes=25182208 amplification=384.25x
output_bytes=262144 retained_frames=512 retained_payload_bytes=100728832 amplification=384.25x

This measures the sum of retained string payload lengths, not OS RSS. The receiver is deliberately paused, so this is a deterministic backlog scenario, not a claim that every normal stream has a 384x footprint. Only 512 snapshots remain: bounded message count alone does not imply a practical byte bound.

Expected behavior / possible fix direction

• Bound live-preview accumulation and queued data by bytes; retain full output as an on-demand artifact if needed.
• Coalesce superseded intermediate snapshots per tool instead of retaining every full replacement.
• Throttle intermediate persistence/UI updates while preserving final output, error/cancel state and terminal-event delivery.
• Add byte-aware WebSocket backpressure and regression coverage for a slow consumer, very long lines and multiple connections.
• Avoid merely increasing queue capacity: that can worsen memory use. Blindly shrinking the queue can also lose terminal events.

The shared cgroup explains why agent-side pressure can freeze the control plane too, but separating process budgets would only contain the impact; it would not remove this allocation pattern.

Related, not asserted duplicates

• [Bug]: WebUI Team mode: REALTIME_BACKPRESSURE drops user messages & status events — UI stuck until full restart AionUi#3832: headless Linux WebUI Team runs saturating outbound queues and requiring a restart. Similar symptoms/backpressure boundary; not proof of the same root cause.
• feat/retained output ui AionUi#4038: retained-output UI proposal, closed without merge when checked. Related preview/on-demand-output design direction.

No production heap profile or full end-to-end stress reproduction is attached. The report deliberately separates the measured incident, verified source behavior and isolated allocation reproduction.