Durable stream persists a full message snapshot per token delta — stored size is O(n²), measured 99.56% redundant in production

[mechiland]

What problem are you trying to solve?

Every message.appended event carries the full accumulated message in messageSoFar alongside the delta, and every event is persisted as a durable workflow stream chunk — so a message's stored footprint is O(n²) in its number of deltas. reasoningSoFar doubles this on reasoning models.

• packages/eve/src/harness/emission.ts:554 — messageSoFar: currentMessage (as of eve@0.29.2)
• packages/eve/src/harness/emission.ts:529 — reasoningSoFar: currentReasoning
• MessageAppendedStreamEvent has carried messageSoFar since the first commit, so this is protocol design rather than a regression.

This is the storage-side counterpart of #1051, which reports the read-side symptom (rebuilding history re-downloads every partial copy) and proposes a history endpoint. A history endpoint helps readers, but the quadratic bytes still land on disk.

Measured impact (production deployment, postgres world)

We run eve agents on @workflow/world-postgres, one database per project:

• One moderately busy agent (~300–450 runs/day) grew its workflow_stream_chunks table to 13 GB in two weeks (~2.5 GB/day) and took the host disk to 85%.
• One representative run: 8,110 chunks, 153 MB written for a final assistant message of 21 KB — ~7,000× amplification. Per-chunk size grows linearly to the end of the run (2.2 KB in the first decile → 37 KB in the last).
• Across the whole database: 18 GB of chunk bytes, of which the sum of each stream's largest chunk — the only bytes carrying new information — is 81 MB. 99.56% of persisted bytes are redundant copies.

The snapshots are not needed for correctness: packages/eve/src/client/message-reducer.ts already accumulates deltas client-side, and since durable sessions moved program memory into workflow step results (packages/eve/src/execution/durable-session-store.ts), the chunk stream is delivery-only.

Proposed solution

Stop persisting the accumulated snapshots on per-delta events; emit them only on terminal events:

1. message.appended / reasoning.appended carry the delta only (drop or empty messageSoFar / reasoningSoFar).
2. message.completed / reasoning.completed keep the full final text (they already do).

Live consumers reduce deltas as they already do; a resuming consumer replays from its cursor or rebuilds from the *.completed events. This turns storage O(n²) → O(n) at the source — on our data it would shrink the chunk table by ~99% — and it composes with #1051's history endpoint rather than replacing it.

If dropping the fields outright is too breaking for one release, an opt-in (e.g. an emission/agent-level streamSnapshots: false, or making the fields optional in the protocol and omitting them by default in a future major) would let deployments that pay for storage opt out immediately.

Happy to contribute a PR if the direction is agreeable.

[NicoMoli]

Hello @AndrewBarba
cc @ruiconti
Can I take this issue? Here I have some ideas that probably need your validation?

Two possible directions:

1. Emit delta-only appends with attempt metadata, bump the stream version, and update consumers. Given eve's pre-1.0 stance, this appears to be the cleaner option.
2. Persist a compact representation containing deltas and attempt metadata, then rehydrate cumulative snapshots in execution/workflow-runtime.ts. This preserves the public stream shape, but an arbitrary startIndex would require scanning a earlier prefix, and legacy full-snapshot chunks would need to coexist with the compact representation.

[Benny-Gil]

Corroborating this from a second, unrelated deployment — same shape, same metric, slightly worse ratio.

Environment: eve@0.47.6, @workflow/world-postgres@5.0.0-beta.34, Neon Postgres.

Applying the metric from the original report — total chunk bytes versus the sum of each stream's largest chunk, the only bytes carrying new information:

	This deployment	Original report
Chunk bytes stored	2,518 MB	18 GB
Bytes carrying new information	1,865 kB	81 MB
Redundant	99.93%	99.56%

Table state at measurement: 438,231 chunks across 63 streams, workflow_stream_chunks at 2,716 MB of a 3,063 MB database — 89% of everything.

Worst single conversation: 81,085 chunks totalling 953 MB, written in 48 minutes. Mean chunk 6.4 kB, largest single chunk 591 kB.

I confirmed the messageSoFar mechanism independently before finding this issue, by reading chunk sizes in ordinal order without decoding payloads. Within one reply consecutive chunks grow monotonically by 4–8 bytes, then reset at the next message:

n=200  1305 bytes
n=210  1341   (+4, +8, +8, 0, +4 …)
n=240  1457
n=260  1541

Worth noting for prioritisation that the failure mode is not graceful: once a managed Postgres hits its size limit it also refuses the DELETE that would recover the space, so the remedy is blocked by the condition it treats. Reaching the cap is effectively unrecoverable without raising the limit or truncating.

One question, in case it is a cheaper interim than the protocol change. The serialization layer already carries gzip and zstd alongside DEVALUE_V1, but compression looks opt-in (compression === true) and every chunk we have on disk is uncompressed devl. Cumulative snapshots are highly self-similar, so compression should recover a large constant factor without touching the stream shape or consumers. Is enabling it for durable session streams viable ahead of the delta work in the comment above, or is there a reason it is off for this path?

Happy to supply more measurements if useful.

[Benny-Gil]

Following up on my earlier comment: this deployment is now on the 0.52.x line,
so the delta-only appends are in effect. The change lands exactly as the issue
predicted.

Same database, comparing chunks written before and after the upgrade:

	mean chunk	largest chunk
before	2,956 B	47,893 B
after	367 B	1,229 B

An 8x drop in the mean and a 39x drop in the largest chunk. The signature I
described before is gone with it: chunk size no longer climbs monotonically
through a reply and then resets at the next message, because each chunk now
carries only its own delta.

Two notes for anyone finding this later. The reduction applies to newly written
chunks only — the quadratic bytes already on disk stay until something deletes
them, and nothing in the world contract does. Streamer.streams exposes
write, writeMulti, close, get, list, getChunks and getInfo, with
no delete verb, and retention exists in world-postgres for hooks alone via
capabilities.hookRetention and WORKFLOW_POSTGRES_HOOK_RETENTION_LIMIT_DAYS.
So the growth rate is fixed while the ceiling still is not.

My question above about enabling compression for durable session streams is
also still open, though it matters much less now that the redundancy is gone.

[mechiland]

Thanks for the before/after numbers — good to see #2867 land as predicted. Since we ran the same failure end to end on the original deployment, a few operational notes for whoever finds this later, on the two things your comment leaves open: the ceiling, and compression.

1. The ceiling has to be enforced outside eve, and it is cheap.

Once a run is terminal, its chunks are delivery-only — session memory lives in workflow step results (durable-session-store.ts), not in the stream. So a sweep that deletes the non-EOF chunks of completed/failed/cancelled runs older than a resume window is safe. We have run this in production since 2026-07-31 (24 h window, batched DELETE ... WHERE ctid IN (... LIMIT 50000) joined through run_id), and it holds the table flat. It is a maintenance job against the database, not a Streamer verb, which is why the World contract does not need one.

Two gotchas from doing it:

• Never delete the eof = true marker row. The streamer reads data rows with eof = false and decides termination through a separate eof = true lookup. Delete the marker and a late reader waits forever instead of ending cleanly. Filter on eof = false.
• DELETE does not give the disk back. The pages become reusable, but relation files do not shrink, and VACUUM FULL / pg_repack need roughly the table's size in free space — which is exactly what you do not have when this bites. For the backlog written before the upgrade, the first reclaim has to be a rewrite: copy the rows you keep (EOF markers plus chunks of non-terminal / recent runs) to a side table, TRUNCATE, re-insert. After that, steady-state DELETE + autovacuum is enough. This is also why "the managed instance refuses the DELETE at the cap" is less of a loss than it looks: the DELETE would not have recovered space anyway.

2. On compression as an interim.

We went a different way and I'd suggest it over compression for anyone still on a cumulative-snapshot line. Chunk-level gzip/zstd compresses each chunk on its own, but the redundancy here is across chunks (chunk n+1 = chunk n + a few bytes), and a per-chunk codec cannot see it. You recover the base64 overhead and normal text entropy — a constant factor, maybe 3–5× — and storage stays O(n²).

What removes the quadratic term without touching eve is stripping at the World layer: on streams.write, parse the frame, drop messageSoFar / reasoningSoFar when they equal accumulator + delta, and rehydrate the exact wire bytes on read, with a checkpoint every N chunks so a cursor resume does not replay from the start. We shipped that in @evelandhq/workflow-world (evelandhq/workflow-world#16, mid-August) and it is transparent to every eve stream version, old or new. With v25 it is mostly a no-op on new writes, but it still handles the tail of older lines.

One more thing that matters after v25: with ~370 B mean chunks, per-row overhead (tuple header, index entries, NOTIFY per chunk) starts to dominate the bytes. Packing multiple logical chunks into one physical row while keeping logical chunk ids stable for cursors / startIndex / tailIndex was worth it for us on top of the delta change.

Full write-up of the original incident, with the queries and the sizing, is in evelandhq/eveland#213.

[mechiland]

Closing this: the eve side of the problem is resolved.

• Source fix: fix(eve): persist text streams as deltas #2867 (stream v25, in 0.50+) persists message.appended / reasoning.appended / action.input.appended as deltas only. The PR's own measurement is -99.25% serialized bytes; @Benny-Gil's production numbers above confirm it.
• The remaining "ceiling" (nothing deletes terminal runs' chunks) is a World concern, and vercel/workflow is already moving on it. [core] Add a retention option to start() workflow#3787 (merged 2026-09-08) adds a retention option to start() that Worlds read at terminal transition, and world-postgres ships a matching purge that blanks stream chunks without breaking EOF. Only retention: 0 (delete immediately) exists so far; time-based windows are explicitly deferred until the unit is decided, and Add an atomic run-tree purge capability for retention workflows workflow#3114 covers whole-run-tree purge. If you need the bounded window, that is the place to push.

Until that lands, the options are: wait for it, run your own sweep against the database as described in my previous comment, or — if you are in a hurry — @evelandhq/workflow-world already does write-side snapshot stripping, chunk packing and deadline-driven terminal retention out of the box :-)

Thanks everyone for the measurements.