eve: compaction is triggered on reported tokens, declined on a character estimate, and reported as completed
A self-contained reproduction against eve 0.39.0, unmodified from npm. No API key, no network, no cost.
Reproduced on 0.38.3 and 0.39.0; the relevant code is unchanged between them.
dist/src/harness/compaction.js, token-estimate.js, compaction-prompt.js and tool-loop.js
are byte-identical in the two published packages.
npm install
npm test # the same assertions, red on stock eve and green on an anchored work step
npm run simulate # what the behaviour costs over one 42-turn session
node repro.mjs # harness-level: real turns, real event stream
node repro-unit.mjs # the same thing at the seam, in 40 lines
node repro-dense.mjs # the same thing on token counts measured by a real tokenizer
Each repro*.mjs script exits 1 when it reproduces the behaviour.
eve decides whether to compact and what compaction does using two different measurements of the same session, and the second silently overrules the first.
shouldCompactusesgetInputTokenCount, which prefers the input-token count the provider reported on the last model call (config.lastKnownInputTokens).compactMessagesre-checks the size withestimateTokens, which isJSON.stringify(messages).length / 4.
When the reported count is over the threshold and the character estimate is under it, the
tool-result-cap heuristic inside compactMessages concludes the session is already within limit and
returns the same message list. The harness then emits compaction.completed.
Nothing shrinks. The next turn repeats the whole sequence, and the one after that, while the real context keeps growing. There is no terminal state and no event that distinguishes this from a successful compaction.
The same disagreement has a second, more expensive form. When the session does hold an oversized tool result, the cap heuristic truncates a few hundred characters, satisfies the character estimate, and accepts. That is a real edit to an early message, so the prompt cache is invalidated from there on — every turn, while the real context still grows. The cost simulation below measures it: nine consecutive turns, 1.4 million invalidated tokens, and a session 27% past the threshold at the end of them.
Expected — a session that shouldCompact judged over-threshold is compacted; or, if the work
step decides there is nothing it can do, the harness says so.
Actual — the work step returns the input untouched, the summariser is never called, and
compaction.completed is emitted anyway. Every subsequent turn does the same.
Scenario A is the defect. Scenario B is a control in which both measurements agree, so compaction does real work — which shows the harness and the mock model in this repository are sound, and isolates the disagreement between the two measurements as the cause.
A. DEFECT — reported count over the threshold, character estimate under it
turn messages before messages after reported tokens estimate (chars/4) trigger fires compaction events model calls
1 20 22 (none yet) 7195 false (none) 1
2 22 24 30000 7222 true compaction.requested → compaction.completed 1
3 24 26 30000 7248 true compaction.requested → compaction.completed 1
4 26 28 30000 7275 true compaction.requested → compaction.completed 1
B. CONTROL — both measurements over the threshold
turn messages before messages after reported tokens estimate (chars/4) trigger fires compaction events model calls
1 20 7 (none yet) 42195 true compaction.requested → compaction.completed 2
2 7 9 200000 6385 true compaction.requested → compaction.completed 1
Read the model calls column: two calls on the control's first turn — the turn plus the compaction
summary. One call on every defect turn: the summariser never ran.
npm test runs one suite — test/honesty.test.mjs — twice. Once against eve's own
compactMessages, and once against anchored-compaction.mjs, a 5-line wrapper in this
repository. The assertions are identical in both runs. Nothing else changes.
npm test # both targets, labelled
npm run test:stock # stock eve alone
npm run test:fixed # the anchored work step alone
Red on stock, green on the wrapper:
target: stock (eve 0.39.0 compactMessages)
✔ the trigger fires on the real input-token count while the character estimate is under the threshold
✖ the work step returns a smaller session than it was given
✖ the summariser runs when the session cannot be reduced by capping tool results
✖ every reported compaction shrinks the session
✖ the session stays within the threshold once compaction has reported success
✔ the loop driver reproduces eve's own tool-loop harness
ℹ pass 2
ℹ fail 4
✖ the work step returns a smaller session than it was given
compaction was asked to reduce a 17749-token session and returned 17758 tokens (80 messages in, 81 out)
✖ the summariser runs when the session cannot be reduced by capping tool results
the compaction summariser was called 0 times; nothing else in this session could have removed
the 17749 tokens the trigger objected to
✖ every reported compaction shrinks the session
4 of 5 turns emitted compaction.completed without removing anything
+ [ 'turn 19: 16224 → 16224 tokens', 'turn 20: 17115 → 17115 tokens',
+ 'turn 21: 18005 → 18005 tokens', 'turn 22: 18897 → 18897 tokens' ]
✖ the session stays within the threshold once compaction has reported success
after compaction first reported success on turn 19, 4 of 6 later prompts were still over the
threshold — the loop repeats and the session keeps growing
npm test exits non-zero only when the fixed target fails. A red stock run is the
reproduction, not a failure of the repository. npm run test:stock on its own exits 1,
because that is what a red suite does.
Two details in that output are worth reading twice. The work step returned one message more
than it was given — withResumptionGuard appends a resumption message, so a "compaction" can
end larger than its input. And turns 19 to 22 are four consecutive turns on which the trigger
fired, compaction.completed was emitted, and the token count afterwards is the token count
before, to the token.
Each one states something a caller of compaction is entitled to assume. None of them names the estimator, the trigger, or any internal function.
| assertion | stock | anchored |
|---|---|---|
| the trigger fires on the real count while the estimate is under the threshold | pass | pass |
| the work step returns a smaller session than it was given | fail | pass |
| the summariser runs when tool-result capping cannot reduce the session | fail | pass |
| every reported compaction shrinks the session | fail | pass |
| the session stays within the threshold once compaction has reported success | fail | pass |
| the loop driver reproduces eve's own tool-loop harness | pass | pass |
The last row is about the test harness, not about either target. eve's createToolLoopHarness
always calls compactMessages and offers no injection point, so loop-driver.mjs reproduces
the loop around compaction — maybeCompact, the model call, createNextCompactionConfig — and
that assertion drives the same sequence through the driver and through the real, unmodified
harness and requires them to agree turn by turn.
anchored-compaction.mjs is a wrapper, not a rewrite. It changes no measurement, keeps the
estimator, and hands the work to the same compactMessages:
export async function compactMessagesAnchored(messages, model, config, ...rest) {
const gap = getInputTokenCount(messages, config) - estimateTokens(messages);
const anchored = { ...config, threshold: config.threshold - gap };
return compactMessages(messages, model, anchored, ...rest);
}The gap between the two rulers is a property of this session's content. Subtract it from the threshold and the work step's comparison, still made in estimate-space, means the same thing as a comparison made in real-token space: score a candidate at the real count minus the saving the candidate makes. The fixed per-call overhead cancels, so a candidate that changed nothing — saving zero — is accepted only when the real count is under the threshold, which is exactly the case in which the trigger would not have fired.
It is a demonstration of the anchoring, not a proposed patch. Upstream would want the same
arithmetic inside compactMessages, which has the config in hand already.
All four functions are in dist/src/harness/compaction.js; the emission is in
dist/src/harness/tool-loop.js. (The published dist is minified; the names below are the ones the
module exports or that appear in the type declarations.)
-
The trigger reads the provider's count.
shouldCompact(messages, config)returnsgetInputTokenCount(messages, config) + COMPACTION_PROMPT_OVERHEAD_TOKENS > config.threshold.getInputTokenCountreturnsconfig.lastKnownInputTokensplus an estimate of the messages appended since that call, and falls back to the estimate only when no count has been recorded.config.lastKnownInputTokensis written bycreateNextCompactionConfigintool-loop.js, fromresult.usage.inputTokens. -
The work step re-reads the character estimate.
compactMessagesrunstoolResultCapHeuristicfirst. That heuristic builds a candidate — older tool results capped, recent messages verbatim — and asksevaluateThreshold(candidate, config, "should-compact"), which computesestimateTokens(candidate) + overheadand compares it withconfig.threshold.estimateTokensisJSON.stringify(messages).length / 4. -
So the heuristic can accept a candidate that changed nothing.
capToolResultsonly truncates tool results whose serialized payload exceedsTRANSCRIPT_PAYLOAD_LIMIT. With no oversized tool result, the candidate is the input. If the character estimate is under the threshold, the heuristic answerswithin-limitandcompactMessagesreturns that candidate. -
The harness reports success regardless.
maybeCompactemitscompaction.requested, callscompactMessages, and then emitscompaction.completed. There is no branch for "the work step returned what it was given", so a no-op is indistinguishable from a compaction that summarised half the session.
repro-dense.mjs measures it, with js-tiktoken (cl100k_base, pure JavaScript, no network):
| content | real tokens | estimate (chars/4) | the estimate reads |
|---|---|---|---|
| English prose | 201 | 233 | 1.16× the truth |
| base64 payload | 500 | 183 | 0.37× the truth |
| minified JSON | 760 | 438 | 0.58× the truth |
| CJK text | 510 | 135 | 0.26× the truth |
| emoji | 1080 | 188 | 0.17× the truth |
Four characters per token is a reasonable rule for English prose, which is the one row where the estimate is safely high. It is not a rule for what an agent session actually accumulates: tool output, source code, logs, structured payloads, and non-Latin text. Tokenizers also differ between vendors, so no single character ratio is right for all of them.
The direction is what matters here. On dense content the estimate reads low, by a multiple rather than a margin — so a threshold check built on it fires late, or never.
repro-dense.mjs then drives the harness using the count the tokenizer actually produced, so no
number in that script is chosen by us: 24 messages of CJK text measure 12,360 real tokens against a
3,301 character estimate, and the same no-op loop follows.
npm run simulate grows one session — mixed prose, code, logs and tool-call JSON, measured
with js-tiktoken — through 42 turns against a 150,000-token threshold, and drives the same
growth sequence through both implementations. The mock provider reports the measured counts as
its usage.input_tokens, so the trigger behaves as it would in production. Prompt caching is
prefix-matched, so for each turn the simulation finds the first message index at which this
turn's prompt differs from the previous one, treats everything before it as a cache read and
everything after it as a cache write, and calls it a cache bust when the divergence lands
inside the previous prompt. Cache read is priced at 0.1× base input and cache write at 1.25×,
which are Anthropic's published multipliers; compaction summary transcripts are priced at 1.0×
because they have no cached prefix to match.
The interesting turns are 31 to 37 of the stock run. The session carries a few oversized tool
results, so capToolResults can truncate something on every pass — enough to satisfy the
character estimate, never enough to matter. Each turn rewrites message 10, invalidates the
~150,000 cached tokens after it, saves a few hundred, reports compaction.completed, and hands
back a session larger than the turn before:
STOCK — eve's own compactMessages
turn context (real) estimate (chars/4) compaction reported work step first changed msg cache bust tokens invalidated
30 146,555 111,595 no — 113
31 151,455 115,421 yes capped 10 YES 134,963
32 156,385 119,301 yes capped 10 YES 139,863
33 161,342 123,228 yes capped 10 YES 144,793
34 166,313 127,182 yes capped 10 YES 149,750
35 171,276 131,104 yes capped 10 YES 154,721
36 176,223 134,965 yes capped 10 YES 159,684
37 181,045 138,650 yes capped 10 YES 164,631
38 185,966 142,411 yes no-op 145
39 190,889 146,176 yes no-op 149
40 8,223 6,258 yes summarised 0 YES 190,889
FIXED — the same compactMessages, work step anchored to the trigger
turn context (real) estimate (chars/4) compaction reported work step first changed msg cache bust tokens invalidated
30 146,555 111,595 no — 113
31 8,243 6,257 yes summarised 0 YES 146,555
32 13,144 10,020 no — 8
33 18,056 13,782 no — 12
34 22,958 17,539 no — 16
Turn 40 is the end of it: the character estimate finally crosses the threshold too, the two measurements agree again, and the session is summarised — nine turns and 1.2 million invalidated tokens after the trigger first asked for it.
Totals over the run:
metric stock anchored
compactions reported 11 2
of which no-ops 2 0
summary passes 1 1
cache busts 9 2
tokens invalidated 1,377,001 284,262
tokens written to cache 1,395,047 346,530
tokens read from cache 2,722,367 2,613,651
summary transcript tokens 183,429 140,114
turns sent over the threshold 9 0
largest prompt sent 190,889 149,299
final context 18,046 62,268
base-input-equivalent tokens 2,199,474 834,642
approx input cost @ $3.00/Mtok $6.60 $2.50
Read final context beside the turn tables, not on its own: the stock run ends smaller only
because it summarised at turn 40, two turns before the end. turns sent over the threshold and
largest prompt sent describe the whole run — the stock session spent nine turns over the limit
it was configured to respect, and reached 27% past it.
Both runs also compact for real at turn 28, where capping genuinely brings the session under the threshold. The anchored version accepts that cheap reduction too. It is not "always summarise"; it summarises when nothing cheaper is enough.
Output tokens are excluded, and the price is one rate applied to both runs, so the ratio is the durable figure rather than the dollar amount.
The estimator is not the problem, and removing it is not the fix we would propose. A provider-reported count exists only per model call, it is stale by whatever has been appended since, and the per-message budgeting inside compaction — how much of the recent window to keep, what transcript budget to give the summariser — has no provider count available at all. A synchronous estimate is the right tool for those jobs.
The staleness also runs in the safe direction for the gate: a stale count under-reports the current context, so a trigger that read over-threshold is genuinely over.
Two things do look wrong:
-
The composition. The work step re-decides a question its caller already decided, using a weaker measurement, and can silently overrule it. Carrying the trigger's figure into the work step — or having the work step accept its caller's decision and only choose how to compact — would remove the class.
anchored-compaction.mjsdoes the first of those from outside, in five lines, and turns every red assertion in the suite green while leaving the estimator in place. -
The event contract.
compaction.completedis emitted for a run that did nothing. Whatever is decided about (1), a no-op that repeats every turn should be observable: a distinct event, a field on the completed event, or a log line. Today the only symptom is the session growing until the model call fails.
mock-model.mjs is an ordinary AI SDK LanguageModel implementation (v4 specification) with no
network calls. The only thing it does that a real provider does not is let the caller choose the
usage figure it reports.
That is not how the behaviour is manufactured — it is how it is made deterministic and free. eve
stores that reported figure on the session itself and uses it for the trigger, exactly as it would
with a real provider, and repro-dense.mjs shows the same result using counts a real tokenizer
produced rather than counts we chose. To reproduce against a live provider instead, replace
resolveModel: async () => model in repro.mjs with your own provider instance and give the
session dense content; the script reads no credentials and ships none.
- eve 0.39.0, installed straight from npm, with no patches applied. Also reproduced on 0.38.3;
the four
distfiles involved are byte-identical in the two packages. - Node 24 or newer (eve 0.39.0 requires it).
js-tiktokenis a dev dependency. It measures the real token counts used byrepro-dense.mjs, the test suite and the cost simulation.
| file | what it is |
|---|---|
repro.mjs |
harness-level reproduction: real turns through createToolLoopHarness |
repro-unit.mjs |
the same thing at the seam, in 40 lines |
repro-dense.mjs |
the same thing on token counts measured by a real tokenizer |
test/honesty.test.mjs |
the assertion suite, run against both implementations |
run-tests.mjs |
runs the suite against both and labels the result (npm test) |
anchored-compaction.mjs |
the alternate implementation — a wrapper, not a rewrite |
simulate.mjs |
the 42-turn cost simulation (npm run simulate) |
loop-driver.mjs |
eve's turn loop, with the compaction function injectable |
session-fixture.mjs |
one realistic session, and the tokenizer that measures it |
mock-model.mjs |
a local AI SDK LanguageModel, no network and no key |
eve-internals.mjs |
the eve functions under test, loaded from the installed package |
Running any script prints one warning from eve —
setEveAttributes() must be called from a 'use workflow' or 'use step' function — because the
harness is driven directly rather than from inside a workflow. It is emitted once and does not
affect the behaviour shown.