eval: the memory question, measured to completion (L4+L5) — plus the driver fix and the honest README claim - #30
Merged
Merged
Conversation
Mechanism PROVEN through the real init-written stack: stored s1 + stored s3 + retrieved s4, every graded sage run. Behavior at 4 sessions/one workspace: bare ties or better (3/3 by rereading logs, $2.17/run); sage produced compliant trees in 2/3 runs but was voided by a driver bug, and spent the third's budget on ceremony before implementing. Three instrument iterations (checkpoint trap, conscientiousness trap, unpassable file-level check) — all recorded in the rationale + MEASURED.md. OPEN DRIVER BUG: error_max_budget_usd on a --resume turn is classified 'nothing ran / unknown error' and VOIDS a run carrying $6+ of committed work — violates the void-vs-truncated rule; test_driver does not pin this shape.
…ver fix L4's deepest finding: bare never read a log — the COMMITTED CODE carried every convention. Git is a memory system. So the memory layer's honest strong case is knowledge git cannot carry, and L5 constructs it: a release freeze closes the write-it-into-the-repo channel while a post-mortem rule (module name 'utils' banned; use support_*) is stated; session 2 runs in a FRESH CHECKOUT and extracts helpers into a new module 'named per the codebase's conventions'. Only memory crosses. First scenario where a sage win + bare loss is possible. Harness: session flag fresh_checkout (git checkout+clean -xdf, sage re-init — a teammate's clone) and scenario key memory_home: 'run' (pins SAGE_PROJECT_ROOT into the run's temp parent via .mcp.json env — run-isolated, survives the fresh checkout, never the developer's real ~/.sage-memory). Driver fix (the L4 v3 bug): error_max_budget_usd on a --resume turn with 0 tokens in ITS OWN usage no longer voids a session whose earlier turns did real work — SESSION tokens decide 'ran', so it grades as truncated+flagged. test_driver +1 (5 total), pinning the shape.
… claim rewritten
L5 verdict (freeze check removed as structurally invalid — session-scoped diff
graders only hold on the final session; compliance verified from transcripts):
sage 3/3, bare 3/3 — the rule survived the fresh checkout in BOTH arms, because
the 'bare' agent used Claude Code's own per-project auto-memory. Third regime,
third tie:
L2 the session log carried it
L4 the committed code carried it
L5 the platform's memory carried it (bare at 1/8th sage's cost)
README: 'Memory That Compounds' -> 'Persistent Memory' — mechanism measured and
stated (store/accumulate/retrieve/survive-checkout all proven through the real
init stack), behavioral edge honestly absent, the cross-PROJECT bet named as the
one unmeasured regime none of the alternatives serve. Top bullet updated to
match. eval-baseline: the L4/L5 arc appended to the memory section.
HARNESS FINDING recorded in the scenario: evals MUST isolate HOME — an L5 bare
run wrote the rule into the developer's real user-global ~/.claude/CLAUDE.md
(cleanup flagged to the owner; classifier rightly blocked automated edits there).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Memory That Compounds" claim, measured until it answered. Three commits: the L4 compounding scenario + verdict, the L5 fresh-checkout scenario + the harness it needed + a driver fix, and the docs rewrite the evidence demands.
The arc (all N=3, both arms, kept workspaces)
sage-memory's mechanism is proven end-to-end through the real
sage initstack — store, accumulate (multiple deposits), retrieve jointly, survive a fresh checkout — every mechanism check green in every run. A measured behavioral edge was not found, because a modern agent is never actually memoryless. The one unmeasured regime none of the alternatives serve: cross-project knowledge (all three alternatives are keyed to one repo/path). Named in the docs as the remaining bet.Harness work this needed
fresh_checkout: truesession flag (tracked files only; sage re-init like a teammate's clone) andmemory_home: "run"(memory pinned outside the workspace, run-isolated).error_max_budget_usdon a--resumeturn with 0 tokens in its own usage no longer voids a session whose earlier turns did real work — SESSION tokens decide "ran"; grades as truncated+flagged.test_driver+1 pins it.Docs
README: "Memory That Compounds" → "Persistent Memory" — mechanism stated as measured, behavioral edge stated as absent, the cross-project bet named. Top bullet matches. eval-baseline: the arc appended.
An L5 bare run wrote the rule into your real
~/.claude/CLAUDE.md(the harness doesn't isolate HOME — recorded as a required harness fix). Cleanup on your machine, at your discretion: restore~/.claude/CLAUDE.md(the eval-injected "Banned module name: utils" section) and optionallyrm -rf ~/.claude/projects/-tmp-sage-evals-*(283 junk dirs).fastcheck14/14 · offline-check 15 scenarios · driver tests 5/5.