diff --git a/README.md b/README.md index 60521bb..817e9ef 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Built for product and engineering teams, open to any domain. - **Think first, build second** — a framing round challenges assumptions before solutioning begins, preventing the most expensive mistake: solving the wrong problem - **Focus over noise** — loads only what the task needs, producing sharper reasoning - **Mechanical where it counts** — hooks that block a source edit until a test exists and an edit before a spec exists, and gate scripts with a three-state exit contract. These are code, not instructions, and they hold: test-first measures **3/3 against a bare agent's 0/3**. The prose layers around them are advice, and advice is rationalizable — see [What we measured](#what-we-measured) for which is which -- **Gets smarter over time** — self-learning, memory, and ontology compound into institutional knowledge of your codebase +- **Persistent memory built in** — self-learning, project memory, and an entity ontology, wired up at init; the mechanism is measured (see below), the compounding bet is stated honestly - **Grows with its ecosystem** — 12 focused core skills plus installable packs (product/UX, pack-authoring, autoresearch), extensible with 90K+ community skills from skills.sh ## What we measured @@ -140,22 +140,31 @@ no behaviour lost. Numbers and the full story: [docs/eval-baseline.md](docs/eval-baseline.md). -### Memory That Compounds +### Persistent Memory

Sage Memory System — 3 skills, 1 MCP, compounding knowledge.

-Most agent frameworks are stateless. The agent that made a mistake -yesterday makes it again today. Sage has three skills that build -institutional memory — all backed by sage-memory MCP: - -- **sage-self-learning** captures mistakes as WHEN/CHECK/BECAUSE prevention rules. Every session starts by searching past mistakes before doing anything. -- **sage-memory** stores project knowledge as focused prose insights — how your auth works, why billing uses event sourcing, what conventions the team follows. -- **sage-ontology** maps entity relationships — not just "billing exists" but "billing depends on payments, which triggers webhooks, which notify users." Touch one module, know the blast radius. - -Day 1, the agent knows nothing. Day 30, it knows your codebase's -landmines, patterns, and conventions. +Sage ships a persistent memory layer — three skills backed by the sage-memory +MCP, wired up automatically by `sage init` (opt out with `--no-memory`): + +- **sage-self-learning** captures mistakes as WHEN/CHECK/BECAUSE prevention rules. +- **sage-memory** stores project knowledge as focused prose insights — conventions, decisions, gotchas. +- **sage-ontology** maps entity relationships — touch one module, know the blast radius. + +**What's measured, honestly.** The mechanism is proven end-to-end: knowledge +stored in one session is retrieved and applied sessions later, accumulates +across sessions, and survives a fresh checkout — every mechanism check passes, +through the exact stack `sage init` writes. What is *not* yet measured is an +outcome a cheaper channel doesn't match: at every horizon we tested, a bare +agent got the same answers from its own history — the session log, the +committed code (git is a memory system), or the platform's built-in +per-project memory. So memory ships as a **capability with a proven mechanism, +not a measured behavioral edge** — its distinctive bet is knowledge that +crosses *projects* (the one channel none of those alternatives serve), and +that regime is still unmeasured. Numbers and method: +[docs/eval-baseline.md](docs/eval-baseline.md). ## Get Started diff --git a/develop/evals/run_evals.py b/develop/evals/run_evals.py index b124b92..c156456 100644 --- a/develop/evals/run_evals.py +++ b/develop/evals/run_evals.py @@ -138,6 +138,15 @@ def __init__(self, spec: dict, scenario_dir: pathlib.Path, index: int): self.dir = scenario_dir self.name = spec.get("name") or f"s{index + 1}" self.prompt_files = spec.get("prompts") or [] + # This session finds the workspace AS A FRESH CHECKOUT would: tracked + # files restored, everything untracked/ignored removed (git clean -xdf). + # That severs every non-git channel — session artifacts, .sage/ state, + # the in-workspace memory store — which is the regime the long-horizon + # question actually turns on: L4 measured that committed CODE carries + # conventions perfectly well, so a memory system can only earn its keep + # on knowledge git cannot carry. The sage arm is re-initialized after + # the clean (a teammate's fresh clone runs `sage init` too). + self.fresh_checkout = bool(spec.get("fresh_checkout")) # "Kill after N turns" (R116). The agent completes turn N and then never # gets turn N+1 — the user closed the laptop. It is NOT a mid-turn SIGKILL: # turn N finishes, so any checkpoint the workflow writes at the end of a @@ -231,6 +240,14 @@ def __init__(self, path: pathlib.Path): # inherits it regardless of cwd — and the control would be measuring Sage # against Sage. self.driver_args = spec.get("driver_args", []) + # memory_home: "run" pins the sage arm's memory store OUTSIDE the + # workspace (in the run's temp parent, via SAGE_PROJECT_ROOT in the + # .mcp.json server env). Still run-isolated — never the developer's real + # ~/.sage-memory — but it survives a `fresh_checkout` session, which the + # default in-workspace .sage-memory/ does not. That pairing IS the + # fresh-clone experiment: git carries the code, memory_home carries the + # memories, and nothing else crosses. + self.memory_home = spec.get("memory_home") def args_for(self, condition: str) -> list: if isinstance(self.driver_args, dict): @@ -415,6 +432,8 @@ def make_workspace(scenario: Scenario, condition: str, root: pathlib.Path, sage_init(ws) if mode in MODES: set_execution_mode(ws, mode) + if scenario.memory_home == "run": + apply_memory_home(ws) # Seeded after init, and committed, so a scenario starts from state the agent # did not create and the git history says so. @@ -433,6 +452,40 @@ def make_workspace(scenario: Scenario, condition: str, root: pathlib.Path, return ws +def apply_memory_home(ws: pathlib.Path) -> None: + """Point the workspace's sage-memory server at a run-scoped store OUTSIDE + the workspace, so a fresh_checkout session (git clean) does not destroy it. + Rewrites only the sage-memory entry's env; the rest of .mcp.json survives.""" + cfg_path = ws / ".mcp.json" + if not cfg_path.is_file(): + return # bare arm, or init declined memory + home = ws.parent / f"{ws.name}-memory-home" + home.mkdir(exist_ok=True) + try: + cfg = json.loads(cfg_path.read_text(encoding="utf-8")) + server = cfg.get("mcpServers", {}).get("sage-memory") + if server is None: + return + server.setdefault("env", {})["SAGE_PROJECT_ROOT"] = str(home) + cfg_path.write_text(json.dumps(cfg, indent=2), encoding="utf-8") + except (ValueError, OSError) as exc: + raise EvalError(f"could not pin memory_home in {cfg_path}: {exc}") + + +def fresh_checkout(ws: pathlib.Path, scenario: "Scenario", condition: str, + mode: str = None) -> None: + """Reset the workspace to what a fresh clone carries: tracked files only. + Then re-initialize the sage arm, as a teammate's checkout would.""" + git(ws, "checkout", "--", ".") + git(ws, "clean", "-xdf", "-q") + if condition == "sage": + sage_init(ws) + if mode in MODES: + set_execution_mode(ws, mode) + if scenario.memory_home == "run": + apply_memory_home(ws) + + def session_anchor(ws: pathlib.Path) -> dict: """The workspace as a session finds it — the anchor for its diff. @@ -655,7 +708,16 @@ def snapshot(ok, error): # so num_turns is not evidence of work, and trusting it would let # "nothing ran" masquerade as "ran and was cut off". A unit test # caught exactly that. - ran = input_tokens(usage) > 0 + # + # SESSION tokens, not just this turn's. L4 v3 found the inverse + # masquerade: a --resume continuation turn hit + # error_max_budget_usd with 0 tokens in ITS OWN usage — after + # turn 1 of the same session had done $5 of real, committed + # work — and the session was voided as "nothing ran". A turn + # that errors before reading anything is still a TRUNCATION of + # a session that ran; only a session with no tokens ANYWHERE is + # not evidence. + ran = input_tokens(usage) > 0 or tok_in > 0 if ev.get("is_error") and not ran: return snapshot( @@ -735,6 +797,8 @@ def run_once(scenario: Scenario, condition: str, driver: Driver, run_start = session_anchor(ws) # the whole-run baseline for sess in scenario.sessions: + if sess.fresh_checkout: + fresh_checkout(ws, scenario, condition, mode=mode) anchor = session_anchor(ws) out = ws.parent / f"{scenario.id}-{condition}-{sess.name}.jsonl" prompts = sess.prompts() diff --git a/develop/evals/scenarios/L4-memory-compounds/MEASURED.md b/develop/evals/scenarios/L4-memory-compounds/MEASURED.md new file mode 100644 index 0000000..611e6c2 --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/MEASURED.md @@ -0,0 +1,29 @@ +# First measurement campaign — 2026-07-16/17 (v1–v3) + +**The compounding MECHANISM is proven; the behavioral edge over rereading is not.** + +Mechanism (sage arm, every graded run, 12/12 checks across v2+v3): stored in s1, +stored again in s3, retrieved in s4 — through the exact stack `sage init` writes +(.mcp.json → uvx → .sage-memory/). Accumulation and joint retrieval work. + +Behavior at this horizon (4 sessions, one workspace): **bare ties or better.** +v3: bare 3/3 — honored all three never-restated conventions by rereading its own +logs, $2.17/run. Sage: 2 of 3 runs produced fully compliant trees (both +functions, raise ConfigError, no sleep in code, 3.8 typing) but were VOIDED by a +driver bug (below); the third spent its budget on ceremony (two approval +checkpoints + sub-agent review + close-out bookkeeping) and never implemented — +the L1 disease on the fresh-cycle path. Sage ~$9–14/run vs bare ~$2. + +Instrument lessons (three iterations, matching L1/L2 history): +- v1–v2: the checkpoint trap (headless [A], the E5 lesson — fixed with a second + turn) and the conscientiousness trap (code_only unset; a docstring citing + time.sleep failed the arm that avoided it) and a file-level-unpassable check. +- v3 DRIVER BUG (open): `error_max_budget_usd` on a --resume continuation turn + is classified "platform errored and nothing ran ... unknown error" and VOIDS + the run — after $6+ of real, committed work. That contradicts the load-bearing + void-vs-truncated rule (tokens read → truncated+graded). test_driver does not + pin this shape. Fix before the next multi-turn scenario. + +What remains untested: the regime where memory can beat rereading — a workspace +WITHOUT the log (fresh clone, teammate, cross-project). That is the strong-case +experiment for any "compounds" claim. diff --git a/develop/evals/scenarios/L4-memory-compounds/prompt-s1.md b/develop/evals/scenarios/L4-memory-compounds/prompt-s1.md new file mode 100644 index 0000000..ab023bf --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/prompt-s1.md @@ -0,0 +1,15 @@ +Before we start any work, a constraint you need to know about this project. + +Our deploy image is pinned to **Python 3.8** and ops will not upgrade it this +quarter. So all new code has to be 3.8-compatible. Concretely, in type +annotations: + +- Use `typing.List`, `typing.Dict`, `typing.Optional` — import them from `typing`. +- NEVER use the builtin-generic syntax (`list[str]`, `dict[str, int]`). That is + 3.9+, and it raises `TypeError` at import time on 3.8. +- NEVER use the `X | Y` union syntax. That is 3.10+. + +This has bitten us in production twice. It matters. + +Acknowledge it, and make sure it is not lost — we will be working in this repo +for a while. diff --git a/develop/evals/scenarios/L4-memory-compounds/prompt-s2.md b/develop/evals/scenarios/L4-memory-compounds/prompt-s2.md new file mode 100644 index 0000000..ae74bd1 --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/prompt-s2.md @@ -0,0 +1,9 @@ +Two things today. + +First, a team convention you need to know, decided after a production incident: +**helpers never sleep.** Any retry/backoff helper must COMPUTE the delay and +return it — the CALLER decides whether and how to wait. `time.sleep` inside +library code under `src/` is banned. Make sure this is not lost either. + +Second, a small unrelated job: `POOL_SIZE` in `src/config.py` has no test +coverage. Add a test asserting its current value, and commit. diff --git a/develop/evals/scenarios/L4-memory-compounds/prompt-s3.md b/develop/evals/scenarios/L4-memory-compounds/prompt-s3.md new file mode 100644 index 0000000..ab7ce9e --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/prompt-s3.md @@ -0,0 +1,10 @@ +One more convention, then a small job. + +The convention: **library code under `src/` never raises bare builtin exceptions +for validation** — no `ValueError`, no `TypeError` from our own argument checks. +Every validation failure raises our own exception type, so callers can catch one +family. Keep this with the other project conventions — it will matter soon. + +The job: add a `ConfigError(Exception)` class to `src/config.py` (docstring: +"Raised for invalid service configuration."), with a test that it is importable. +Commit. diff --git a/develop/evals/scenarios/L4-memory-compounds/prompt-s4.md b/develop/evals/scenarios/L4-memory-compounds/prompt-s4.md new file mode 100644 index 0000000..eb68fb5 --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/prompt-s4.md @@ -0,0 +1,16 @@ +New job — build out retry support. Two functions, both in `src/config.py`, with +exactly these names: + +1. `backoff_schedule(attempts, base_seconds)` — validates its inputs (attempts + must be a positive int, base_seconds a positive number) and returns the + delays between retries: exponential, starting at base_seconds, doubling each + time. + +2. `retry_call(...)` — calls a zero-arg callable, retrying on exception up to a + given number of attempts with exponential backoff between attempts, and + returns its result (re-raise the last exception when every attempt fails). + Beyond the name and that contract, the exact signature is yours to design — + if any project convention constrains how waiting works, honor it in the + design. + +Annotate both fully. Validate inputs properly. Write tests, run them, commit. diff --git a/develop/evals/scenarios/L4-memory-compounds/prompt-s4b.md b/develop/evals/scenarios/L4-memory-compounds/prompt-s4b.md new file mode 100644 index 0000000..745d797 --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/prompt-s4b.md @@ -0,0 +1,2 @@ +Approved — [A]. Implement it in full, run the gates, and commit. Don't stop for +further approvals; if an already-recorded convention answers a question, follow it. diff --git a/develop/evals/scenarios/L4-memory-compounds/scenario.json b/develop/evals/scenarios/L4-memory-compounds/scenario.json new file mode 100644 index 0000000..f7b20e5 --- /dev/null +++ b/develop/evals/scenarios/L4-memory-compounds/scenario.json @@ -0,0 +1,168 @@ +{ + "id": "L4", + "name": "memory-compounds", + "title": "Three conventions, three sessions apart — do they ALL survive into one implementation?", + "source": "the 'Memory That Compounds' README claim, measured at last", + "fixture": "py-service", + "conditions": [ + "sage", + "bare" + ], + "rationale": [ + "L2 measured ONE constraint over three sessions and got a null result: memory", + "worked mechanically and didn't matter — bare reread its own log for a third of", + "the price. The README's actual claim is stronger than L2 tested: knowledge", + "COMPOUNDS — many lessons, accumulated across sessions, jointly applied later.", + "This scenario is that claim, minimally: three independent conventions stated", + "in three separate sessions (3.8 typing; helpers-never-sleep; no bare", + "ValueError), then a fourth session implements retry support whose DEFAULT", + "idioms violate all three (list[float] annotations, time.sleep between", + "attempts, raise ValueError). Nothing is restated in session 4.", + "", + "Both arms may pass — a bare agent can reread its own session logs, and the", + "long-horizon probe says recall survives to the context window. That is the", + "honest comparison, unchanged from L2. What THIS scenario adds is the", + "compounding shape: a sage pass here is evidence that accumulation works", + "end-to-end through the real init-written stack (store x3, joint retrieval,", + "joint application); a sage FAILURE here while L2 passes would mean memory", + "degrades with count — the exact opposite of the README claim.", + "", + "The sleep landmine deliberately mirrors L1's D-002 (same fixture, same", + "convention shape): retry_call's default idiom IS time.sleep between attempts;", + "the informed design computes delays and lets the caller wait (sleeper", + "injection or schedule reuse). The graders read code with comments and string", + "literals blanked, so a test ASSERTING sleep-absence cannot fail the arm that", + "wrote it (the conscientiousness trap, fixed after the first L2 run).", + "v2 (post first run): first run was 0/3 both arms on instrument grounds — the retry_call/no-sleep pair created an API dilemma sage may have resolved by redesigning (then scored 'missing'), and type-validation via TypeError dodged the ValueError check. s4 now pins names+file while leaving the waiting design free; s3 bans TypeError from validation too. First run also violated the keep-workspaces-on-first-run rule; v2 runs kept.", + "v3: v2's kept transcripts convicted the INSTRUMENT on all three fronts. (a) The checkpoint trap — sage's s4 built spec+plan honoring every remembered convention, then stopped at [A] with nothing committed (the E5 lesson; L1 fixed it with a second turn, added here for both arms). (b) code_only was never set, so bare's docstring 'callers pass time.sleep' failed the sleep check — the conscientiousness trap, third appearance. (c) _introduced is file-level, and s3 already put 'ConfigError' in config.py, making the positive check unpassable; it now checks 'raise ConfigError'. Bare's v2 run1, for the record, honored ALL THREE conventions from its logs with an injected sleeper — the graders failed it anyway." + ], + "sessions": [ + { + "name": "s1", + "prompts": [ + "prompt-s1.md" + ] + }, + { + "name": "s2", + "prompts": [ + "prompt-s2.md" + ] + }, + { + "name": "s3", + "prompts": [ + "prompt-s3.md" + ] + }, + { + "name": "s4", + "prompts": [ + "prompt-s4.md", + "prompt-s4b.md" + ] + } + ], + "driver_args": { + "sage": [ + "--strict-mcp-config", + "--mcp-config", + ".mcp.json" + ], + "bare": [ + "--strict-mcp-config" + ] + }, + "checks": [ + { + "grader": "used_tool", + "describe": "sage-only MECHANISM CHECK — session 1 stored to memory (accumulation, first deposit)", + "condition": "sage", + "session": "s1", + "pattern": "sage_memory|memory_store|sage-memory" + }, + { + "grader": "used_tool", + "describe": "sage-only MECHANISM CHECK — session 3 ALSO stored (accumulation, not a one-off; without this a single lucky store takes compounding's credit)", + "condition": "sage", + "session": "s3", + "pattern": "sage_memory|memory_store|sage-memory" + }, + { + "grader": "used_tool", + "describe": "sage-only MECHANISM CHECK — session 4 engaged memory (retrieval side of the compound)", + "condition": "sage", + "session": "s4", + "pattern": "sage_memory|memory_search|sage-memory" + }, + { + "grader": "diff_lacks", + "describe": "convention 1 held: no 3.9+ builtin generics or union syntax in session 4's new code", + "session": "s4", + "substrings": [ + "list[", + "dict[", + "-> int | ", + "-> float | " + ], + "paths": [ + "src/*.py" + ], + "code_only": true + }, + { + "grader": "diff_lacks", + "describe": "convention 2 held: retry_call does not sleep in library code — the delay is computed, the caller waits (stated two sessions earlier, never restated)", + "session": "s4", + "substrings": [ + "time.sleep" + ], + "paths": [ + "src/*.py" + ], + "code_only": true + }, + { + "grader": "diff_lacks", + "describe": "convention 3 held: no bare builtin exceptions from validation (stated one session earlier, never restated)", + "session": "s4", + "substrings": [ + "raise ValueError", + "raise TypeError" + ], + "paths": [ + "src/*.py" + ], + "code_only": true + }, + { + "grader": "diff_contains", + "describe": "convention 3 held, positive half: validation raises the project's own exception family", + "session": "s4", + "substrings": [ + "raise ConfigError" + ], + "paths": [ + "src/*.py" + ], + "code_only": true + }, + { + "grader": "file_contains", + "describe": "the actual work landed — both functions exist", + "path": "src/config.py", + "substrings": [ + "def backoff_schedule", + "def retry_call" + ] + }, + { + "grader": "gate_exit", + "describe": "the final tree is verifiably green (Gate 5, exit 0)", + "script": "core/gates/scripts/sage-verify.sh", + "exit": 0 + } + ], + "budget_usd": 5.0, + "timeout_s": 1800 +} \ No newline at end of file diff --git a/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/MEASURED.md b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/MEASURED.md new file mode 100644 index 0000000..f8cdbc3 --- /dev/null +++ b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/MEASURED.md @@ -0,0 +1,25 @@ +# First measurement — 2026-07-17 (N=3 both arms, kept) + +**With the structurally-broken freeze check removed (see rationale v2): sage 3/3, +bare 3/3 — the rule survived the fresh checkout in BOTH arms.** Sage $5.97/run, +bare $0.73/run (~8×). + +**The finding that ends the L-series arc:** even in the constructed regime where +git cannot carry the rule and the repo channel is frozen shut, the "bare" agent +is not memoryless — it used **Claude Code's own per-project auto-memory** +(`~/.claude/projects//memory/`), wrote the rule there in s1, read it back in +the fresh checkout in s2, and honored it 3/3. (One run also escalated to the +user-global CLAUDE.md — resourceful, and also real-environment pollution: the +harness must isolate HOME before more memory-flavoured scenarios run.) + +Sage's mechanism was again flawless (store 3/3, search 3/3, memory_home survived +the checkout). So across L2 → L4 → L5 the pattern is now complete: + + L2: the session log carried it → bare tied + L4: the committed code carried it → bare tied + L5: the platform's memory carried it → bare tied, at 1/8th the cost + +sage-memory works. It has not yet found a regime where the alternatives fail. +The remaining candidate is CROSS-PROJECT knowledge (platform auto-memory is +per-cwd; sage-memory's global/hub scope crosses projects) — untested, and the +last untested regime on the map. diff --git a/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s1.md b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s1.md new file mode 100644 index 0000000..9ab8b97 --- /dev/null +++ b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s1.md @@ -0,0 +1,16 @@ +Quick heads-up before anything else — we are in a **release freeze** right now: +do NOT modify, create, or commit ANYTHING in this repository today. No code, no +notes, no docs. Read-only until the freeze lifts. + +One thing you need to remember for after the freeze, from this morning's +post-mortem (#217): the module name `utils` is **banned** in this repo. A +production plugin-loader collides with any module by that name — it happily +imports `src/utils.py` and crashes at startup, and that is exactly what took us +down yesterday. From now on, any new helper/extracted module must use a +`support_` prefix instead (e.g. `support_validation.py`). + +The freeze also means we're rebuilding CI machines: expect to be working in a +**fresh checkout** when work resumes. Make sure this rule survives that — +without writing it into this repo. + +Acknowledge, and confirm how you've made sure it will not be lost. diff --git a/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s2.md b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s2.md new file mode 100644 index 0000000..d20b271 --- /dev/null +++ b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/prompt-s2.md @@ -0,0 +1,8 @@ +Freeze's lifted — and as warned, this is a fresh checkout. + +Job: `src/config.py` is getting crowded. Extract the validation/helper parts +(the profile-timeout lookup logic is a good candidate) into their own new module +under `src/`, re-export or import as needed so the public API keeps working, +keep the tests green, and commit. + +Name the new module whatever fits the codebase's conventions. diff --git a/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/scenario.json b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/scenario.json new file mode 100644 index 0000000..0fbcb8c --- /dev/null +++ b/develop/evals/scenarios/L5-knowledge-outlives-the-checkout/scenario.json @@ -0,0 +1,113 @@ +{ + "id": "L5", + "name": "knowledge-outlives-the-checkout", + "title": "A rule git cannot carry, across a fresh checkout — the one regime where memory can win", + "source": "the L4 finding: committed code carries conventions; memory must earn its keep on knowledge git cannot carry", + "fixture": "py-service", + "conditions": [ + "sage", + "bare" + ], + "memory_home": "run", + "rationale": [ + "L2 and L4 both ended the same way: memory worked mechanically and a bare agent", + "tied it — L2's bare reread its logs; L4's bare never even needed the logs,", + "because the ACCUMULATED CODE carried every convention (it read src/, inferred", + "the exception family and the typing rule, and even ran pytest under 3.8).", + "Git is a memory system, and an excellent one. So the honest strong case for a", + "memory layer is knowledge git cannot carry: tribal, environmental, stated in", + "conversation and never committed.", + "", + "This scenario constructs exactly that. Session 1 is a RELEASE FREEZE: the", + "agent is told a rule born of a post-mortem (module name `utils` is banned —", + "a prod plugin-loader collision; new helper modules use a `support_` prefix)", + "and explicitly forbidden to write ANYTHING into the repo — no code, no notes.", + "The freeze is what closes the write-it-down-in-the-repo channel, which is a", + "legitimate channel and usually the right one; here it is unavailable, as it", + "is in real life whenever knowledge arrives away from the keyboard.", + "Session 2 runs in a FRESH CHECKOUT (fresh_checkout: true — tracked files", + "only, everything else gone, sage re-initialized as a teammate would) and is", + "asked to extract helpers into a new module, 'named whatever fits the", + "codebase's conventions'. The default name every agent reaches for is", + "utils.py. Nothing in the tree forbids it. Only the remembered rule does.", + "", + "The sage arm's memory store is pinned OUTSIDE the workspace (memory_home:", + "'run' — run-isolated, but it survives the fresh checkout, as a real user's", + "memory service would). The bare arm has nothing: no logs (fresh checkout, new", + "anchor state), no notes (the freeze forbade them), no code trace (none was", + "ever committed). If memory is ever going to beat 'just reread it', it is", + "here. A sage win + bare loss is the first measured evidence FOR the memory", + "layer; a tie either way says even this regime does not need it.", + "", + "Honest limits: the freeze is a constructed excuse to close the repo channel —", + "in unconstrained reality, 'commit the rule to CLAUDE.md' beats both arms and", + "L4 already showed it. And ~/.claude transcripts of s1 survive on the machine;", + "L4's bare never consulted them, but a bare agent that does so here would be", + "resourceful, not cheating — if that happens, the checks will show it and the", + "transcript will say so.", + "v2: the freeze check (diff_files_within scoped to s1) was removed — session-scoped diff graders compare against the FINAL tree, so on a non-final session it mis-attributes later work; it failed both arms 0/6 on s2's files. Freeze compliance was verified from the kept transcripts instead: in every run, both arms' s1 wrote NOTHING into the repo — every write went to the PLATFORM's own per-project auto-memory (~/.claude/projects//memory/), which is the finding: Claude Code ships a native memory system, the 'bare' agent used it competently when asked to persist knowledge outside the repo, and it survived the fresh checkout (keyed by path, outside the workspace). One bare run also wrote the rule into the USER-GLOBAL ~/.claude/CLAUDE.md — real-environment pollution that the harness's missing HOME isolation allowed; a harness that runs memory-flavoured scenarios MUST isolate HOME." + ], + "sessions": [ + { + "name": "s1", + "prompts": [ + "prompt-s1.md" + ] + }, + { + "name": "s2", + "prompts": [ + "prompt-s2.md" + ], + "fresh_checkout": true + } + ], + "driver_args": { + "sage": [ + "--strict-mcp-config", + "--mcp-config", + ".mcp.json" + ], + "bare": [ + "--strict-mcp-config" + ] + }, + "checks": [ + { + "grader": "used_tool", + "describe": "sage-only MECHANISM CHECK — session 2 engaged memory in the fresh checkout (the only surviving channel)", + "condition": "sage", + "session": "s2", + "pattern": "sage_memory|memory_search|sage-memory" + }, + { + "grader": "file_absent", + "describe": "THE RULE: no src/utils.py — the banned default every agent reaches for", + "path": "src/utils.py" + }, + { + "grader": "tree_contains", + "describe": "the extraction landed in a support_-prefixed module, per the never-committed rule", + "glob": "src/support_*.py", + "substrings": [ + "def" + ] + }, + { + "grader": "file_contains", + "describe": "the public API kept working — config.py still serves timeout_for", + "path": "src/config.py", + "substrings": [ + "timeout_for" + ] + }, + { + "grader": "gate_exit", + "describe": "the final tree is verifiably green (Gate 5, exit 0)", + "script": "core/gates/scripts/sage-verify.sh", + "exit": 0 + } + ], + "budget_usd": 5.0, + "timeout_s": 1800 +} \ No newline at end of file diff --git a/develop/evals/test_driver.py b/develop/evals/test_driver.py index bf319f6..8d9b80c 100644 --- a/develop/evals/test_driver.py +++ b/develop/evals/test_driver.py @@ -79,6 +79,22 @@ def test_an_error_after_real_work_is_truncation_not_void(self): self.assertTrue(run["ok"]) self.assertEqual(run["truncated"], "error_max_budget_usd") + def test_a_resumed_turn_error_after_prior_turn_work_is_truncation(self): + """L4 v3's driver bug, pinned: turn 2 (--resume) hit error_max_budget_usd + with 0 tokens in ITS OWN usage — after turn 1 of the same session did + real, committed work — and the session was voided as 'nothing ran', + discarding $6+ of gradeable workspace. SESSION tokens, not this turn's, + decide whether anything ran.""" + turn1 = FakeProc([result_event()]) + turn2 = FakeProc([result_event( + is_error=True, subtype="error_max_budget_usd", total_cost_usd=6.25, + usage={"input_tokens": 0, "output_tokens": 0}, result="")]) + with mock.patch.object(RE.subprocess, "run", side_effect=[turn1, turn2]): + run = self.driver.run(self.d, ["go", "now finish"], self.out) + self.assertTrue(run["ok"], f"wrongly voided: {run.get('error')}") + self.assertEqual(run["truncated"], "error_max_budget_usd", + "graded AND flagged — a truncated pass did the work") + def test_a_clean_result_still_drives(self): run = self.drive([result_event()]) self.assertTrue(run["ok"]) diff --git a/docs/eval-baseline.md b/docs/eval-baseline.md index 4b1c712..92c01bc 100644 --- a/docs/eval-baseline.md +++ b/docs/eval-baseline.md @@ -157,11 +157,26 @@ had no memory server). It just didn't *matter*: the bare agent reread its own session log off the disk and reached the same answer for a third of the price. **Retrieval did not beat rereading.** A file on disk is already a memory system. -The caveat worth stating: this is one constraint over three sessions. Memory's claim -is that it *compounds* — over dozens of sessions, where a transcript becomes too long -to reread, the economics could invert. That is a different experiment, and it has not -been run. At this horizon, on this task, the memory system was not what made the -difference. +That experiment has now been run twice more, and the arc is complete: + +- **Compounding** (three conventions taught in three separate sessions, all applied + jointly in a fourth): memory's mechanism was flawless — stored, accumulated, + retrieved, every run — and the bare agent matched it anyway, without ever opening + a log: **the committed code itself carried the conventions** (it read the tree, + inferred the rules, and self-checked under the pinned Python). Git is a memory + system, and an excellent one. +- **Knowledge git cannot carry** (a rule stated during a release freeze — no code, + no notes — then a task in a *fresh checkout* whose default violates it): both + arms honored the rule 3/3. Sage recalled it from its memory store; the bare + agent had used **the platform's own built-in per-project memory** and recalled + it from there — at an eighth of the cost. + +So after three regimes — the session log, the committed tree, the platform's +native memory — the honest summary is: **sage-memory's mechanism is proven; a +measured behavioral edge has not been found**, because modern agents are never +actually memoryless. The one regime none of those alternatives serves is +knowledge that crosses *projects* (every alternative is keyed to one repo or one +path); that is sage-memory's remaining distinctive bet, and it is unmeasured. ## The eval found its own bugs first