eval: the memory question, measured to completion (L4+L5) — plus the driver fix and the honest README claim

README.md

@@ -15,7 +15,7 @@ Built for product and engineering teams, open to any domain.
 - **Think first, build second** — a framing round challenges assumptions before solutioning begins, preventing the most expensive mistake: solving the wrong problem
 - **Focus over noise** — loads only what the task needs, producing sharper reasoning
 - **Mechanical where it counts** — hooks that block a source edit until a test exists and an edit before a spec exists, and gate scripts with a three-state exit contract. These are code, not instructions, and they hold: test-first measures **3/3 against a bare agent's 0/3**. The prose layers around them are advice, and advice is rationalizable — see [What we measured](#what-we-measured) for which is which
-- **Gets smarter over time** — self-learning, memory, and ontology compound into institutional knowledge of your codebase
+- **Persistent memory built in** — self-learning, project memory, and an entity ontology, wired up at init; the mechanism is measured (see below), the compounding bet is stated honestly
 - **Grows with its ecosystem** — 12 focused core skills plus installable packs (product/UX, pack-authoring, autoresearch), extensible with 90K+ community skills from skills.sh
 
 ## What we measured
@@ -140,22 +140,31 @@ no behaviour lost.
 
 Numbers and the full story: [docs/eval-baseline.md](docs/eval-baseline.md).
 
-### Memory That Compounds
+### Persistent Memory
 
 <p align="center">
   <img src="docs/assets/sage_memory.svg" alt="Sage Memory System — 3 skills, 1 MCP, compounding knowledge." width="600" />
 </p>
 
-Most agent frameworks are stateless. The agent that made a mistake
-yesterday makes it again today. Sage has three skills that build
-institutional memory — all backed by sage-memory MCP:
-
-- **sage-self-learning** captures mistakes as WHEN/CHECK/BECAUSE prevention rules. Every session starts by searching past mistakes before doing anything.
-- **sage-memory** stores project knowledge as focused prose insights — how your auth works, why billing uses event sourcing, what conventions the team follows.
-- **sage-ontology** maps entity relationships — not just "billing exists" but "billing depends on payments, which triggers webhooks, which notify users." Touch one module, know the blast radius.
-
-Day 1, the agent knows nothing. Day 30, it knows your codebase's
-landmines, patterns, and conventions.
+Sage ships a persistent memory layer — three skills backed by the sage-memory
+MCP, wired up automatically by `sage init` (opt out with `--no-memory`):
+
+- **sage-self-learning** captures mistakes as WHEN/CHECK/BECAUSE prevention rules.
+- **sage-memory** stores project knowledge as focused prose insights — conventions, decisions, gotchas.
+- **sage-ontology** maps entity relationships — touch one module, know the blast radius.
+
+**What's measured, honestly.** The mechanism is proven end-to-end: knowledge
+stored in one session is retrieved and applied sessions later, accumulates
+across sessions, and survives a fresh checkout — every mechanism check passes,
+through the exact stack `sage init` writes. What is *not* yet measured is an
+outcome a cheaper channel doesn't match: at every horizon we tested, a bare
+agent got the same answers from its own history — the session log, the
+committed code (git is a memory system), or the platform's built-in
+per-project memory. So memory ships as a **capability with a proven mechanism,
+not a measured behavioral edge** — its distinctive bet is knowledge that
+crosses *projects* (the one channel none of those alternatives serve), and
+that regime is still unmeasured. Numbers and method:
+[docs/eval-baseline.md](docs/eval-baseline.md).
 
 ## Get Started
 

develop/evals/run_evals.py

@@ -138,6 +138,15 @@ def __init__(self, spec: dict, scenario_dir: pathlib.Path, index: int):
         self.dir = scenario_dir
         self.name = spec.get("name") or f"s{index + 1}"
         self.prompt_files = spec.get("prompts") or []
+        # This session finds the workspace AS A FRESH CHECKOUT would: tracked
+        # files restored, everything untracked/ignored removed (git clean -xdf).
+        # That severs every non-git channel — session artifacts, .sage/ state,
+        # the in-workspace memory store — which is the regime the long-horizon
+        # question actually turns on: L4 measured that committed CODE carries
+        # conventions perfectly well, so a memory system can only earn its keep
+        # on knowledge git cannot carry. The sage arm is re-initialized after
+        # the clean (a teammate's fresh clone runs `sage init` too).
+        self.fresh_checkout = bool(spec.get("fresh_checkout"))
         # "Kill after N turns" (R116). The agent completes turn N and then never
         # gets turn N+1 — the user closed the laptop. It is NOT a mid-turn SIGKILL:
         # turn N finishes, so any checkpoint the workflow writes at the end of a
@@ -231,6 +240,14 @@ def __init__(self, path: pathlib.Path):
         # inherits it regardless of cwd — and the control would be measuring Sage
         # against Sage.
         self.driver_args = spec.get("driver_args", [])
+        # memory_home: "run" pins the sage arm's memory store OUTSIDE the
+        # workspace (in the run's temp parent, via SAGE_PROJECT_ROOT in the
+        # .mcp.json server env). Still run-isolated — never the developer's real
+        # ~/.sage-memory — but it survives a `fresh_checkout` session, which the
+        # default in-workspace .sage-memory/ does not. That pairing IS the
+        # fresh-clone experiment: git carries the code, memory_home carries the
+        # memories, and nothing else crosses.
+        self.memory_home = spec.get("memory_home")
 
     def args_for(self, condition: str) -> list:
         if isinstance(self.driver_args, dict):
@@ -415,6 +432,8 @@ def make_workspace(scenario: Scenario, condition: str, root: pathlib.Path,
         sage_init(ws)
         if mode in MODES:
             set_execution_mode(ws, mode)
+        if scenario.memory_home == "run":
+            apply_memory_home(ws)
 
     # Seeded after init, and committed, so a scenario starts from state the agent
     # did not create and the git history says so.
@@ -433,6 +452,40 @@ def make_workspace(scenario: Scenario, condition: str, root: pathlib.Path,
     return ws
 
 
+def apply_memory_home(ws: pathlib.Path) -> None:
+    """Point the workspace's sage-memory server at a run-scoped store OUTSIDE
+    the workspace, so a fresh_checkout session (git clean) does not destroy it.
+    Rewrites only the sage-memory entry's env; the rest of .mcp.json survives."""
+    cfg_path = ws / ".mcp.json"
+    if not cfg_path.is_file():
+        return                       # bare arm, or init declined memory
+    home = ws.parent / f"{ws.name}-memory-home"
+    home.mkdir(exist_ok=True)
+    try:
+        cfg = json.loads(cfg_path.read_text(encoding="utf-8"))
+        server = cfg.get("mcpServers", {}).get("sage-memory")
+        if server is None:
+            return
+        server.setdefault("env", {})["SAGE_PROJECT_ROOT"] = str(home)
+        cfg_path.write_text(json.dumps(cfg, indent=2), encoding="utf-8")
+    except (ValueError, OSError) as exc:
+        raise EvalError(f"could not pin memory_home in {cfg_path}: {exc}")
+
+
+def fresh_checkout(ws: pathlib.Path, scenario: "Scenario", condition: str,
+                   mode: str = None) -> None:
+    """Reset the workspace to what a fresh clone carries: tracked files only.
+    Then re-initialize the sage arm, as a teammate's checkout would."""
+    git(ws, "checkout", "--", ".")
+    git(ws, "clean", "-xdf", "-q")
+    if condition == "sage":
+        sage_init(ws)
+        if mode in MODES:
+            set_execution_mode(ws, mode)
+        if scenario.memory_home == "run":
+            apply_memory_home(ws)
+
+
 def session_anchor(ws: pathlib.Path) -> dict:
     """The workspace as a session finds it — the anchor for its diff.
 
@@ -655,7 +708,16 @@ def snapshot(ok, error):
                     # so num_turns is not evidence of work, and trusting it would let
                     # "nothing ran" masquerade as "ran and was cut off". A unit test
                     # caught exactly that.
-                    ran = input_tokens(usage) > 0
+                    #
+                    # SESSION tokens, not just this turn's. L4 v3 found the inverse
+                    # masquerade: a --resume continuation turn hit
+                    # error_max_budget_usd with 0 tokens in ITS OWN usage — after
+                    # turn 1 of the same session had done $5 of real, committed
+                    # work — and the session was voided as "nothing ran". A turn
+                    # that errors before reading anything is still a TRUNCATION of
+                    # a session that ran; only a session with no tokens ANYWHERE is
+                    # not evidence.
+                    ran = input_tokens(usage) > 0 or tok_in > 0
 
                     if ev.get("is_error") and not ran:
                         return snapshot(
@@ -735,6 +797,8 @@ def run_once(scenario: Scenario, condition: str, driver: Driver,
     run_start = session_anchor(ws)          # the whole-run baseline
 
     for sess in scenario.sessions:
+        if sess.fresh_checkout:
+            fresh_checkout(ws, scenario, condition, mode=mode)
         anchor = session_anchor(ws)
         out = ws.parent / f"{scenario.id}-{condition}-{sess.name}.jsonl"
         prompts = sess.prompts()

develop/evals/scenarios/L4-memory-compounds/MEASURED.md

@@ -0,0 +1,29 @@
+# First measurement campaign — 2026-07-16/17 (v1–v3)
+
+**The compounding MECHANISM is proven; the behavioral edge over rereading is not.**
+
+Mechanism (sage arm, every graded run, 12/12 checks across v2+v3): stored in s1,
+stored again in s3, retrieved in s4 — through the exact stack `sage init` writes
+(.mcp.json → uvx → .sage-memory/). Accumulation and joint retrieval work.
+
+Behavior at this horizon (4 sessions, one workspace): **bare ties or better.**
+v3: bare 3/3 — honored all three never-restated conventions by rereading its own
+logs, $2.17/run. Sage: 2 of 3 runs produced fully compliant trees (both
+functions, raise ConfigError, no sleep in code, 3.8 typing) but were VOIDED by a
+driver bug (below); the third spent its budget on ceremony (two approval
+checkpoints + sub-agent review + close-out bookkeeping) and never implemented —
+the L1 disease on the fresh-cycle path. Sage ~$9–14/run vs bare ~$2.
+
+Instrument lessons (three iterations, matching L1/L2 history):
+- v1–v2: the checkpoint trap (headless [A], the E5 lesson — fixed with a second
+  turn) and the conscientiousness trap (code_only unset; a docstring citing
+  time.sleep failed the arm that avoided it) and a file-level-unpassable check.
+- v3 DRIVER BUG (open): `error_max_budget_usd` on a --resume continuation turn
+  is classified "platform errored and nothing ran ... unknown error" and VOIDS
+  the run — after $6+ of real, committed work. That contradicts the load-bearing
+  void-vs-truncated rule (tokens read → truncated+graded). test_driver does not
+  pin this shape. Fix before the next multi-turn scenario.
+
+What remains untested: the regime where memory can beat rereading — a workspace
+WITHOUT the log (fresh clone, teammate, cross-project). That is the strong-case
+experiment for any "compounds" claim.

develop/evals/scenarios/L4-memory-compounds/prompt-s1.md

@@ -0,0 +1,15 @@
+Before we start any work, a constraint you need to know about this project.
+
+Our deploy image is pinned to **Python 3.8** and ops will not upgrade it this
+quarter. So all new code has to be 3.8-compatible. Concretely, in type
+annotations:
+
+- Use `typing.List`, `typing.Dict`, `typing.Optional` — import them from `typing`.
+- NEVER use the builtin-generic syntax (`list[str]`, `dict[str, int]`). That is
+  3.9+, and it raises `TypeError` at import time on 3.8.
+- NEVER use the `X | Y` union syntax. That is 3.10+.
+
+This has bitten us in production twice. It matters.
+
+Acknowledge it, and make sure it is not lost — we will be working in this repo
+for a while.

develop/evals/scenarios/L4-memory-compounds/prompt-s2.md

@@ -0,0 +1,9 @@
+Two things today.
+
+First, a team convention you need to know, decided after a production incident:
+**helpers never sleep.** Any retry/backoff helper must COMPUTE the delay and
+return it — the CALLER decides whether and how to wait. `time.sleep` inside
+library code under `src/` is banned. Make sure this is not lost either.
+
+Second, a small unrelated job: `POOL_SIZE` in `src/config.py` has no test
+coverage. Add a test asserting its current value, and commit.

develop/evals/scenarios/L4-memory-compounds/prompt-s3.md

@@ -0,0 +1,10 @@
+One more convention, then a small job.
+
+The convention: **library code under `src/` never raises bare builtin exceptions
+for validation** — no `ValueError`, no `TypeError` from our own argument checks.
+Every validation failure raises our own exception type, so callers can catch one
+family. Keep this with the other project conventions — it will matter soon.
+
+The job: add a `ConfigError(Exception)` class to `src/config.py` (docstring:
+"Raised for invalid service configuration."), with a test that it is importable.
+Commit.

develop/evals/scenarios/L4-memory-compounds/prompt-s4.md

@@ -0,0 +1,16 @@
+New job — build out retry support. Two functions, both in `src/config.py`, with
+exactly these names:
+
+1. `backoff_schedule(attempts, base_seconds)` — validates its inputs (attempts
+   must be a positive int, base_seconds a positive number) and returns the
+   delays between retries: exponential, starting at base_seconds, doubling each
+   time.
+
+2. `retry_call(...)` — calls a zero-arg callable, retrying on exception up to a
+   given number of attempts with exponential backoff between attempts, and
+   returns its result (re-raise the last exception when every attempt fails).
+   Beyond the name and that contract, the exact signature is yours to design —
+   if any project convention constrains how waiting works, honor it in the
+   design.
+
+Annotate both fully. Validate inputs properly. Write tests, run them, commit.

develop/evals/scenarios/L4-memory-compounds/prompt-s4b.md

@@ -0,0 +1,2 @@
+Approved — [A]. Implement it in full, run the gates, and commit. Don't stop for
+further approvals; if an already-recorded convention answers a question, follow it.