diff drift soak: mechanical long-chain pytest + fluency --diff-soak depth eval

README.md

@@ -127,6 +127,9 @@ repeated objects, not just strings) is built. Cross-call diffing is built as an
 **opt-in** lossless tier (`proxy --diff` / `install-mcp --diff`); its fluency gate
 (`fluency --diff`) has passed on both the record surface (4-model panel, 100%) and the
 nested-record surface (`structure`: diff 100% vs full-terse 94%), so it's cleared for
-live use. The Tier 1 lossy modes `truncate` and
+live use. Long-chain drift is soaked from both sides: mechanically
+(`tests/test_diff_soak.py` — exact reconstruction hundreds of chained hops deep) and
+behaviorally (`fluency --diff-soak` — model accuracy vs chain depth up to the keyframe
+bound). The Tier 1 lossy modes `truncate` and
 `drop-to-retrieve` are built (opt-in, off by default); `summarize` remains designed but
 not yet built — see TECHNICAL.md "Known Limitations".

TECHNICAL.md

@@ -273,8 +273,14 @@ gitignored because captured tool output may contain real data.
   default 5; 0 disables). A client *reconnect* (a new MCP `initialize`) is a stronger
   reset signal: the proxy drops every per-tool diff base on it, so the next result of
   each tool re-anchors as a full rather than a delta against a base the rebuilt context
-  no longer holds (#20). The residual gap — a context *compaction* with no reconnect — is
-  unobservable over stdio, and is the standing reason `--diff` is opt-in.
+  no longer holds (#20). Both properties are **soaked, not just unit-tested**: the drift
+  soak (`tests/test_diff_soak.py`) drives the real Interceptor hundreds of chained hops
+  deep — interleaved tools, error interludes, a mid-soak reconnect, and a 300-diff
+  unbounded chain — with an independent client-side reconstructor asserting exactness at
+  every hop; `terse fluency --diff-soak` measures the model-side analogue (accuracy vs
+  chain depth, up to the keyframe bound). The residual gap — a context *compaction* with
+  no reconnect — is unobservable over stdio, and is the standing reason `--diff` is
+  opt-in.
 - **Text diff (Tier 0.7 text, #25) covers non-JSON results, but only the codec side is
   measured so far.** File reads, source excerpts, and log tails get `applicable: False`
   in `measure_payload` — zero Tier-0 compression — and used to get zero cross-call

USAGE.md

@@ -354,6 +354,25 @@ questions and PASS/FAILs on the worst model — run it before enabling `proxy --
 your consumer. The diff is always lossless and only sent when smaller; it falls back to
 the full compressed form whenever no diff applies or the prior result isn't available.
 
+### The depth dimension: `fluency --diff-soak`
+
+`--diff` tests one hop (full result + one diff). In production a model reads up to
+`--diff-keyframe-interval` (default 5) **consecutive** diffs off one full anchor
+before the proxy re-anchors — so the question gating a default-flip is whether
+comprehension *drifts* with chain depth:
+
+```
+# real corpus runs, depths 1..5 (6 chain windows per depth, round-robin across tools):
+TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
+  uv run terse fluency --diff-soak --corpus corpus --trials 3
+```
+
+The report shows accuracy by depth (chain form vs full-terse control on the same
+final-state questions) and PASS/FAILs both overall and at the deepest tested depth,
+worst model. The mechanical half of the same soak — hundreds of chained hops
+reconstructed exactly, keyframe cadence, reconnect resets — is pinned in
+`tests/test_diff_soak.py` and runs in CI.
+
 ### Text diffing and its fluency check (`fluency --text-diff-eval`)
 
 The diff above only reasons about JSON. Non-JSON tool output (file reads, source

src/terse/cli.py

@@ -376,6 +376,7 @@ def _cmd_fluency(args: argparse.Namespace) -> int:
     from .policy import default_policy, load_policy
     from .report import (
         build_diff_report,
+        build_diff_soak_report,
         build_dropeval_report,
         build_fluency_report,
         build_text_diff_report,
@@ -432,6 +433,23 @@ def _cmd_fluency(args: argparse.Namespace) -> int:
             print("\n" + build_terminal_diff_report(results))
         return 0
 
+    # Diff-chain soak: the DEPTH dimension --diff can't see (#8/#20 follow-up) — does
+    # comprehension drift as consecutive diffs chain off one full anchor? Depth 5 is
+    # the production keyframe bound, so a PASS here covers the deployed worst case.
+    if args.diff_soak:
+        answerers = _build_answerers(args, fluency.openai_answerer)
+        if not answerers:
+            print("`fluency --diff-soak` needs a configured model: set TERSE_FLUENCY_BASE_URL/"
+                  "_API_KEY/_MODELS.")
+            return 1
+        results = fluency.run_diff_soak(envelopes, answerers, trials=args.trials,
+                                        max_depth=args.soak_depth,
+                                        per_depth_cap=args.soak_windows)
+        _write_report(build_diff_soak_report(results), args.out)
+        if args.bars:
+            print("\n" + build_terminal_diff_report(results, form_label="chain-form"))
+        return 0
+
     # Text-diff mode: does a model reconstruct the current TEXT as well from (previous
     # text + text-diff) as from the full current text? The text-payload analogue of
     # --diff above (text_diff.py, Tier 0.7 — non-JSON tool output).
@@ -811,6 +829,16 @@ def main(argv: list[str] | None = None) -> int:
                    help="behavioral eval: does a model reconstruct the current TEXT as "
                         "accurately from (previous text + text-diff) as from the full "
                         "text? needs same-tool TEXT corpus pairs + a configured model")
+    f.add_argument("--diff-soak", action="store_true",
+                   help="drift soak: does comprehension degrade as a model reads chains "
+                        "of consecutive diffs off one full anchor? scores depths "
+                        "1..--soak-depth over real corpus runs (needs a configured model)")
+    f.add_argument("--soak-depth", type=int, default=5, metavar="K",
+                   help="--diff-soak: deepest chain to test (default 5 — the production "
+                        "keyframe interval, i.e. the deployed worst case)")
+    f.add_argument("--soak-windows", type=int, default=6, metavar="N",
+                   help="--diff-soak: max chain windows per depth, round-robin across "
+                        "tools (default 6)")
     f.add_argument("--drop-eval", action="store_true",
                    help="behavioral eval: does a real tool-calling model call terse.retrieve "
                         "when a dropped field is needed (recall), and leave it alone when "

src/terse/fluency.py

@@ -41,7 +41,8 @@
 from . import text_diff
 from .capture import LONG_TEXT, OTHER, classify_shape, extract_records
 from .tokenize import count_cl100k
-from .transforms import compress, compress_structure, dict_encode, diff_wire, minify, _uniform_dict_list
+from .transforms import (compress, compress_structure, dict_encode, diff_wire,
+                         has_terse_marker, minify, _uniform_dict_list)
 
 # Loopback hosts where cleartext http is safe (never leaves the machine), so a Bearer
 # key over http to one of these is fine — a local LiteLLM/CCR gateway is a common setup.
@@ -249,13 +250,54 @@ def _nested_questions(obj: Any) -> list[Question]:
     return qs
 
 
+def _flat_record_questions(obj: Any) -> list[Question]:
+    """Questions for a SINGLE flat record — a dict of mostly scalar fields (a search
+    hit, a status receipt, one KB row). These payloads tabularize to nothing, but they
+    are a real diff surface: consecutive single-record results diff via the keys shape,
+    and before this generator the soak/diff evals were blind to every such chain.
+    Same contract as the other sources: deterministic, programmatically checkable,
+    fluency-local (the codec/tabularizer rules are untouched)."""
+    if not isinstance(obj, dict) or has_terse_marker(obj):
+        return []
+    # int/float lookups are unambiguous; strings must be short, non-empty scalars —
+    # an empty expected would be indistinguishable from _safe_ask's empty-string
+    # error return (same exclusion the text-diff questions apply).
+    lookable = {
+        k: v for k, v in obj.items()
+        if (isinstance(v, (int, float)) and not isinstance(v, bool))
+        or (isinstance(v, str) and 0 < len(v) <= 60)
+    }
+    if len(lookable) < 3:
+        return []
+    qs: list[Question] = [Question(
+        "keys-count", "count", "flat-record",
+        "How many top-level keys does the object have?",
+        "Reply with only the integer count.",
+        len(obj),
+    )]
+    keys = sorted(lookable)
+    num_key = next((k for k in keys
+                    if isinstance(lookable[k], (int, float))), None)
+    str_key = next((k for k in keys if isinstance(lookable[k], str)), None)
+    for i, k in enumerate(x for x in (num_key, str_key) if x is not None):
+        qs.append(Question(
+            f"field-{i}", "lookup", "flat-record",
+            f"What is the value of the top-level field {k!r}?",
+            "Reply with only the value, with no quotes and no extra words.",
+            lookable[k],
+        ))
+    return qs
+
+
 def gen_questions(obj: Any) -> list[Question]:
     """Generate deterministic, programmatically-checkable questions for a payload.
 
     Uniform record-shaped payloads (what terse tabularizes) yield questions directly;
     payloads whose records the strict extractor skips (structure's dict-map of non-uniform
-    symbol lists) fall through to `_nested_questions` (#71); everything else returns [].
-    Selection is fully deterministic so a re-run is reproducible.
+    symbol lists) fall through to `_nested_questions` (#71); a SINGLE flat record (a
+    search hit, a receipt) falls through to `_flat_record_questions` — the keys-diff
+    surface the chain soak needs; everything else returns []. Selection is fully
+    deterministic so a re-run is reproducible.
     """
     # Structure-shaped payloads (a dict-map of records) need GROUP-SCOPED questions; prefer
     # them over the uniform extractor, which would otherwise emit an unscoped, ambiguous
@@ -265,7 +307,7 @@ def gen_questions(obj: Any) -> list[Question]:
         return nested
     records = extract_records(obj)
     if not records:
-        return []
+        return _flat_record_questions(obj)
     cols = list(records[0].keys())
     n = len(records)
     aliased = _aliased_strings(obj)
@@ -452,8 +494,8 @@ def run_fluency(envelopes: list[dict], answerers: dict[str, Answerer],
     """Run the eval for each named answerer over every record-shaped payload.
 
     Returns {model_name: [scored_row, ...]} where each row carries tool/sha plus the
-    per-form success counts and trial count. Non-record payloads are skipped (they
-    generate no questions — terse does not transform them).
+    per-form success counts and trial count. Payloads that generate no questions
+    (no record list, no nested dict-map, not a flat record) are skipped.
     """
     results: dict[str, list[dict]] = {}
     for name, fn in answerers.items():
@@ -552,6 +594,134 @@ def run_diff_fluency(envelopes: list[dict], answerers: dict[str, Answerer],
     return _aggregate_by_model(pairs, answerers, trials, run_diff_payload)
 
 
+# --------------------------------------------------------------------------- #
+# Diff-chain soak — the DEPTH dimension run_diff_fluency can't see (#8/#20 follow-up).
+# run_diff_payload tests one hop (full + 1 diff). In production a model reads up to
+# `diff_keyframe_interval` (default 5) consecutive diffs off one full anchor before
+# the proxy re-anchors — so the open question before any default-flip is whether
+# comprehension DRIFTS with chain depth. These functions build real depth-k windows
+# from the corpus (chronological order, #67) and score the same final-state questions
+# at every depth, so the report can show accuracy as a function of depth.
+# --------------------------------------------------------------------------- #
+def build_chain_windows(envelopes: list[dict], max_depth: int = 5,
+                        per_depth_cap: int = 6) -> list[tuple]:
+    """Depth-k windows (k = 1..max_depth) of consecutive same-tool payloads where
+    EVERY hop admits a lossless diff and the final payload generates questions —
+    the exact between-keyframe chains `proxy --diff` emits. Envelope order is
+    preserved (load_corpus replays chronologically, #67), so windows are real
+    session sequences, not synthetic evolutions.
+
+    Sampling per depth is round-robin across tools (one window per tool per pass,
+    starts spread along each tool's maximal diffable run) until `per_depth_cap` —
+    tool diversity beats stacking one chatty tool's windows. Returns
+    [(tool, final_sha, depth, [obj_0 .. obj_k])]."""
+    runs: dict[str, list[list[tuple]]] = {}   # tool -> maximal runs of (sha, obj)
+    by_tool: dict[str, list[dict]] = {}
+    for env in envelopes:
+        by_tool.setdefault(env["tool"], []).append(env)
+    for tool, envs in by_tool.items():
+        seq: list[tuple] = []
+        for e in envs:
+            try:
+                seq.append((e.get("sha", "?"), json.loads(e["raw"])))
+            except (json.JSONDecodeError, TypeError):
+                seq.append((e.get("sha", "?"), None))
+        cur: list[tuple] = []
+        for (psha, p), (csha, c) in zip(seq, seq[1:]):
+            ok = (p is not None and c is not None
+                  and diff_wire(p, c, tool) is not None)
+            if ok:
+                if not cur:
+                    cur = [(psha, p)]
+                cur.append((csha, c))
+            elif cur:
+                runs.setdefault(tool, []).append(cur)
+                cur = []
+        if cur:
+            runs.setdefault(tool, []).append(cur)
+
+    windows: list[tuple] = []
+    for depth in range(1, max_depth + 1):
+        # every eligible start per tool, spread over its runs; then round-robin
+        candidates: dict[str, list[tuple]] = {}
+        for tool, tool_runs in sorted(runs.items()):
+            starts: list[tuple] = []
+            for run in tool_runs:
+                for i in range(0, len(run) - depth, depth):   # non-overlapping
+                    window = run[i:i + depth + 1]
+                    if gen_questions(window[-1][1]):
+                        starts.append(window)
+            if starts:
+                candidates[tool] = starts
+        picked = 0
+        idx = 0
+        while picked < per_depth_cap and candidates:
+            for tool in sorted(list(candidates)):
+                starts = candidates[tool]
+                if idx >= len(starts):
+                    del candidates[tool]
+                    continue
+                window = starts[idx]
+                windows.append((tool, window[-1][0], depth,
+                                [obj for _, obj in window]))
+                picked += 1
+                if picked >= per_depth_cap:
+                    break
+            idx += 1
+    return windows
+
+
+def run_chain_payload(objs: list, answerer: Answerer, tool: str = "",
+                      trials: int = 1) -> list[dict]:
+    """run_diff_payload generalized to a depth-N chain: the same questions about the
+    FINAL state, control = full-terse of the final result, form = the base full-terse
+    plus every intermediate diff wire in order — exactly the context a model has
+    accumulated after N consecutive diffs, with no system primer (production
+    condition). [] if any hop stops admitting a lossless diff or no questions."""
+    questions = gen_questions(objs[-1])
+    if not questions:
+        return []
+    wires: list[str] = []
+    for prev, curr in zip(objs, objs[1:]):
+        wire = diff_wire(prev, curr, tool)
+        if wire is None:
+            return []
+        wires.append(wire)
+    curr_terse = compress(objs[-1])
+    chain_data = f"PREVIOUS RESULT:\n{compress(objs[0])}" + "".join(
+        f"\n\nUPDATE (diff against the result above, applied in order):\n{w}"
+        for w in wires)
+    out: list[dict] = []
+    for q in questions:
+        full_u = _user_prompt(q.prompt, q.instruction, curr_terse)
+        chain_u = _user_prompt(q.prompt, q.instruction, chain_data)
+        out.append({
+            "qid": q.qid, "qtype": q.qtype, "transform": q.transform, "trials": trials,
+            "depth": len(wires),
+            "terse_ok": _ask_n(answerer, "", full_u, q.qtype, q.expected, trials),
+            "diff_ok": _ask_n(answerer, "", chain_u, q.qtype, q.expected, trials),
+        })
+    return out
+
+
+def run_diff_soak(envelopes: list[dict], answerers: dict[str, Answerer],
+                  trials: int = 1, max_depth: int = 5,
+                  per_depth_cap: int = 6) -> dict:
+    """Score every answerer over the same depth-1..max_depth chain windows.
+    Returns {model: [row,...]} where each row also carries `depth` — the report
+    aggregates by it to show comprehension as a function of chain depth."""
+    windows = build_chain_windows(envelopes, max_depth=max_depth,
+                                  per_depth_cap=per_depth_cap)
+    results: dict[str, list[dict]] = {}
+    for name, fn in answerers.items():
+        rows: list[dict] = []
+        for tool, sha, _depth, objs in windows:
+            for row in run_chain_payload(objs, fn, tool, trials=trials):
+                rows.append({"tool": tool, "sha": sha, **row})
+        results[name] = rows
+    return results
+
+
 # --------------------------------------------------------------------------- #
 # Text-diff fluency — the text-payload analogue of the diff eval above.
 #