diff drift soak: mechanical long-chain pytest + fluency --diff-soak depth eval#75
Merged
Conversation
…epth eval The #8/#20 follow-up gating any --diff default-flip: per-hop losslessness was proven, but nothing exercised DRIFT — reconstruction and comprehension over long chains of consecutive diffs. Mechanical (tests/test_diff_soak.py): drives the real Interceptor with an independent client-side reconstructor (ModelView, public decode fns only) and asserts exactness at EVERY hop — 400 interleaved hops across three evolving workloads (records / dict-map / text log) with error interludes and a mid-soak reconnect; a 300-diff unbounded chain (interval 0); a 200-step text-only chain. Keyframe cadence <=5 verified throughout. Zero drift. Behavioral (fluency --diff-soak [--soak-depth K --soak-windows N]): builds real depth-1..K chain windows from the corpus (every hop must admit a lossless diff), scores final-state questions as (full anchor + k wires) vs full-terse control, reports accuracy by depth + worst-model verdicts overall and at the deepest depth (build_diff_soak_report). Enabler: _flat_record_questions — fluency-local questions for single flat record payloads (keys-count + deterministic scalar lookups), the keys-diff surface the soak was blind to (kb payloads are per-block flat dicts). Same pattern as #72's nested-record source; codec/tabularizer rules untouched. Live result (loopback gateway, deepseek-v4-flash + glm-5.2, 3 trials, ~50 real corpus windows): overall worst gap -2% +/-1, depth-5 gap -1% +/-2 — PASS at 5% tolerance, no depth-correlated drift. 471 tests pass (+8), ruff clean.
inth3shadows
added a commit
that referenced
this pull request
Jul 13, 2026
The validation program that kept --diff opt-in is complete — pair fluency (4-model panel 100%), nested-record coverage (#72), and the drift soak (#75: mechanical zero-drift at depth 300, behavioral depth-1..5 PASS on real corpus chains). The default now matches the measured reality. - Policy.diff defaults True; load_policy's doc.get("diff") default flips to match. An explicit policy-file "diff": false is still honored. - proxy: tri-state CLI — --diff forces on (only useful against an opting-out policy file), new --no-diff forces off, neither keeps the policy value. Mutually exclusive, exit 2. multiproxy diff_override becomes Optional[bool] (None = leave each peer's policy alone) so --no-diff is proxy-wide too. - install-mcp: diff param is tri-state — plain wraps write NO diff flag and inherit the proxy default; --diff/--no-diff bake an explicit override into the entry; keyframe-interval no longer requires --diff (but is dropped with --no-diff, where it would be dead weight). - Tests: the two off-by-default pins become on-by-default + policy-false opt-out pins; install-mcp tri-state covered (None drops flags, False bakes --no-diff). 471 pass, ruff clean. - Docs: README tier/Status, TECHNICAL diff bullet (opt-in rationale rewritten as history + the compaction residual keeps --no-diff as escape hatch), USAGE proxy/install-mcp/text-diff sections; also documents the stash hand-edit gotcha found during the #74 rollout. Live smoke: flagless proxy diffs the repeat structure call (117,587 -> 205 chars); --no-diff sends the full form both times.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The #8/#20 follow-up gating any
--diffdefault-flip. Per-hop losslessness is proven at encode time and unit-pinned, but nothing exercised drift: (1) does client-side reconstruction stay exact hundreds of chained hops deep, across keyframes, shape flips, and reconnects; (2) does model comprehension degrade as consecutive diffs chain off one full anchor (production allows up todiff_keyframe_interval= 5 before re-anchoring).Mechanical half —
tests/test_diff_soak.pyDrives the real
Interceptorwith an independent client-side reconstructor (ModelView, built only on the public decode functions — it never peeks at proxy state), asserting reconstruction == raw payload at every hop:interval 0) — exact at depth 300.Behavioral half —
terse fluency --diff-soakBuilds real depth-1..K chain windows from the corpus (chronological, every hop must admit a lossless diff), scores the same final-state questions as (full anchor + k wires) vs a full-terse control — no system primer, production condition — and reports accuracy by depth with worst-model verdicts overall and at the deepest depth (
build_diff_soak_report;--soak-depth/--soak-windowsto size it).Enabler:
_flat_record_questions— fluency-local questions for single flat-record payloads (keys-count + deterministic scalar lookups). kb payloads are per-block flat dicts that generated zero questions, leaving the deepest real chains untestable; same fluency-local pattern as #72's nested-record source, codec/tabularizer rules untouched.Live result (this PR's gate)
Loopback gateway, deepseek-v4-flash + glm-5.2, 3 trials, ~50 real-corpus windows, depths 1–5:
No depth-correlated drift: the depth-3 dip (−6/−8%) hits both models on the same windows and recovers at depths 4–5 — window-specific, not accumulation.
Verification
ruffclean.--diff-soaksection), TECHNICAL (soak note on the diff limitation bullet), README (status line).