fluency: cover nested-record tools (structure) so proxy --diff is validated (#71)#72
Merged
Merged
Conversation
…idated there (#71) The diff-fluency gate generated zero questions for runecho.structure — a dict-map of file records whose symbols[] are non-uniform (imports carry name/kind; functions also carry line/hash), which the strict identical-keyset extractor skips. So the high-value diff surface (72-96% cross-call overlap, the +31.7% savings) was unvalidated while the harness still reported PASS on locate alone. Add fluency-local _nested_record_group + _nested_questions: group-scoped count/enumerate/lookup (+ aggregate when a numeric column is shared) over the columns present in EVERY record, scoped to the first file ("...under files[\"a/x.py\"]..."), so non-uniform lists are handled and ground truth stays well-posed. Preferred over the uniform extractor for dict-map payloads to avoid an ambiguous unscoped count. Does NOT touch the shared extract_records/_uniform_dict_list — the tabularizer, probe, and drop-path logic (#47) still see the strict rule unchanged. Verified: real structure payloads now generate 3 questions each; structure diff-fluency runs and PASSES (deepseek-v4-flash, 0 regressions). 457 tests pass (5 new nested-path).
… review) Code review caught that _nested_questions picked the lookup's addressing column by most-distinct values, not full uniqueness. When that column has duplicates (common in structure: repeated 'kind', overloaded 'name'), the prompt 'the record whose X is V' matches multiple records but bakes in one arbitrary expected value — a truthful answer about another matching record scores as a false-negative regression, exactly on the structure payloads this feature enables. Split the concerns: enumerate keeps using the most-distinct string column (order/count check tolerates duplicates); lookup now reuses the uniform path's _pick_id_col (requires len(set)==n) and is skipped entirely when no column uniquely addresses a record. Add a regression test for the all-duplicate case.
inth3shadows
added a commit
that referenced
this pull request
Jul 13, 2026
#74) The #63->#72 validation arc passed (4-model panel 100% on records; structure diff 100% vs 94% full-terse) but the diff tier was never deployable through the installer, and help/docs still called its fluency unverified. - wrap()/do_install() gain diff + diff_keyframe_interval; the wrapped proxy args carry --diff [--diff-keyframe-interval K]. Flags reflect the latest install invocation - a re-install without --diff drops it (never accumulates). - CLI: install-mcp --diff / --diff-keyframe-interval; _cmd_install_mcp prints the diff state. - Stale wording fixed: proxy --diff help, install_mcp module docstring, README tier/Status sections, USAGE install-mcp section. - Tests +2: flag placement before the -- separator, keyframe only rides with --diff, do_install threading, drop-on-reinstall. 460 pass, ruff clean. Evidence for enabling live (972-payload session corpus): tier-0 saves +3.4% overall; codegraph_explore alone is 25% of live tokens at +0.0%, with 25-73% cross-call overlap - the diff tier's exact target. Live smoke: repeat runecho.structure call emitted a 205-char diff vs the 117,587-char full result.
inth3shadows
added a commit
that referenced
this pull request
Jul 13, 2026
…epth eval (#75) The #8/#20 follow-up gating any --diff default-flip: per-hop losslessness was proven, but nothing exercised DRIFT — reconstruction and comprehension over long chains of consecutive diffs. Mechanical (tests/test_diff_soak.py): drives the real Interceptor with an independent client-side reconstructor (ModelView, public decode fns only) and asserts exactness at EVERY hop — 400 interleaved hops across three evolving workloads (records / dict-map / text log) with error interludes and a mid-soak reconnect; a 300-diff unbounded chain (interval 0); a 200-step text-only chain. Keyframe cadence <=5 verified throughout. Zero drift. Behavioral (fluency --diff-soak [--soak-depth K --soak-windows N]): builds real depth-1..K chain windows from the corpus (every hop must admit a lossless diff), scores final-state questions as (full anchor + k wires) vs full-terse control, reports accuracy by depth + worst-model verdicts overall and at the deepest depth (build_diff_soak_report). Enabler: _flat_record_questions — fluency-local questions for single flat record payloads (keys-count + deterministic scalar lookups), the keys-diff surface the soak was blind to (kb payloads are per-block flat dicts). Same pattern as #72's nested-record source; codec/tabularizer rules untouched. Live result (loopback gateway, deepseek-v4-flash + glm-5.2, 3 trials, ~50 real corpus windows): overall worst gap -2% +/-1, depth-5 gap -1% +/-2 — PASS at 5% tolerance, no depth-correlated drift. 471 tests pass (+8), ruff clean.
inth3shadows
added a commit
that referenced
this pull request
Jul 13, 2026
The validation program that kept --diff opt-in is complete — pair fluency (4-model panel 100%), nested-record coverage (#72), and the drift soak (#75: mechanical zero-drift at depth 300, behavioral depth-1..5 PASS on real corpus chains). The default now matches the measured reality. - Policy.diff defaults True; load_policy's doc.get("diff") default flips to match. An explicit policy-file "diff": false is still honored. - proxy: tri-state CLI — --diff forces on (only useful against an opting-out policy file), new --no-diff forces off, neither keeps the policy value. Mutually exclusive, exit 2. multiproxy diff_override becomes Optional[bool] (None = leave each peer's policy alone) so --no-diff is proxy-wide too. - install-mcp: diff param is tri-state — plain wraps write NO diff flag and inherit the proxy default; --diff/--no-diff bake an explicit override into the entry; keyframe-interval no longer requires --diff (but is dropped with --no-diff, where it would be dead weight). - Tests: the two off-by-default pins become on-by-default + policy-false opt-out pins; install-mcp tri-state covered (None drops flags, False bakes --no-diff). 471 pass, ruff clean. - Docs: README tier/Status, TECHNICAL diff bullet (opt-in rationale rewritten as history + the compaction residual keeps --no-diff as escape hatch), USAGE proxy/install-mcp/text-diff sections; also documents the stash hand-edit gotcha found during the #74 rollout. Live smoke: flagless proxy diffs the repeat structure call (117,587 -> 205 chars); --no-diff sends the full form both times.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #71.
Problem
terse fluency --diff(the gate forproxy --diff) generated zero questions forrunecho.structure— so the tool where the diff tier's value actually lives(72–96% cross-call overlap, ~all of the measured +31.7% savings) was behaviorally
unvalidated, even though the harness reported a clean PASS on
locatealone.Root cause (corrected from the issue's original write-up —
extract_recordsdoesrecurse into lists):
runecho.structureis a dict-map keyed by path(
files{"a/x.py": {symbols:[…]}}), andsymbols[]is non-uniform — functions carry{name,kind,line,hash}, imports only{name,kind}._uniform_dict_listrequires anidentical key set, so the list is skipped → no records → no questions.
Fix
Fluency-local
_nested_record_group+_nested_questions:files{}→symbols[]).count/enumerate/lookup(+aggregatewhen a numericcolumn is shared) over the intersection columns (present in every record), scoped to
the first file (
…under files["a/x.py"]…) so non-uniform lists stay well-posed.extract_recordsreturns whichever single file's symbols happen to be uniform, yieldingan ambiguous unscoped "how many records" the model can't answer against a multi-file map.
extract_records/_uniform_dict_list— the tabularizer,probe, and drop-path logic (policy generate: auto-detect drop-to-retrieve candidates (large high-cardinality fields) #47) still see the strict rule unchanged. This only widens
what the fluency harness can ASK about.
Verification
structurepayloads now generate 3 questions each.(diff 100% vs full-terse 94% at n=6×3). Combined with the earlier 4-model panel on
locate(100%/100%), the diff format is fluent on both the low- and high-value surfaces.Scope
Coverage gap only. The
proxy --diffdefault-flip / per-tool policy heuristic is adeliberate follow-up — this PR just makes that decision measurable.