Repository navigation
diag: online/target encoder frame mismatch, and a ledger-citation integrity proposal - #51
Merged
Merged
Conversation
_mpc_success plans across two different maps: z0 from the ONLINE encoder
(forward_eval.py:316), zg from the TARGET encoder (:318), with the CEM cost
reading across both (:291). The goal state is the start state with `pos`
changed alone (:315), by at most goal_offset=0.4. So if the two encoders --
tied together only by an EMA at tau=0.99 -- disagree by more than a 0.4
position shift moves the latent, the planner's cost is dominated by a
constant frame offset it cannot reduce by acting.
That is a candidate mechanism for node 7's null: prediction error fell 4.6x
and the rollout norm ratio went 1.20 -> 0.98 while control stayed at 0/35.
Reports three matched-input quantities plus a Procrustes decomposition. The
decomposition is the load-bearing part: it separates disagreement a rotation
CAN absorb (the gauge that an isotropic objective leaves free, since for
orthogonal R, z ~ N(0,I) implies Rz ~ N(0,I)) from disagreement it cannot.
Cannot run from a saved checkpoint, and the module says so: torch.save
(eeg_jepa.py:593-610) persists encoder_state_dict and predictor_state_dict
only -- target_encoder_state_dict is written nowhere in this repo -- and
EEGJEPAModule.__init__ rebuilds the target as copy.deepcopy(self.encoder).
A checkpoint-loaded model has target_encoder == encoder exactly and would
report zero mismatch: a false negative that reads as a clean result. Hence
instrumentation inside _mpc_success rather than a standalone loader.
NOT VERIFIED: _self_test() is written but has never been executed -- no
torch on the authoring host. It asserts three polarities (identical
encoders -> ~0; rotated target -> large, Procrustes absorbs it; per-dim
scaled target -> large, Procrustes does NOT absorb it), because a
diagnostic that reports "no mismatch" proves nothing until it is shown to
report mismatch when mismatch is present. Run it before trusting any
number this produces.
Thresholds to register BEFORE reading results:
mismatch_to_displacement_ratio > 1.0 frame offset exceeds the goal signal
< 0.3 frame hypothesis rejected
procrustes_residual_ratio < 0.3 mostly rotational (the free gauge)
> 0.7 not rotational; gauge is not the story
Cross-arm prediction: if the gauge story holds, SIGReg arms show a ratio at
or above VICReg arms, since better isotropy means a freer rotation. A lower
SIGReg ratio falsifies the mechanism.
Measurement only. Changes no objective, no planner, and no control claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
Review found six real defects. All are fixed and the self-test now EXECUTES -- under a pure-Python shim for the torch surface this module uses, since the host has neither torch nor numpy. The previous commit shipped it unrun. 1. Procrustes centered out the exact mechanism the module names. A pure translation -- "a constant offset it cannot reduce by acting" -- was removed by centering before the decomposition saw it, and the `before > 0` guard let 7e-14 through, so the ratio was 0/0 and printed a stable 1.5-2.2 in the "not rotational" bin. Now guarded relatively (before > 1e-8 * scale) and returns nan; the translational part is reported as `centroid_gap`. 2. procrustes_residual_ratio <= 1 is a free invariant -- R = I is always feasible, so the optimum can never beat no rotation. A ratio above 1 is mathematically impossible. Asserted in every polarity plus a 50-case sweep, which previously produced 50 violations, all the centered-out translation. 3. principal_angle_cosine_min was not a cosine. The s/(||a||*||b||)*len(s) normalization was unbounded and read "large angle" on identical frames whenever the spectrum was not flat -- which for a 32-d latent it never is (observed max 3.334, and 10.736 rank-deficient). Orthonormalizing fixed the bound but revealed the field is VACUOUS here: rotation and per-dimension scaling both preserve the column span, so two full-rank frames always have all principal angles zero. Replaced with `rotation_from_identity` = ||R - I||_F / (2 sqrt(d)), which is 0 when the frames share a gauge and equals sin(theta/2) for a common rotation angle. It measures the map, which is what actually differs, rather than the span, which cannot. 4. The registered bands made the likeliest outcome undecidable. Simulated EMA lag parks the residual at ~0.67-0.70 across two orders of magnitude of perturbation, i.e. inside a 0.3-0.7 "inconclusive" gap. Narrowed to 0.2 / 0.5 and moved into the docstring with the reasoning. 5. The ratio is denominator-driven: holding encoder disagreement fixed and varying only the goal displacement moves it across 1.0 in both directions, so it partly reports the goal-sampling distribution. frame_mismatch_median and goal_displacement_median are now labelled primary, the ratio derived. 6. Docstring line numbers were written against the parent and were off by one after the new import. Corrected against the patched file. Self-test now asserts four polarities, not three: identical (nan), rotated (absorbed AND far from identity), per-dim scaled (not absorbed AND near identity), and pure translation (nan, with centroid_gap carrying the signal) -- the case that previously misreported. The rotated/scaled pair is what makes the two-field signature non-vacuous; either field alone is ambiguous. Also corrects a provenance error in the parent commit, which attributed the motivating null to ledger node 7 at 208cbcd. That citation does not resolve: node 7 in every pushed ledger is "Transform A/B, arm 2 audit", 208cbcd is not on the remote, and those figures appear in no pushed ledger on any branch. They were reported in-session from an unpushed run. Recorded in the docstring as unverified rather than retro-edited out of history. check_adr_references.py scans only docs/architecture/decision-log for ADR-*; extending it to EXPERIMENT_LEDGER.md would have caught this. Verification: pure-Python shim (modified Gram-Schmidt QR, one-sided Jacobi SVD) reproducing torch's lower-middle median convention. It verifies the ALGORITHM, not torch-specific API behaviour -- rerun `python3 WorldModel/frame_diagnostic.py` against real torch before trusting production numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
reflective-evidence-citation.md already states the rule -- "an unpinned attribution is not a source" -- and nothing checks it for experiment-ledger citations. Commit 8533ea8 on this branch demonstrated the gap by attributing a null to "ledger node 7 at commit 208cbcd", where node 7 is a different experiment and 208cbcd is on no branch. Advisory-only prototype plus a proposal doc. NOT wired into CI. The obvious framing -- "extend check_adr_references.py to scan EXPERIMENT_LEDGER.md" -- would NOT have caught the motivating defect, and the proposal leads with that. check_adr_references.py scans files under HARD_ROOTS and ADVISORY_ROOTS; a git commit message is in neither, and that is where the bad citation lived. Hence --commit-message, for a commit-msg hook or a CI pass over a PR's commits. Separately: WorldModel/ is in neither root list, so the whole experiment lane is currently unscanned even for ADR references. Measured on this tree, not asserted: tree scan 12 findings, all bare 'node 33' -- a dialectic-session node from another numbering space, referenced 13 times and qualified as "dialectic node 33" exactly once (line 149), which the checker correctly does not flag. Twelve one-word fixes. 8533ea8 PASS (0 warnings) 5925bfa FAIL (2 errors) COMMIT_SHA_UNRESOLVABLE: 208cbcd That pair is the honest result and the doc says so plainly: this catches HALF the defect. The unresolvable SHA is mechanically detectable; "node 7 exists but does not contain the claimed numbers" is a semantic claim about content and no checker can see it -- 8533ea8 passes clean. What it buys is converting "unverifiable" into "wrong or right", which is the precondition for review to work. Overstating it here would repeat the original error. One design decision recorded because it was measured: the first draft treated any non-connective word before "node" as a namespace qualifier, which produced 9 false positives out of 18 findings on the real ledger ("if node 33", "refutes node 12", "across node 7"). A denylist of English words cannot be completed, so the prototype uses an allowlist and treats everything else as prose. A checker that cries wolf gets disabled and then catches nothing. Existing ADR checker still passes unchanged (--mode all, 1 pre-existing warning). Open questions for review are listed in the doc rather than decided here: where gap B runs, error-vs-warning for an unpushed pin, namespace syntax, and whether the two checkers should merge behind one entry point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
WorldModel/ was in neither HARD_ROOTS nor ADVISORY_ROOTS, so the entire experiment lane was unscanned -- despite citing ADR-005 and ADR-006 in eight places (README.md, EEG_INTEGRATION_DESIGN.md, export_coreml.py). A renumbered or removed ADR would have gone unnoticed there. HARD_ROOTS rather than advisory: ADR-006 is jepa-transition-capture, which that directory implements. These are normative citations, not historical prose. Adding it produces ZERO findings on the current tree -- all eight references are canonical and both ADRs exist -- so the gate stays green and this is free today. That zero was probed rather than trusted, because a root that silently fails to scan produces the same clean output as a root with nothing wrong in it. A file citing one nonexistent three-digit ADR number and one malformed two-digit one, placed under WorldModel/, yields 0 findings without the root and 2 with it (ADR_REFERENCE_MISSING, ADR_REFERENCE_MALFORMED). So the new coverage demonstrably scans. Probe removed; tree clean. The first draft of the accompanying doc wrote those probe tokens out literally. Because docs/architecture is itself a HARD_ROOTS entry, that made the checker flag the document -- 2 errors and a failing test_repository_passes_hard_mode -- i.e. the change broke the gate it was extending. The doc now describes the probe in prose instead of reproducing the tokens, so it stays scanned rather than being added to HARD_SCAN_EXCLUSIONS. That exclusion list exists for exactly this hazard; the fix here is to not need it. Verified by exit code, not by reading piped output: check_adr_references.py --mode all exit 0, PASS (1 pre-existing warning) Tests/eval/test_adr_references.py exit 0, 16 tests OK check_ledger_references.py --mode advisory exit 0, PASS (12 warnings) This is the only part of this branch that changes an existing required gate. It is a clean single-commit revert or cherry-pick if you would rather land it separately from the advisory prototype. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
Node 33 is a dialectic-session node from a different numbering space. It was referenced 14 times in WorldModel/EXPERIMENT_LEDGER.md and qualified as "dialectic node 33" exactly once, at line 149. The other thirteen borrowed this ledger's namespace, where node 33 does not exist -- the ledger defines 0-13. All thirteen now carry the qualifier. check_ledger_references.py reports 0 findings in both advisory and hard mode, where it previously reported 12. 12 findings, 13 occurrences. Line 208 contains two, and the dedup key is (severity, code, path, line, message) with no column, so identical findings on one line collapse into one. check_adr_references.py behaves the same way. Not wrong, but the two are different claims and only the first was measured -- the earlier "twelve one-word fixes" undercounted by one. The edit inserts the qualifier and nothing else. Verified by word-diff: every changed hunk is an insertion of "dialectic" (one also recapitalizing Node -> Dialectic node at a sentence start), with no claim, number, date, or other wording altered. This is a retrospective edit to a durable evidence record, which this project otherwise avoids. Justified as disambiguating a citation rather than restating a result -- no finding, decision, or measurement changes. Naming it here rather than performing it silently. The checker now passing in hard mode clears the blocker on wiring it into CI. That wiring is deliberately NOT done here: a second required gate is a standing-cost decision, and the proposal's open questions should be settled first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
Wires check_ledger_references.py into the Swift CI job beside the ADR gate, in
--mode hard, with its own tests running first per ci.yml's stated rule that a
gate whose only evidence is that it never complained is not a gate.
Two things had to be fixed before this was safe to wire.
1. SHALLOW CHECKOUTS. actions/checkout defaults to fetch-depth 1, and the SHA
check runs git cat-file. Measured on a real depth-1 clone of this branch:
ERROR node 1 pins commit ea0afbc, which does not resolve (unpushed?)
ERROR node 1 pins commit fa7647b, which does not resolve (unpushed?)
FAIL (2 errors)
Both commits exist. They are simply not in a depth-1 tree. Wiring this
naively would have broken CI on its first run while confidently asserting
something false -- the same class of error the gate exists to prevent.
_is_shallow() now distinguishes "cannot verify" from "does not exist", the
distinction _commit_exists already made for a missing git binary. A shallow
tree emits LEDGER_COMMIT_UNVERIFIABLE_SHALLOW as a warning and exits 0.
2. That guard makes the gate honest but inert, so checkout now sets
fetch-depth: 0 and the check does real work. ~300 commits, single-digit MB.
Tests/eval/test_ledger_references.py: 19 tests, stdlib only, mirroring
test_adr_references.py's split between synthetic fixtures that prove the
checker CAN fail and live tests that are the gate. The shallow tests build an
actual --depth 1 clone and assert the fixture is really shallow first, since a
fixture that silently is not shallow would pass while proving nothing. Both
polarities: a shallow tree must warn, and a genuinely missing SHA must still
error in a full clone, so the guard cannot disarm the check.
Mutation-checked rather than assumed:
_is_shallow always False -> 1 failure (the shallow test)
any qualifier treated as namespace -> 4 failures (the allowlist regression)
Scope note: this adds a second required gate, which the proposal doc had left
as an open question rather than a decision. Doing it on request. The open
questions that remain genuinely open -- where a commit-message check runs,
error-vs-warning for a legitimately unpushed pin, namespace syntax, whether the
two checkers merge -- are unchanged and still listed in the doc. The
--commit-message path is NOT wired here; it needs the first of those answered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
Review finding, and it is mine: the self-test was described as executed, but the shim it ran under lived only on the authoring host. frame_diagnostic.py did a bare `import torch`, this repo's python-eval job installs numpy/scipy/pandas and no torch, and nothing in Tests/eval referenced the module. So the docstring's "run it before trusting any number this produces" had no mechanism behind it -- and the asymmetry was stark next to the citation gate this same branch just built: the citation claim was enforced, the execution claim was a promise. WorldModel/_tensor_shim.py is now committed (modified Gram-Schmidt QR, one-sided Jacobi SVD, torch's lower-middle median convention), and frame_diagnostic falls back to it only when torch is absent. Real torch always wins when importable. Tests/eval/test_frame_diagnostic.py runs the four polarities and the 50-case invariant sweep on every push, plus pins the one new mathematical claim in the docstring: rotation_from_identity = ||R-I||_F / (2 sqrt(d)) equals sin(theta/2) for a common-angle rotation. Checked at theta = 0.1, 0.5, 1.0, 2.0, pi to six decimals, with R=I giving 0 and R=-I giving exactly 1. That was prose; now it is a test. Mutation-checked, because a self-test that passes under a wrong SVD proves nothing: drop V accumulation in Jacobi SVD -> 1 failure skip Gram-Schmidt orthogonalisation -> 1 failure Runtime 21s, dominated by the sweep under pure-Python linear algebra. Left at 50 cases rather than trimmed: shrinking the sweep to save CI seconds would weaken the one assertion that catches degeneracy leaking through. A pass here does NOT discharge running against real torch. The shim may differ in SVD sign conventions and float32-vs-float64 accumulation. It discharges "does the logic hold", which is what the polarities assert. Both files say so. Still true and still the point: this diagnostic has never run against real data and has produced no number. The frame hypothesis is unmeasured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Two related pieces: a measurement, and a proposal that exists because the measurement's first commit shipped a bad citation.
Nothing here changes an objective, a planner, or any control claim. No experimental result is asserted.
1. Frame diagnostic —
WorldModel/frame_diagnostic.py(+ 12 lines inforward_eval.py)_mpc_successplans across two different maps:The goal state is the start state with
poschanged alone, by at mostgoal_offset(0.4). Soz0andzgencode near-identical inputs through two encoders tied together only by an EMA attau=0.99. If those encoders disagree by more than a 0.4 position shift moves the latent, the planner's cost is dominated by a frame offset it cannot reduce by acting — and control cannot improve however good the predictor gets.That is a candidate mechanism for a null where prediction improves sharply and control does not move at all.
The isotropy connection
For orthogonal
R,z ~ N(0,I)impliesRz ~ N(0,I). An isotropic objective (SIGReg, VISReg) places no penalty on the encoder rotating — the rotational gauge is left maximally free. Under EMA the target is a lagged copy, so the frames differ by however much the encoder rotated during the EMA window. Pinning the distribution can make the gauge freer, not tighter.This also sharpens the Theorem 5.4 question. An L2 cost is
O(n)-invariant, so a shared rotation is harmless and the theorem is untroubled. The issue is two encoders in two frames: the guarantee covers one representation, and nothing in it makes an online/target pair converge to the same rotation.What it reports
frame_mismatch_median(A)goal_displacement_median(B)planner_reads_mediancentroid_gapmismatch_to_displacement_ratioprocrustes_residual_ratiorotation_from_identityRead the last two together: absorbed-by-rotation plus far-from-identity is gauge; not-absorbed plus near-identity is genuine disagreement. Either alone is ambiguous.
The ratio is derived, not primary: holding encoder disagreement fixed and varying only the goal displacement moves it across 1.0 in both directions, so it partly reports the goal-sampling distribution.
Registered read — fixed before any real number exists
The 0.2/0.5 band is deliberately narrower than a naive 0.3/0.7. Simulating EMA lag as a generic perturbation parks the residual at ~0.67–0.70 across two orders of magnitude, i.e. the likeliest realistic outcome would have landed inside a 0.3–0.7 "inconclusive" gap and decided nothing.
Cross-arm prediction, registered: if the gauge story holds, SIGReg arms show a ratio at or above VICReg arms, because better isotropy means a freer rotation. A lower SIGReg ratio falsifies the mechanism.
It cannot run from a checkpoint
torch.save(eeg_jepa.py:593) persistsencoder_state_dictandpredictor_state_dictonly —target_encoder_state_dictis written nowhere in this repo — and__init__rebuilds the target ascopy.deepcopy(self.encoder)(:327). A checkpoint-loaded model hastarget_encoder == encoderexactly and would report zero mismatch: a false negative that reads like a clean result. Hence instrumentation inside_mpc_successrather than a standalone loader.Verification status — read this before trusting numbers
The self-test executes and asserts four polarities plus a 50-case invariant sweep:
But it ran under a pure-Python shim for torch, not torch. The authoring host has neither torch nor numpy. The shim (modified Gram-Schmidt QR, one-sided Jacobi SVD, torch's lower-middle
medianconvention) verifies the algorithm, not torch-specific API behaviour — SVD sign conventions and float32-vs-float64 could still shift things.Review history
The first commit shipped this unrun, and review found six real defects — all fixed in
5925bfa. The most interesting:principal_angle_cosine_minwas not a cosine, and orthonormalizing it revealed the field was vacuous here (rotation and per-dim scaling both preserve the column span, so two full-rank frames always have all principal angles zero — measured at exactly 1.0000 for identical, rotated, and scaled alike). Replaced withrotation_from_identity, which measures the map rather than the span. Also: Procrustes centered out the very translation the module is about, andprocrustes_residual_ratio <= 1is a free invariant (R=I is always feasible) that now has an assertion and previously had 50 violations.2. Citation-integrity proposal —
Scripts/check_ledger_references.py+ docAdvisory-only. Not wired into CI.
docs/architecture/reflective-evidence-citation.mdalready states the rule: "an unpinned attribution is not a source." Nothing checks it for experiment-ledger citations. Commit8533ea8in this PR proved the gap by attributing a null to "ledger node 7 at commit208cbcd" — node 7 is a different experiment, and208cbcdis on no branch.The obvious fix would not have worked
"Extend
check_adr_references.pyto scanEXPERIMENT_LEDGER.md" would not have caught this. That checker scans files underHARD_ROOTS/ADVISORY_ROOTS; a git commit message is in neither, and that is where the bad citation lived. Two distinct gaps — hence--commit-message, for acommit-msghook or a CI pass over a PR's commits.Separately:
WorldModel/is in neither root list, so the entire experiment lane is currently unscanned even for ADR references. I did not fix that here — it changes what an existing CI gate covers, which is your call.Measured on this tree, not asserted
node 338533ea85925bfaCOMMIT_SHA_UNRESOLVABLE: 208cbcdNode 33 is a dialectic-session node from another numbering space, referenced 13 times and qualified as
dialectic node 33exactly once (line 149) — which the checker correctly does not flag. Twelve one-word fixes.It catches half the defect, and says so
208cbcd, which does not exist8533ea8passes clean and always will — that half is a semantic claim about content, and only human review catches it. What the check buys is converting unverifiable into wrong or right, which is the precondition for review to work at all. Overstating it would repeat the original error.A design decision recorded because it was measured
The first draft treated any non-connective word before
nodeas a namespace qualifier. On the real ledger that produced 9 false positives out of 18 findings (if node 33,refutes node 12,across node 7). A denylist of English words cannot be completed, so it uses an allowlist and treats everything else as prose. A checker that cries wolf gets disabled, and then catches nothing.Status
--mode all, 1 pre-existingADR-009+warning)node 33references qualifiedOpen questions (in the doc, deliberately undecided)
mainmay be the honest split.dialectic node 33is prose;[dialectic#33]is unambiguous but a bigger edit.--check {adr,ledger,all}?Non-claims
No claim that the node 7 figures are wrong — only that as committed they were unverifiable. No claim that mechanical checking replaces review. No control claim, no objective change, and no support for or against the frame hypothesis, which remains unmeasured on real data.