From 0a6f326cbaf2fbf6dc226af0114e16b69a797d3f Mon Sep 17 00:00:00 2001 From: aurascoper Date: Sun, 2 Aug 2026 01:25:37 -0500 Subject: [PATCH] =?UTF-8?q?bench(worldmodel):=20nuisance=20jitter=20fails?= =?UTF-8?q?=20structurally=20=E2=80=94=20reverted,=20recorded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ledger node 20. Node 19's preferred generator repair: jitter chi/offset between the pre- and post-window so predicting the held factors is no longer free. Acceptance criteria G1-G4 registered BEFORE implementing; depth fixed at 0.10, the constant _render_state already uses, chosen by symmetry and explicitly not tuned. Registered rule: fail any gate = revert, do not adjust the constant. ALL FOUR GATES PASSED. vel exact to 2.33e-15 (jitter cancels in the ratio); raw probe recovers vel at 0.3605; ladder bit-identical (100.0 / 91.5 / 73.5 / 37.0 / 34.5); alphaPower post-from-pre fell 1.000 -> 0.9709; chi still 0.6653. AND THE OUTCOME REGRESSED. Paired, 3 seeds, identical apart from the constant: pos +0.6065 -> +0.3118 -0.2947 vel +0.0227 -> -0.0280 -0.0508 chi +0.9566 -> +0.9306 -0.0260 peak_amp +0.9947 -> +0.9814 -0.0133 ax -0.0467 -> -0.3264 -0.2797 ay -0.2305 -> -0.4947 -0.2642 It degraded the TASK factors 10-20x more than the NUISANCE factors -- the exact inverse of its intent. MECHANISM: the task factors are multiplicative modulations ON the nuisance carriers. c0 = alpha*(1 + 0.10*pos), alpha = f(chi, peak_amp, offset). Carrier sd 0.220 against a total pos contribution of 0.0270 -- leverage 8.2:1. A 10% carrier perturbation injects 0.0259, i.e. 96% of the entire task signal. The intervention put noise straight into the channel carrying the signal. NO USABLE DEPTH EXISTS. Depth vs G3 vs injected-noise-to-signal: 0.01 -> 0.9997 (G3 fails), 0.10x; 0.05 -> 0.9932 (fails), 0.48x; 0.10 -> 0.9731 (passes), 0.96x; 0.20 -> 0.8953 (passes), 1.97x. The minimum depth that removes the subsidy already injects noise the size of the whole task signal. Structural, not mistuned. REVERTED per the registered rule, constant untouched. synthetic_1f.py is unchanged on main; this node is the record. Node 19's stated advantage of repair (2) over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it. THE GATE DESIGN WAS INSUFFICIENT, and that is the transferable part. G1-G4 verified the information is present in the raw observation and the control task is unchanged. Neither is the same as "a model can still learn it" -- a repair can pass every raw-signal check and still halve what the encoder recovers. Any future generator change must add G5: factor recovery for pos and vel under the INCUMBENT objective must not degrade. Training the incumbent is permitted; node 14 excludes candidate arms, not the existing baseline. Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap --- WorldModel/EXPERIMENT_LEDGER.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 00be826..ffa2f92 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -363,3 +363,28 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - **Velocity is not destroyed.** `vel = 10 * (c1 / betaPower - 1)` exactly — max error 2e-15 over 4000 samples. It is a ratio of two observed channels: nonlinear, hence invisible to the linear probe that caps at R^2 = 0.033, but trivially within an MLP's reach. Nothing prevents the encoder learning it; nothing rewards it. - Decision: **CLOSE the representation-objective line as specified.** No representation objective can encode a variable the generator made 0.01% of the observed variance while paying a variance penalty for ignoring the other 99.99%. This is a generator design fault, not an encoder, planner, objective or benchmark fault. Nodes 6-18 were measuring the wrong thing correctly. - Next question: repair the generator, under node-14 blind discipline. Two candidates, NOT equivalent: **(1)** raise the velocity modulation depth — cheapest, but it is a knob, and tuning it until an arm wins is unfalsifiable; **(2)** let `chi`/`offset` vary within-trajectory — removes the free-prediction subsidy that makes the nuisance attractive without touching velocity's coefficient, and changes what the objective is REWARDED for rather than what it is SHOWN. Prefer (2). Either way: specify using only oracle/random/zero, never a candidate arm, and require the repaired generator to reproduce the existing ladder ordering (oracle > ay-oracle > ax-oracle > zero ~ random ~ MPC) before any arm is rerun. Add to the acceptance gate an assertion that a probe on the RAW observation recovers `vel` above a floor, so a future generator edit cannot silently reintroduce this. + +## 20 — Generator repair attempt: within-transition nuisance jitter FAILS, structurally (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 19) `chi`/`peak_amp`/`offset` are held constant per transition, so predicting them at t+1 is exact and free; the JEPA loss is therefore minimised by encoding the nuisance. Jittering `chi`/`offset` between the pre- and post-window should remove that subsidy WITHOUT touching velocity's coefficient — changing what the objective is REWARDED for rather than what it is SHOWN. Node 19 preferred this over raising the velocity depth precisely because the latter is a knob that could be tuned until an arm wins. +- Prediction: acceptance criteria G1-G4 registered BEFORE implementing. Jitter depth fixed at **0.10 — the constant `_render_state` already uses for the pos/vel modulation**, chosen by symmetry with the file's own value, explicitly not tuned. Registered rule: FAIL any gate = revert, do not adjust the constant. +- Implementation: `_jitter_held` perturbing `chi` and `offset` multiplicatively for RENDERING ONLY, drawn independently for the pre- and post-windows. `_step` untouched, so dynamics, reachable set and the exact oracle are identical. `peak_amp` deliberately left fixed, matching node 19's scope. +- Evidence: + - **All four gates PASSED.** G1a `vel = 10*(c1/betaPower - 1)` exact to 2.33e-15 (jitter cancels in the ratio, as predicted). G1b raw-observation linear probe recovers `vel` at R^2 0.3605 (>= 0.30; was 0.386). G2 ladder bit-identical — ORACLE full 100.0% > ay 91.5% > ax 73.5% > zero 37.0% ~ random 34.5%. G3 `alphaPower` post-from-pre fell 1.000 -> 0.9709 (< 0.99). G4 `chi` still recoverable at 0.6653 (> 0.50). + - **And the outcome was a clear REGRESSION.** Paired, 3 seeds each, identical apart from the constant: + + | metric | jitter=0 | jitter=0.10 | delta | + |---|---|---|---| + | `pos` | +0.6065 | +0.3118 | **-0.2947** | + | `vel` | +0.0227 | -0.0280 | -0.0508 | + | `chi` | +0.9566 | +0.9306 | -0.0260 | + | `peak_amp` | +0.9947 | +0.9814 | -0.0133 | + | `ax` | -0.0467 | -0.3264 | -0.2797 | + | `ay` | -0.2305 | -0.4947 | -0.2642 | + + It degraded the TASK factors by 10-20x more than the NUISANCE factors — the exact inverse of its intent. `pos` halved; `chi` and `peak_amp` barely moved. + - **MECHANISM: the task factors are multiplicative modulations ON the nuisance carriers.** `c0 = alpha * (1 + 0.10*pos)` with `alpha = f(chi, peak_amp, offset)`. Carrier sd is **0.220**; the entire `pos` contribution is **0.0270** — a leverage of **8.2 : 1**. Perturbing the carrier by 10% injects 0.0259, i.e. **96% of the whole task signal**. The intervention put noise directly into the channel carrying the signal. + - **NO USABLE DEPTH EXISTS — this is structural, not mistuned.** Sweeping depth against G3 and against the injected-noise-to-task-signal ratio: 0.01 -> R^2 0.9997 (G3 fails), noise 0.10x; 0.05 -> 0.9932 (fails), 0.48x; 0.10 -> 0.9731 (passes), 0.96x; 0.20 -> 0.8953 (passes), 1.97x. **The minimum depth that removes the subsidy already injects noise the size of the entire task signal.** There is no window. +- Decision: **REVERTED, per the registered rule, without adjusting the constant.** `synthetic_1f.py` is unchanged on main; this node is the record. **REJECT node 19's preferred repair (2) — not as mistuned but as structurally unavailable**, and note that its stated advantage over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it, so (1) is now the better of the two even though it remains a knob. +- **THE GATE DESIGN WAS INSUFFICIENT, and that is the transferable lesson.** G1-G4 verified that the information is present in the raw observation and that the control task is unchanged. Neither is the same as "a model can still learn it". A repair can pass every raw-signal check and still halve what the encoder recovers. **Add G5 to any future generator change: factor recovery for `pos` and `vel` under the INCUMBENT objective must not degrade.** Training the incumbent is permitted under the blind rule — node 14 excludes candidate arms, not the existing baseline — and G5 would have caught this before the outcome measurement. +- Next question: the fault is the generator's STRUCTURE, not its constants. Task factors are small multiplicative modulations riding on large nuisance carriers, so any perturbation of the carrier is amplified 8.2x relative to the signal, and any attempt to suppress the nuisance suppresses the task with it. The candidate is therefore structural: give `pos` and `vel` their OWN additive observation channels, independent of `alpha`/`beta`, so the task signal is not a rounding error on a nuisance carrier and the two can be perturbed independently. That is a larger change than either node-19 candidate and needs its own pre-registration, including G5.