Real-embeddings backend at the swap point + paraphrase probe — with a measured negative result

[Micka420-collab]

What

Makes the README's declared featurizer swap point real and measures whether off-the-shelf sentence embeddings can support free-form goal language. Spoiler: they can't (at this model scale), and the numbers below are the interesting part of this PR.

Infrastructure

• make_featurizer factory — model.text_features: hashing (default, bit-identical behavior) or stransformer (sentence-transformers, optional embeddings extra, frozen + cached).
• build.py now derives the bridge's language dim from the featurizer (384 for MiniLM vs lang_dim for hashing) — checkpoints rebuild correctly either way.
• scripts/paraphrase_probe.py — guided success when the planner phrases the goal freely (6 templates, incl. swapped word order) instead of the canonical text. Same pattern as wm_probe.py: never touches the judge.
• 4 new tests (55 total); ST tests auto-skip without the dependency.

The measured finding (mechanism level)

Cosine similarities vs canonical goal "row 2 col 3", gridworld text format:

Comparison	hashing+bigram	paraphrase-MiniLM-L3	all-MiniLM-L6
transposed goal "row 3 col 2" (must be FAR)	0.81	1.000	0.999
other goal "row 5 col 0" (must be FAR)	0.70	0.956	0.963
paraphrase of the SAME goal (must be CLOSE)	0.44	0.556	0.744

Small sentence-embedding models cluster by surface template, not by coordinates — they are worse than the hashing stand-in on the property that actually drives control (goal discrimination), and their paraphrase advantage doesn't discriminate either (same-goal 0.744 vs other-goal 0.706 for MiniLM-L6). This is a scale-independent pattern across the two models tested, not a fixable hyperparameter.

Implication for the frozen-LLM story: free-form language should be normalized in the planner (LLM classifies intention → goal key → canonical text — the path that already exists via goal_menu/_classify_llm), not in the embedding space. The probe + backend make this trade-off measurable for any future featurizer candidate.

End-to-end confirmations

Running now; will be posted as a comment: (a) stransformer on gridworld canonical (predicted: guided collapse from transposed aliasing), (b) paraphrase probe on hashing (predicted: strong degradation off-template).

🤖 Generated with Claude Code

https://claude.ai/code/session_018FBEJut651qfyEWR5A3shP

---

Generated by Claude Code

[Micka420-collab]

End-to-end confirmations are in — and both mechanism-level predictions were wrong, in instructive ways. Guided success on held-out gridworld episodes (30/seed):

Backend	Canonical goals	Paraphrase retention (5 templates)
hashing+bigram (default)	1.000 ± 0.000 (2 seeds)	0.957 — 96% of canonical (worst template 0.917; swapped word order 0.950)
stransformer (MiniLM-L3)	0.933 (probe) / 0.90 (judge run, score 0.792)	0.813 — 87% of canonical (worst 0.733)

What the cosine geometry got wrong, in both directions:

1. ST's transposed-goal cosine of 1.000 predicted a guided collapse — it didn't happen (0.90–0.93). The ground loss is an MSE regression that amplifies tiny but consistent embedding differences; even 0.956–0.999 cosines leave enough signal in 384 dims.
2. Hashing's same-goal-paraphrase cosine of 0.44 predicted paraphrase failure — also didn't happen (96% retention). The load-bearing tokens and bigrams (row_2, col_3, the digits) survive most rephrasings, including swapped order.

Final verdict: the default hashing+bigram space wins both axes end-to-end — canonical control (1.00 vs 0.93) and paraphrase robustness (96% vs 87%). The stransformer backend stays available as an opt-in swap-point implementation and as the measurement harness for any future embedding candidate, but there is no case for switching defaults. The deeper lesson is the repo's own thesis vindicated a third time (after CEM and multistep_k): mechanism-level geometry — in either direction — is not a substitute for the held-out measurement.

Also worth noting: the probe shows the current pipeline already tolerates freely-phrased goals at 96% — the frozen-LLM story is in better shape than the "canonical text only" caveat suggested.

---

Generated by Claude Code