protocol v2.1: re-pin the engine, and make preview_play the engine again

CITATION.cff

@@ -4,7 +4,7 @@ title: "Jackhammer: a paired Balatro agent benchmark"
 type: software
 authors:
   - name: "idIing"
-version: 0.1.0
-date-released: 2026-09-02
+version: 1.1.0
+date-released: 2026-09-05
 license: MIT
 repository-code: "https://github.com/idIing/jackhammer"

CONTRIBUTING.md

@@ -49,4 +49,7 @@ expectation. Generally useful fixes should go upstream to Jackdaw when practical
 
 Keep changes focused. Include the commands you ran and their outcomes. Do not mix a protocol change
 with an agent result or unrelated cleanup. Changes to the engine pin, v1 battery, primary metric,
-holdout rule, baseline slate, or evaluation procedure require a new protocol version.
+holdout rule, baseline slate, or evaluation procedure require a new protocol version — a point
+release (v2.*x*) when the contract is untouched and only the numbers move, a major one when a
+clause changes meaning. [The protocol](docs/protocol-v2.md) states the rule and records what each
+release moved.

README.md

@@ -24,14 +24,19 @@ install -> run a baseline -> add an agent -> compare on identical seeds -> inspe
 |---|---|---|---:|
 | Random Legal | `random-legal` | Samples a legal action type uniformly, then a legal target; no tactical layer | 1.000 |
 | Random Shop | `random-shop` | `GreedyTactical` card play; random legal shop actions | 1.637 |
-| Cheapest-Joker Shop | `greedy-shop` | `GreedyTactical` card play; buys the cheapest affordable Joker | 3.204 |
+| Cheapest-Joker Shop | `greedy-shop` | `GreedyTactical` card play; buys the cheapest affordable Joker | 3.196 |
 
-Those are the complete 240-seed v2 results on the `idIing/jackdaw-balatro` fork of Jackdaw at
-[`4d6f19d`](https://github.com/idIing/jackdaw-balatro/commit/4d6f19d9fe73f96603412a59ad5eab16d08937e7),
+Those are the complete 240-seed v2.1 results on the `idIing/jackdaw-balatro` fork of Jackdaw at
+[`de733eb`](https://github.com/idIing/jackdaw-balatro/commit/de733ebd494a5da71fb7049b3b6b18ecd039786c),
 the exact commit the lockfile installs; all three went 0/240 on wins. The stable ID `greedy-shop`
 predates the clearer display name and is retained so existing artifacts stay comparable. The paired
-Random Shop -> Cheapest-Joker Shop difference is +1.567 ante with a bootstrap-95 interval of
-`[+1.400, +1.729]`.
+Random Shop -> Cheapest-Joker Shop difference is +1.558 ante with a bootstrap-95 interval of
+`[+1.396, +1.721]`.
+
+v2.1 re-pinned the engine and fixed four `game_state` keys `preview_play` handed the scorer inside
+`GreedyTactical`'s ranking; only `greedy-shop` moved, and by 0.008 ante. Artifacts stamped
+`jackhammer/v2` were produced at the previous pin and are not directly comparable — see
+[what changed](docs/protocol-v2.md#what-changed-in-v21).
 
 ### The shared card play: `GreedyTactical`
 
@@ -47,12 +52,12 @@ it at all, which is what makes it an honest floor rather than a third variation
 The 300 cap is exhaustive for a standard 8-card hand, where the complete set of ≤5-card subsets is
 218. Above 8 cards it truncates, and it drops the largest subsets first. It also does not bind
 equally on the two arms: shopping grows the hand, so `greedy-shop` truncates ~2.4x as often, and the
-`+1.567` understates the contrast by about 0.1 ante — see [known limits](docs/known-limits.md).
+`+1.558` understates the contrast by about 0.1 ante — see [known limits](docs/known-limits.md).
 
 This ladder is intentionally weak. `greedy-shop` does not understand Joker text, quality, rarity,
 or synergy; it never rerolls or sells, it does not buy vouchers or consumables, and it never opens a
 booster pack. Over the whole battery it emits 6 of the engine's 21 action types; `random-shop` emits
-16. So the `+1.567` prices buying the cheapest Joker at all against an arm that touches most of the
+16. So the `+1.558` prices buying the cheapest Joker at all against an arm that touches most of the
 shop at random — the narrowest agent on the slate beating the widest one. Every run records that
 histogram and the evaluator checks it against each baseline's published description
 ([declared repertoires](docs/known-limits.md#declared-repertoires)). That gap is an open
@@ -74,7 +79,7 @@ To use the kit as a library from your own project instead, install it — the fr
 the pinned simulator commit come with it, so a benchmark run from an install is attributable:
 
 ```bash
-pip install "jackhammer-benchmark @ git+https://github.com/idIing/jackhammer.git@v1.0.0"
+pip install "jackhammer-benchmark @ git+https://github.com/idIing/jackhammer.git@v1.1.0"
 ```
 
 ```python
@@ -85,7 +90,7 @@ from jackhammer.playground.seeds import load_battery
 spec = agents.get("greedy-shop")
 results = run_battery_with(load_battery("train")[:8], spec.make_decider, out_path="runs.jsonl",
                            config_label=spec.name, slot1=spec.slot1, slot2=spec.slot2)
-print(provenance.kit_pin())   # {'version': '1.0.0', 'commit': None, 'dirty': None}
+print(provenance.kit_pin())   # {'version': '1.1.0', 'commit': None, 'dirty': None}
 ```
 
 `scripts/` is not installed; the CLI below runs from a clone. An installed copy reports
@@ -108,16 +113,16 @@ uv run python scripts/evaluate.py \
   --agent greedy-shop --vs random-shop --limit 8 --workers 4
 ```
 
-Remove `--limit 8` for the reportable v1 comparison over all 240 seeds. The command writes raw
+Remove `--limit 8` for the reportable comparison over all 240 seeds. The command writes raw
 JSONL decision records, one result artifact per arm, and a paired comparison under `data/bench/`.
 The reportable run prints (artifact paths omitted here):
 
 ```text
 greedy-shop: 240 runs
-  mean highest ante: 3.204
+  mean highest ante: 3.196
 random-shop: 240 runs
   mean highest ante: 1.637
-paired (240 seeds): greedy-shop - random-shop = +1.567 ante [+1.400, +1.729] boot-95
+paired (240 seeds): greedy-shop - random-shop = +1.558 ante [+1.396, +1.721] boot-95
   interval excludes zero
 ```
 
@@ -143,8 +148,8 @@ the closest baseline with `--vs`.
 
 ## Add a seed dataset
 
-Protocol v2 always means the committed 240-seed `train` split. New coverage, stress, curriculum, or
-seed-difficulty questions belong in versioned sidecar manifests:
+The headline protocol always means the committed 240-seed `train` split. New coverage, stress,
+curriculum, or seed-difficulty questions belong in versioned sidecar manifests:
 
 ```bash
 uv run python scripts/evaluate.py \
@@ -153,7 +158,7 @@ uv run python scripts/evaluate.py \
   --split sample
 ```
 
-Custom datasets are stamped as `jackhammer/dataset-eval/v2`, never `jackhammer/v2`. That makes
+Custom datasets are stamped as `jackhammer/dataset-eval/v2.1`, never `jackhammer/v2.1`. That makes
 future questions such as “how does seed coverage affect measured agent strength?” additive without
 silently moving the headline benchmark. See [Adding datasets](docs/datasets.md).
 

docs/datasets.md

@@ -32,8 +32,8 @@ uv run python scripts/evaluate.py \
   --dataset path/to/dataset.json --split sample
 ```
 
-Artifacts from this route use protocol `jackhammer/dataset-eval/v2` and dataset scope `diagnostic`.
-They cannot be paired with `jackhammer/v2` artifacts.
+Artifacts from this route use protocol `jackhammer/dataset-eval/v2.1` and dataset scope `diagnostic`.
+They cannot be paired with `jackhammer/v2.1` artifacts.
 
 ## Coverage studies
 

docs/known-limits.md

@@ -22,8 +22,8 @@
   `_open_tag_pack` for a tag's pack — and over the whole battery this agent emits neither
   `OpenBooster` nor `SkipBlind` nor any pack action, so the pack-choice branch of `decide_shop` is
   unreachable in practice. Over the published battery it emits **6 of the engine's 21 action types
-  across 15,349 decisions**, with zero fallback substitutions, against 16 for `random-shop`.
-  The paired `+1.567` is a clean A/B, but what it prices is *buying the cheapest Joker at all*
+  across 15,307 decisions**, with zero fallback substitutions, against 16 for `random-shop`.
+  The paired `+1.558` is a clean A/B, but what it prices is *buying the cheapest Joker at all*
   against an arm that touches most of the shop surface at random — the headline is not "greedy beats
   random" but "the narrowest agent on the slate beats the widest one". See
   [declared repertoires](#declared-repertoires) below.
@@ -34,10 +34,10 @@
   [the tactical scan cap](#the-tactical-scan-cap) below.
 - **Scan-cap asymmetry:** the shared tactical layer is not automatically a symmetric control.
   `greedy-shop` truncates 2.4x as often as `random-shop`, and re-running the battery with the cap
-  raised moves the paired delta from `+1.567` to `+1.667` — a paired difference-of-differences of
+  raised moves the paired delta from `+1.558` to `+1.658` — a paired difference-of-differences of
   **`+0.100 [+0.046, +0.167]`** boot-95, which excludes zero, so the published number slightly
-  *understates* the shop contrast. `+1.667` still falls inside the published interval
-  `[+1.400, +1.729]`, so no v1 conclusion changes.
+  *understates* the shop contrast. `+1.658` still falls inside the published interval
+  `[+1.396, +1.721]`, so no published conclusion changes.
 - **Published battery:** the 240 training seeds are public and therefore overfittable. The old
   validation split has already been consumed and is retired, not a reusable secret leaderboard.
 - **No live client:** the benchmark and text run inspector work headlessly. This repository does not
@@ -46,6 +46,9 @@
   relabeled as ordinary-distribution policy strength. Sampling and overlap must be reported.
 - **Artifact schema:** v1 validates the stable envelope and preserves the raw decision records by
   reference; it does not cryptographically sign results or fully validate every nested summary field.
+- **The preview is not the engine on every board.** `preview_play` sets every key `score_hand`
+  reads directly, and v2.1 fixed the four it got wrong — but it builds a *partial* mirror of the
+  live state, and engine code other than the scorer runs against that mirror. See below.
 
 ## Declared repertoires
 
@@ -86,12 +89,12 @@ agent cannot drift apart silently. The prose around it is still prose.
 | `random-shop` | 16 | `PlayHand` · `Discard` · `SelectBlind` · `CashOut` · `Reroll` · `NextRound` · `SkipPack` · `BuyCard` · `SellJoker` · `SellConsumable` · `UseConsumable` · `RedeemVoucher` · `OpenBooster` · `PickPackCard` · `SwapJokersLeft` · `SwapJokersRight` |
 <!-- END declared-repertoire -->
 
-Measured on the 240-seed `train` split at the pinned engine, protocol v2: `greedy-shop` 15,349
+Measured on the 240-seed `train` split at the pinned engine, protocol v2.1: `greedy-shop` 15,307
 decisions, `random-shop` 9,157, `random-legal` 5,713, all three with **zero** fallback
 substitutions. The reference agent's six, in full:
 
 ```
-PlayHand 4,527 · Discard 3,814 · SelectBlind 2,132 · CashOut 1,892 · NextRound 1,892 · BuyCard 1,092
+PlayHand 4,508 · Discard 3,803 · SelectBlind 2,128 · CashOut 1,888 · NextRound 1,888 · BuyCard 1,092
 ```
 
 `scripts/evaluate.py` prints the same line for whatever agent it runs, and `summary.repertoire` in
@@ -107,6 +110,31 @@ separate claim about them. Two things a reader should not over-read:
   `SelectBlind` and `CashOut` calls v2 hands back to the agent. Repertoire counts are only
   comparable within one protocol version.
 
+## The preview is not the engine on every board
+
+`GreedyTactical` ranks candidate plays with `preview_play`
+(`src/jackhammer/playground/exact_score.py`), which reconstructs by hand the synthetic `game_state`
+the engine's `_handle_play_hand` passes to `score_hand`, and then calls the engine's own scorer.
+Protocol v2.1 repaired the four keys the scorer reads that it had wrong, and
+`tests/test_exact_score.py` holds that line. **What is not repaired is the mirror itself**, and the
+gap shows wherever engine code *other than* `score_hand` runs against it. Three reproducible
+divergences, all on The Hook, which discards held cards during `Blind:press_play` and so runs
+discard handlers against the synthetic dict:
+
+| board | wrong scorer input | preview | engine |
+|---|---|---:|---:|
+| Hook + Mail-In Rebate + Bull | `money` 4, not 14 | 24 | 44 |
+| Hook + Castle | no discard-time suit target | 16 | 19 |
+| Hook destroys a negative Ramen + Stencil | `joker_slots` 6, not 5 | 96 | 80 |
+
+Two distinct causes: the synthetic dict does not carry the nested state a discard handler reads,
+and `joker_slots` is copied from the *pre*-`press_play` live value, so a joker destroyed during the
+press does not shrink it. Neither is new in v2.1 — both predate the published v1 numbers — and
+neither is reached by any position the frozen battery visits, which is why the differential tests
+do not catch them. They are a live hazard for an agent that meets a Hook board with one of those
+jokers, and a contribution surface: the durable fix is to build `synth` from the live state rather
+than key by key.
+
 ## The tactical scan cap
 
 **How it degrades.** Enumeration is small-k first and stops at the budget, so severity is a ladder,
@@ -116,10 +144,10 @@ can be selected and only high card, pair and three of a kind stay reachable at a
 in ordinary play — Juggler +1, Troubadour +2, the Paint Brush and Palette vouchers +1 each, and
 transiently Turtle Bean +5 and the Juggle Tag +3.
 
-**How often.** Instrumenting the true hand size at every scan, the cap binds on 576 of 8341
-`greedy-shop` play scans (6.91%, in 30/240 games) against 151 of 5176 for `random-shop` (2.92%,
-14/240); a scan runs on every in-blind decision, discards included. The instrumentation is pure
-observation — the seeds re-run under it reproduce their published `highest_ante`, 16/16
+**How often.** Instrumenting the true hand size at every scan, the cap binds on 576 of 8311
+`greedy-shop` play scans (6.93%, in 30/240 games) against 151 of 5176 for `random-shop` (2.92%,
+14/240) — 2.4x as often; a scan runs on every in-blind decision, discards included. The
+instrumentation is pure observation — the seeds re-run under it reproduce their published `highest_ante`, 16/16
 spot-checked — but it is not shipped, because the published decision records store the subset
 played, not the hand it was drawn from. These counts supersede the 19/240 and 8/240 published at
 launch, which came from a terminal-state estimate that omitted Troubadour and could not see
@@ -142,12 +170,12 @@ uv run python scripts/evaluate.py --agent greedy-shop --vs random-shop \
 
 Any budget other than the frozen `300` is stamped `jackhammer/tactical-sweep/v1` with
 `scope: diagnostic`, and the result's `agent.tactical` records the budget that actually ran, so a
-sweep can never be read as a v1 number. Doing so shifts `greedy-shop` by +0.104 ante
+sweep can never be read as a headline number. Doing so shifts `greedy-shop` by +0.104 ante
 `[+0.046, +0.175]` and `random-shop` by +0.004 `[+0.000, +0.013]`, and changes the outcome of 13/240
-seeds against 1/240. The cost is concentrated rather than diffuse: on the 201 seeds that never
-truncate the difference-of-differences is exactly zero with zero variance — as it must be, since an
-untruncated scan enumerates the same subsets at either budget — while the 39 exposed seeds shift
-`+0.615 [+0.308, +0.974]`.
+seeds against 1/240. The cost is concentrated rather than diffuse: on the 201 seeds where the cap
+never binds for either arm the difference-of-differences is exactly zero with zero variance — as it
+must be, since an untruncated scan enumerates the same subsets at either budget — while the 39
+exposed seeds shift `+0.615 [+0.308, +0.974]`.
 
 **The worst case.** The Psychic scores any play of fewer than five cards as zero (jackdaw's
 `h_size_ge=5` boss debuff), and at a true hand size of 10 or more the cap enumerates no five-card
@@ -165,10 +193,11 @@ uv run python scripts/inspect_run.py data/bench/greedy-shop.jsonl --seed 657P5QG
 which prints three `High Card score=0` plays.
 
 **Why the frozen protocol keeps it.** Raising the cap moves published numbers, so it is a question
-for the next protocol version — v3 — and not a patch to the current one. The outcome plateaus at `score_budget=2000` (mean highest ante 3.308, unchanged at 4000, 8000
+for the next protocol version — v3 — and not a point release to the current one. The outcome
+plateaus at `score_budget=2000` (mean highest ante 3.300, unchanged at 4000, 8000
 and 16000), and 2379 — every subset of size <=5 of the largest hand this battery dealt, 13
 cards — is the budget above which no scan in these runs can truncate at all. Going from 300 to 8000
-scans 7.4% more combos for 14.5% more wall clock (82.9s -> 94.9s, 14 workers).
+scans 9.6% more combos for 9.6% more wall clock (63.6s -> 69.7s, 14 workers, this machine).
 
 **If you are submitting an agent.** An agent with its own tactical layer is not subject to the cap,
 but its measured margin over `greedy-shop` still carries this handicap on the exposed seeds. An