diff --git a/CITATION.cff b/CITATION.cff index f9a04d4..d89cf5d 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -4,7 +4,7 @@ title: "Jackhammer: a paired Balatro agent benchmark" type: software authors: - name: "idIing" -version: 0.1.0 -date-released: 2026-09-02 +version: 1.1.0 +date-released: 2026-09-05 license: MIT repository-code: "https://github.com/idIing/jackhammer" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1fa6714..106af9c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -49,4 +49,7 @@ expectation. Generally useful fixes should go upstream to Jackdaw when practical Keep changes focused. Include the commands you ran and their outcomes. Do not mix a protocol change with an agent result or unrelated cleanup. Changes to the engine pin, v1 battery, primary metric, -holdout rule, baseline slate, or evaluation procedure require a new protocol version. +holdout rule, baseline slate, or evaluation procedure require a new protocol version — a point +release (v2.*x*) when the contract is untouched and only the numbers move, a major one when a +clause changes meaning. [The protocol](docs/protocol-v2.md) states the rule and records what each +release moved. diff --git a/README.md b/README.md index 43270c5..f8853ca 100644 --- a/README.md +++ b/README.md @@ -24,14 +24,19 @@ install -> run a baseline -> add an agent -> compare on identical seeds -> inspe |---|---|---|---:| | Random Legal | `random-legal` | Samples a legal action type uniformly, then a legal target; no tactical layer | 1.000 | | Random Shop | `random-shop` | `GreedyTactical` card play; random legal shop actions | 1.637 | -| Cheapest-Joker Shop | `greedy-shop` | `GreedyTactical` card play; buys the cheapest affordable Joker | 3.204 | +| Cheapest-Joker Shop | `greedy-shop` | `GreedyTactical` card play; buys the cheapest affordable Joker | 3.196 | -Those are the complete 240-seed v2 results on the `idIing/jackdaw-balatro` fork of Jackdaw at -[`4d6f19d`](https://github.com/idIing/jackdaw-balatro/commit/4d6f19d9fe73f96603412a59ad5eab16d08937e7), +Those are the complete 240-seed v2.1 results on the `idIing/jackdaw-balatro` fork of Jackdaw at +[`de733eb`](https://github.com/idIing/jackdaw-balatro/commit/de733ebd494a5da71fb7049b3b6b18ecd039786c), the exact commit the lockfile installs; all three went 0/240 on wins. The stable ID `greedy-shop` predates the clearer display name and is retained so existing artifacts stay comparable. The paired -Random Shop -> Cheapest-Joker Shop difference is +1.567 ante with a bootstrap-95 interval of -`[+1.400, +1.729]`. +Random Shop -> Cheapest-Joker Shop difference is +1.558 ante with a bootstrap-95 interval of +`[+1.396, +1.721]`. + +v2.1 re-pinned the engine and fixed four `game_state` keys `preview_play` handed the scorer inside +`GreedyTactical`'s ranking; only `greedy-shop` moved, and by 0.008 ante. Artifacts stamped +`jackhammer/v2` were produced at the previous pin and are not directly comparable — see +[what changed](docs/protocol-v2.md#what-changed-in-v21). ### The shared card play: `GreedyTactical` @@ -47,12 +52,12 @@ it at all, which is what makes it an honest floor rather than a third variation The 300 cap is exhaustive for a standard 8-card hand, where the complete set of ≤5-card subsets is 218. Above 8 cards it truncates, and it drops the largest subsets first. It also does not bind equally on the two arms: shopping grows the hand, so `greedy-shop` truncates ~2.4x as often, and the -`+1.567` understates the contrast by about 0.1 ante — see [known limits](docs/known-limits.md). +`+1.558` understates the contrast by about 0.1 ante — see [known limits](docs/known-limits.md). This ladder is intentionally weak. `greedy-shop` does not understand Joker text, quality, rarity, or synergy; it never rerolls or sells, it does not buy vouchers or consumables, and it never opens a booster pack. Over the whole battery it emits 6 of the engine's 21 action types; `random-shop` emits -16. So the `+1.567` prices buying the cheapest Joker at all against an arm that touches most of the +16. So the `+1.558` prices buying the cheapest Joker at all against an arm that touches most of the shop at random — the narrowest agent on the slate beating the widest one. Every run records that histogram and the evaluator checks it against each baseline's published description ([declared repertoires](docs/known-limits.md#declared-repertoires)). That gap is an open @@ -74,7 +79,7 @@ To use the kit as a library from your own project instead, install it — the fr the pinned simulator commit come with it, so a benchmark run from an install is attributable: ```bash -pip install "jackhammer-benchmark @ git+https://github.com/idIing/jackhammer.git@v1.0.0" +pip install "jackhammer-benchmark @ git+https://github.com/idIing/jackhammer.git@v1.1.0" ``` ```python @@ -85,7 +90,7 @@ from jackhammer.playground.seeds import load_battery spec = agents.get("greedy-shop") results = run_battery_with(load_battery("train")[:8], spec.make_decider, out_path="runs.jsonl", config_label=spec.name, slot1=spec.slot1, slot2=spec.slot2) -print(provenance.kit_pin()) # {'version': '1.0.0', 'commit': None, 'dirty': None} +print(provenance.kit_pin()) # {'version': '1.1.0', 'commit': None, 'dirty': None} ``` `scripts/` is not installed; the CLI below runs from a clone. An installed copy reports @@ -108,16 +113,16 @@ uv run python scripts/evaluate.py \ --agent greedy-shop --vs random-shop --limit 8 --workers 4 ``` -Remove `--limit 8` for the reportable v1 comparison over all 240 seeds. The command writes raw +Remove `--limit 8` for the reportable comparison over all 240 seeds. The command writes raw JSONL decision records, one result artifact per arm, and a paired comparison under `data/bench/`. The reportable run prints (artifact paths omitted here): ```text greedy-shop: 240 runs - mean highest ante: 3.204 + mean highest ante: 3.196 random-shop: 240 runs mean highest ante: 1.637 -paired (240 seeds): greedy-shop - random-shop = +1.567 ante [+1.400, +1.729] boot-95 +paired (240 seeds): greedy-shop - random-shop = +1.558 ante [+1.396, +1.721] boot-95 interval excludes zero ``` @@ -143,8 +148,8 @@ the closest baseline with `--vs`. ## Add a seed dataset -Protocol v2 always means the committed 240-seed `train` split. New coverage, stress, curriculum, or -seed-difficulty questions belong in versioned sidecar manifests: +The headline protocol always means the committed 240-seed `train` split. New coverage, stress, +curriculum, or seed-difficulty questions belong in versioned sidecar manifests: ```bash uv run python scripts/evaluate.py \ @@ -153,7 +158,7 @@ uv run python scripts/evaluate.py \ --split sample ``` -Custom datasets are stamped as `jackhammer/dataset-eval/v2`, never `jackhammer/v2`. That makes +Custom datasets are stamped as `jackhammer/dataset-eval/v2.1`, never `jackhammer/v2.1`. That makes future questions such as “how does seed coverage affect measured agent strength?” additive without silently moving the headline benchmark. See [Adding datasets](docs/datasets.md). diff --git a/docs/datasets.md b/docs/datasets.md index efe32aa..c945a2c 100644 --- a/docs/datasets.md +++ b/docs/datasets.md @@ -32,8 +32,8 @@ uv run python scripts/evaluate.py \ --dataset path/to/dataset.json --split sample ``` -Artifacts from this route use protocol `jackhammer/dataset-eval/v2` and dataset scope `diagnostic`. -They cannot be paired with `jackhammer/v2` artifacts. +Artifacts from this route use protocol `jackhammer/dataset-eval/v2.1` and dataset scope `diagnostic`. +They cannot be paired with `jackhammer/v2.1` artifacts. ## Coverage studies diff --git a/docs/known-limits.md b/docs/known-limits.md index 48b111e..add4638 100644 --- a/docs/known-limits.md +++ b/docs/known-limits.md @@ -22,8 +22,8 @@ `_open_tag_pack` for a tag's pack — and over the whole battery this agent emits neither `OpenBooster` nor `SkipBlind` nor any pack action, so the pack-choice branch of `decide_shop` is unreachable in practice. Over the published battery it emits **6 of the engine's 21 action types - across 15,349 decisions**, with zero fallback substitutions, against 16 for `random-shop`. - The paired `+1.567` is a clean A/B, but what it prices is *buying the cheapest Joker at all* + across 15,307 decisions**, with zero fallback substitutions, against 16 for `random-shop`. + The paired `+1.558` is a clean A/B, but what it prices is *buying the cheapest Joker at all* against an arm that touches most of the shop surface at random — the headline is not "greedy beats random" but "the narrowest agent on the slate beats the widest one". See [declared repertoires](#declared-repertoires) below. @@ -34,10 +34,10 @@ [the tactical scan cap](#the-tactical-scan-cap) below. - **Scan-cap asymmetry:** the shared tactical layer is not automatically a symmetric control. `greedy-shop` truncates 2.4x as often as `random-shop`, and re-running the battery with the cap - raised moves the paired delta from `+1.567` to `+1.667` — a paired difference-of-differences of + raised moves the paired delta from `+1.558` to `+1.658` — a paired difference-of-differences of **`+0.100 [+0.046, +0.167]`** boot-95, which excludes zero, so the published number slightly - *understates* the shop contrast. `+1.667` still falls inside the published interval - `[+1.400, +1.729]`, so no v1 conclusion changes. + *understates* the shop contrast. `+1.658` still falls inside the published interval + `[+1.396, +1.721]`, so no published conclusion changes. - **Published battery:** the 240 training seeds are public and therefore overfittable. The old validation split has already been consumed and is retired, not a reusable secret leaderboard. - **No live client:** the benchmark and text run inspector work headlessly. This repository does not @@ -46,6 +46,9 @@ relabeled as ordinary-distribution policy strength. Sampling and overlap must be reported. - **Artifact schema:** v1 validates the stable envelope and preserves the raw decision records by reference; it does not cryptographically sign results or fully validate every nested summary field. +- **The preview is not the engine on every board.** `preview_play` sets every key `score_hand` + reads directly, and v2.1 fixed the four it got wrong — but it builds a *partial* mirror of the + live state, and engine code other than the scorer runs against that mirror. See below. ## Declared repertoires @@ -86,12 +89,12 @@ agent cannot drift apart silently. The prose around it is still prose. | `random-shop` | 16 | `PlayHand` · `Discard` · `SelectBlind` · `CashOut` · `Reroll` · `NextRound` · `SkipPack` · `BuyCard` · `SellJoker` · `SellConsumable` · `UseConsumable` · `RedeemVoucher` · `OpenBooster` · `PickPackCard` · `SwapJokersLeft` · `SwapJokersRight` | -Measured on the 240-seed `train` split at the pinned engine, protocol v2: `greedy-shop` 15,349 +Measured on the 240-seed `train` split at the pinned engine, protocol v2.1: `greedy-shop` 15,307 decisions, `random-shop` 9,157, `random-legal` 5,713, all three with **zero** fallback substitutions. The reference agent's six, in full: ``` -PlayHand 4,527 · Discard 3,814 · SelectBlind 2,132 · CashOut 1,892 · NextRound 1,892 · BuyCard 1,092 +PlayHand 4,508 · Discard 3,803 · SelectBlind 2,128 · CashOut 1,888 · NextRound 1,888 · BuyCard 1,092 ``` `scripts/evaluate.py` prints the same line for whatever agent it runs, and `summary.repertoire` in @@ -107,6 +110,31 @@ separate claim about them. Two things a reader should not over-read: `SelectBlind` and `CashOut` calls v2 hands back to the agent. Repertoire counts are only comparable within one protocol version. +## The preview is not the engine on every board + +`GreedyTactical` ranks candidate plays with `preview_play` +(`src/jackhammer/playground/exact_score.py`), which reconstructs by hand the synthetic `game_state` +the engine's `_handle_play_hand` passes to `score_hand`, and then calls the engine's own scorer. +Protocol v2.1 repaired the four keys the scorer reads that it had wrong, and +`tests/test_exact_score.py` holds that line. **What is not repaired is the mirror itself**, and the +gap shows wherever engine code *other than* `score_hand` runs against it. Three reproducible +divergences, all on The Hook, which discards held cards during `Blind:press_play` and so runs +discard handlers against the synthetic dict: + +| board | wrong scorer input | preview | engine | +|---|---|---:|---:| +| Hook + Mail-In Rebate + Bull | `money` 4, not 14 | 24 | 44 | +| Hook + Castle | no discard-time suit target | 16 | 19 | +| Hook destroys a negative Ramen + Stencil | `joker_slots` 6, not 5 | 96 | 80 | + +Two distinct causes: the synthetic dict does not carry the nested state a discard handler reads, +and `joker_slots` is copied from the *pre*-`press_play` live value, so a joker destroyed during the +press does not shrink it. Neither is new in v2.1 — both predate the published v1 numbers — and +neither is reached by any position the frozen battery visits, which is why the differential tests +do not catch them. They are a live hazard for an agent that meets a Hook board with one of those +jokers, and a contribution surface: the durable fix is to build `synth` from the live state rather +than key by key. + ## The tactical scan cap **How it degrades.** Enumeration is small-k first and stops at the budget, so severity is a ladder, @@ -116,10 +144,10 @@ can be selected and only high card, pair and three of a kind stay reachable at a in ordinary play — Juggler +1, Troubadour +2, the Paint Brush and Palette vouchers +1 each, and transiently Turtle Bean +5 and the Juggle Tag +3. -**How often.** Instrumenting the true hand size at every scan, the cap binds on 576 of 8341 -`greedy-shop` play scans (6.91%, in 30/240 games) against 151 of 5176 for `random-shop` (2.92%, -14/240); a scan runs on every in-blind decision, discards included. The instrumentation is pure -observation — the seeds re-run under it reproduce their published `highest_ante`, 16/16 +**How often.** Instrumenting the true hand size at every scan, the cap binds on 576 of 8311 +`greedy-shop` play scans (6.93%, in 30/240 games) against 151 of 5176 for `random-shop` (2.92%, +14/240) — 2.4x as often; a scan runs on every in-blind decision, discards included. The +instrumentation is pure observation — the seeds re-run under it reproduce their published `highest_ante`, 16/16 spot-checked — but it is not shipped, because the published decision records store the subset played, not the hand it was drawn from. These counts supersede the 19/240 and 8/240 published at launch, which came from a terminal-state estimate that omitted Troubadour and could not see @@ -142,12 +170,12 @@ uv run python scripts/evaluate.py --agent greedy-shop --vs random-shop \ Any budget other than the frozen `300` is stamped `jackhammer/tactical-sweep/v1` with `scope: diagnostic`, and the result's `agent.tactical` records the budget that actually ran, so a -sweep can never be read as a v1 number. Doing so shifts `greedy-shop` by +0.104 ante +sweep can never be read as a headline number. Doing so shifts `greedy-shop` by +0.104 ante `[+0.046, +0.175]` and `random-shop` by +0.004 `[+0.000, +0.013]`, and changes the outcome of 13/240 -seeds against 1/240. The cost is concentrated rather than diffuse: on the 201 seeds that never -truncate the difference-of-differences is exactly zero with zero variance — as it must be, since an -untruncated scan enumerates the same subsets at either budget — while the 39 exposed seeds shift -`+0.615 [+0.308, +0.974]`. +seeds against 1/240. The cost is concentrated rather than diffuse: on the 201 seeds where the cap +never binds for either arm the difference-of-differences is exactly zero with zero variance — as it +must be, since an untruncated scan enumerates the same subsets at either budget — while the 39 +exposed seeds shift `+0.615 [+0.308, +0.974]`. **The worst case.** The Psychic scores any play of fewer than five cards as zero (jackdaw's `h_size_ge=5` boss debuff), and at a true hand size of 10 or more the cap enumerates no five-card @@ -165,10 +193,11 @@ uv run python scripts/inspect_run.py data/bench/greedy-shop.jsonl --seed 657P5QG which prints three `High Card score=0` plays. **Why the frozen protocol keeps it.** Raising the cap moves published numbers, so it is a question -for the next protocol version — v3 — and not a patch to the current one. The outcome plateaus at `score_budget=2000` (mean highest ante 3.308, unchanged at 4000, 8000 +for the next protocol version — v3 — and not a point release to the current one. The outcome +plateaus at `score_budget=2000` (mean highest ante 3.300, unchanged at 4000, 8000 and 16000), and 2379 — every subset of size <=5 of the largest hand this battery dealt, 13 cards — is the budget above which no scan in these runs can truncate at all. Going from 300 to 8000 -scans 7.4% more combos for 14.5% more wall clock (82.9s -> 94.9s, 14 workers). +scans 9.6% more combos for 9.6% more wall clock (63.6s -> 69.7s, 14 workers, this machine). **If you are submitting an agent.** An agent with its own tactical layer is not subject to the cap, but its measured margin over `greedy-shop` still carries this handicap on the exposed seeds. An diff --git a/docs/protocol-v2.md b/docs/protocol-v2.md index 8b9962a..d589511 100644 --- a/docs/protocol-v2.md +++ b/docs/protocol-v2.md @@ -1,18 +1,43 @@ # Benchmark protocol v2 -Status: frozen on 2026-09-04. Changing a numbered item creates v3 rather than silently editing v2. +Current revision: **v2.1**, frozen on 2026-09-05. Artifacts stamp `jackhammer/v2.1`. Supersedes [protocol v1](protocol-v1.md), which stays published so existing `jackhammer/v1` artifacts remain interpretable. +**How this document is versioned — amended at v2.1.** As frozen at v2.0 this document said that +changing *any* numbered item creates v3. Read literally that makes every engine re-pin a major +version, because item 1 names a commit; the number would then inflate on routine maintenance while +telling a reader nothing about comparability. The rule is now split by *what* moved rather than by +whether a clause was edited: + +- **Major — v3**, and this document is frozen and superseded the way v1 was: a clause changes what + it *means*. The action set, the primary metric, the comparison procedure, the battery, the + holdout rule, the slate, or a deliberate change to how a baseline plays. +- **Point — v2.*x***, edited into this document: the contract is unchanged and the conditions + under it move. The engine pin advances, or a defect in the evaluation machinery is corrected. + Every published figure is re-baselined and § What changed records what moved. + +Both are a new protocol version and both are stamped, so no artifact is ever ambiguous about which +conditions produced it. The difference is what a reader may do with two numbers: v2.0 and v2.1 ask +the same question of two different apparatus, v2 and v3 ask different questions. + +**Being explicit about what that reclassifies.** v2.1 edits the commit in item 1, and it changes how +`greedy-shop` plays by repairing the scorer it ranks candidate plays with. Under the rule as frozen +at v2.0 that was v3. It is a point release under the rule above, and amending the rule is this +release's only change to the contract's own text — recorded here rather than made quietly. + +Point releases so far: **v2.1** (2026-09-05) — engine re-pin and a scorer fidelity fix, § What +changed in v2.1. + 1. **Engine:** public `idIing/jackdaw-balatro` commit - `4d6f19d9fe73f96603412a59ad5eab16d08937e7`. Every artifact records the resolved commit and dirty + `de733ebd494a5da71fb7049b3b6b18ecd039786c`. Every artifact records the resolved commit and dirty state. An unknown or dirty engine produces a local artifact marked non-attributable. 2. **Battery:** `config/seed_battery_v1.json`, `train`, 240 seeds, in committed order. The raw file SHA-256 is stamped. The 60-seed `val` split was consumed during development and is retired; the CLI requires an explicit warning flag to read it. 3. **Environment:** Jackdaw defaults: Red Deck, White Stake, seeded play, maximum 2,000 steps. 4. **Action set:** **an agent is offered every action the engine makes legal, in every phase.** The - episode loop holds no policy. New in v2 — see § What changed. + episode loop holds no policy. New in v2 — see § What changed in v2. 5. **Primary metric:** mean highest ante reached. Win rate, blind-clear summaries, conditional advance curves, and per-seed records remain in the artifact. 6. **Comparison:** both arms run on the identical seed list. Results are joined by seed; the primary @@ -21,7 +46,64 @@ artifacts remain interpretable. 7. **Published slate:** `random-legal`, `random-shop`, and `greedy-shop`. Their policy behavior and stable IDs are frozen. `greedy-shop` is displayed as Cheapest-Joker Shop because that is what its selection logic actually does. Both shop baselines declare a fixed blind-select and cash-out - policy of their own (§ What changed). + policy of their own (§ What changed in v2). + +## What changed in v2.1 + +Two changes, both of which move published numbers. Between them they edit item 1's commit and +change how the reference agent plays; § How this document is versioned says why that is a point +release and not v3. + +**The engine pin moved** from `4d6f19d` to `de733eb`. The fork this benchmark runs on was rebased +onto canonical Jackdaw `8712c1e`, taking eleven upstream commits — among them an O(n²) hot-loop fix +in `get_x_same` and stake-sticker flag handling — and dropping two of its own that upstream had +since absorbed. Neither upstream commit touching scoring-relevant code changes an output *at this +battery's configuration*: the `get_x_same` change is exact-output by construction, and the +stake-sticker change only moves where the enable flags are read, which at item 3's White Stake +enables nothing under either reading. At Gold Stake the sticker change is a real behaviour change, +so the claim is scoped to the published configuration rather than to the commits. Run on its own — +new pin, scorer untouched — the battery returns `greedy-shop` 3.204, +`random-shop` 1.637, paired `+1.567 [+1.400, +1.729]`, 15,349 decisions: **the v2.0 headline to the +digit.** Everything that moves below is therefore the second change, not the pin. + +**`preview_play` now builds the scorer's `game_state` the way the engine does.** +`playground/exact_score.py` reconstructs by hand the synthetic `game_state` that `_handle_play_hand` +passes to `score_hand`, and `GreedyTactical` ranks every candidate play through it — so it sits +inside the reference agent's play selection, not in a diagnostic path. Four of the nineteen keys +`score_hand` reads were wrong: `hands_played`, `skips` and `chips` were never set at all +(jackhammer#9), and `current_round_hands_played` was passed one ahead of the engine, which increments +both hand counters *after* `score_hand` returns. A wrong key does not raise — it defaults — so the +preview simply disagreed with the engine on the boards that read it: Loyalty Card, Throwback, +Mr. Bones, DNA, Sixth Sense. + +Measured before the fix, over six battery seeds: 218 of 36,941 previewed plays disagreed with the +engine's own dry-run of the same play, every one a Loyalty Card x4 the engine applied and the +preview did not. `tests/test_exact_score.py` is the standing gate: it diffs the two `game_state` +dicts key for key at real positions, and compares preview totals against the engine over every play +the tactical enumerates. + +### The numbers that moved + +| | v2.0 (`4d6f19d`) | v2.1 (`de733eb` + scorer fix) | +|---|---:|---:| +| `greedy-shop` mean highest ante | 3.204 | **3.196** | +| `random-shop` mean highest ante | 1.637 | 1.637 | +| `random-legal` mean highest ante | 1.000 | 1.000 | +| paired `greedy-shop` − `random-shop` | +1.567 `[+1.400, +1.729]` | **+1.558 `[+1.396, +1.721]`** | +| `greedy-shop` decisions recorded | 15,349 | **15,307** | +| `random-shop` / `random-legal` decisions | 9,157 / 5,713 | 9,157 / 5,713 | +| wins, all three agents | 0/240 | 0/240 | + +Only `greedy-shop` moves, and that is the expected shape. `random-legal` never runs the tactical +layer at all. `random-shop` does, and a corrected key only changes a preview on a board that reads +it — so what matters is whether an arm ever holds one of the five jokers *while playing a hand*. +Replaying the decision streams: `greedy-shop` does in **25 of 240** runs, `random-shop` in **none**. +It is not that it never buys them; it acquires one in 5 runs (`K1U9J9UF`, `Q358C3MG`, `K9ADQ6YV` +Mr. Bones, `EZG9JGQS`, `JBVSIWHI` Loyalty Card) and sells each before it reaches a play decision. +Its per-seed outcomes and its whole decision histogram are unchanged. + +`build_comparison` refuses to pair arms whose engine commits differ, so a v2.0 artifact and a v2.1 +artifact cannot be silently joined. Read `provenance.engine.commit` before comparing anything. ## What changed in v2 @@ -42,8 +124,9 @@ Both are real decisions, not formalities: In v2 the loop delegates every phase, and an agent that wants a blind selected must select it. **This changed no v1 number.** All three baselines returned byte-identical per-seed outcomes on the -240-seed `train` split, and the headline paired delta is unchanged at **+1.567 ante -[+1.400, +1.729]**. That is expected rather than lucky: the two shop baselines now *declare* the +240-seed `train` split, and the headline paired delta was unchanged at **+1.567 ante +[+1.400, +1.729]** — the v2.0 figure, at engine `4d6f19d`; § What changed in v2.1 carries the +current one. That is expected rather than lucky: the two shop baselines now *declare* the same never-skip, always-cash-out policy the loop used to impose, and `random-legal` is too weak for its new options to reach the primary metric — it cleared 1 blind in 240 games under both protocols. @@ -71,4 +154,4 @@ limited smoke run is not a v2 result even though it uses the same machinery. The protocol measures policies inside the pinned Jackdaw simulator. Deterministic replay establishes simulator reproducibility, not equivalence to live Balatro. Results from custom datasets are stamped -under `jackhammer/dataset-eval/v1` and are not v2 benchmark numbers. +under `jackhammer/dataset-eval/v2.1` and are not v2 benchmark numbers. diff --git a/pyproject.toml b/pyproject.toml index 9fd1e1d..40abbe5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "jackhammer-benchmark" -version = "1.0.0" +version = "1.1.0" description = "A reproducible, paired Balatro agent benchmark on the Jackdaw simulator" readme = "README.md" requires-python = ">=3.12" @@ -16,7 +16,7 @@ dependencies = [ # installs a stranger's package and `import jackdaw.env` fails at the first agent. # The commit is the one protocol v2 section 1 names: pinning it here is what makes # a pip-installed kit produce an attributable result rather than a warning. - "jackdaw @ git+https://github.com/idIing/jackdaw-balatro.git@4d6f19d9fe73f96603412a59ad5eab16d08937e7", + "jackdaw @ git+https://github.com/idIing/jackdaw-balatro.git@de733ebd494a5da71fb7049b3b6b18ecd039786c", "numpy>=1.24", ] diff --git a/scripts/evaluate.py b/scripts/evaluate.py index 1b454ed..742d924 100755 --- a/scripts/evaluate.py +++ b/scripts/evaluate.py @@ -17,8 +17,8 @@ between two agents should be reported. The ``val`` split is retired. It is gated behind an explicit flag on purpose: its -value is destroyed by repeated looks and it was consumed during development. v1 -evaluations use ``train`` (``docs/protocol-v1.md`` section 2). +value is destroyed by repeated looks and it was consumed during development. Headline +evaluations use ``train`` (``docs/protocol-v2.md`` section 2). """ from __future__ import annotations diff --git a/src/jackhammer/bench/datasets.py b/src/jackhammer/bench/datasets.py index 75fa0d7..66a42a8 100644 --- a/src/jackhammer/bench/datasets.py +++ b/src/jackhammer/bench/datasets.py @@ -15,10 +15,11 @@ from jackhammer.playground.seeds import _check_seeds DATASET_SCHEMA = "jackhammer.seed-dataset/v1" -# v2 alongside the headline protocol: this route runs the same episode loop, so the -# v2 action-set change (docs/protocol-v2.md) applies to dataset artifacts too. The -# separate name keeps them unpairable with headline numbers, which is its whole job. -DATASET_PROTOCOL = "jackhammer/dataset-eval/v2" +# Versioned alongside the headline protocol: this route runs the same episode loop and +# the same scorer, so every change that creates a new headline version -- including a +# point release (docs/protocol-v2.md) -- reaches dataset artifacts too. The separate +# name keeps them unpairable with headline numbers, which is its whole job. +DATASET_PROTOCOL = "jackhammer/dataset-eval/v2.1" @dataclass(frozen=True) diff --git a/src/jackhammer/bench/provenance.py b/src/jackhammer/bench/provenance.py index c4004d7..b3358b8 100644 --- a/src/jackhammer/bench/provenance.py +++ b/src/jackhammer/bench/provenance.py @@ -32,13 +32,25 @@ from pathlib import Path from typing import Any -# Bump when the *meaning* of a stamped field changes, not when a value changes. +# Two kinds of bump, and the shape of the string says which. A MAJOR bump means a +# clause of the protocol changed meaning, so the questions the two versions answer are +# different and their numbers are not comparable. A POINT bump means the contract is +# untouched but the conditions under it moved -- a new engine pin, a corrected defect +# in the evaluation machinery -- so every published figure is re-baselined while still +# answering the same question. Neither is triggered by a value that is already stamped +# on its own (a battery digest, a runtime version). # # v2 (2026-09-04): the episode loop stopped auto-playing blind select and cash-out, # so an agent's action set is now the engine's. Every v1 number was produced by an # agent that could not skip a blind or use a consumable before the shop was rolled. # v1 and v2 artifacts are not comparable; see docs/protocol-v2.md § What changed. -PROTOCOL = "jackhammer/v2" +# +# v2.1 (2026-09-05): the engine pin moved to the rebased fork, and `preview_play` -- +# which GreedyTactical ranks every candidate play through -- stopped handing the +# scorer four game_state keys whose values the engine sets differently. The pin alone +# would not have bumped this; the scorer sits inside the reference agent's play +# selection. See docs/protocol-v2.md § What changed in v2.1. +PROTOCOL = "jackhammer/v2.1" # A run whose shared tactical budget is not the frozen cap is not a benchmark result. # Stamped separately so a sweep can never be mistaken for the benchmark number. diff --git a/src/jackhammer/playground/exact_score.py b/src/jackhammer/playground/exact_score.py index 4b026c9..2e3d464 100644 --- a/src/jackhammer/playground/exact_score.py +++ b/src/jackhammer/playground/exact_score.py @@ -86,6 +86,23 @@ def preview_play(gs: dict[str, Any], indices: tuple[int, ...]) -> ScoreResult: played/held split in selection order, hands_left decrement, per-card stat flags, ``_press_play`` boss effects (The Hook consumes rng + moves held cards), the Group-A game_state keys, then the engine's ``score_hand``. + + Every key ``score_hand`` reads *directly* out of ``game_state`` + (scoring.py:437-458, 535, 908) is set here, with the value the engine would + hold at the moment it calls the scorer -- which for the two ``hands_played`` + counters is the PRE-increment value. An absent or stale key does not raise; + it defaults, and the preview then silently disagrees with the engine on + exactly the boards that read it. + + That is a statement about the scorer's own inputs, **not** a guarantee of + exactness on every board. ``synth`` is a partial mirror of the live state, + and the gap shows wherever engine code other than ``score_hand`` runs + against it: ``_press_play`` hands ``synth`` to The Hook's discard handlers, + which read state this dict does not carry, and ``joker_slots`` below is + copied from the *pre*-``_press_play`` live value, so a joker destroyed + during the press does not shrink it. Three reproducible Hook divergences are + recorded under "The preview is not the engine on every board" in + ``docs/known-limits.md``. """ hand: list[Card] = gs.get("hand", []) if not indices or not hand: @@ -110,9 +127,13 @@ def preview_play(gs: dict[str, Any], indices: tuple[int, ...]) -> ScoreResult: played = [hand_copy[i] for i in indices] held = [c for i, c in enumerate(hand_copy) if i not in idx_set] - # Step 3: hands_left decrement / hands_played increment (game.py:520-522). + # Step 3: hands_left decrement only (game.py:632). The two hands_played + # counters do NOT move before scoring: the engine increments them in the + # event queued after evaluate_play (game.py:704-708, state_events.lua:523-24), + # so every scoring context reads PRE-increment values -- Loyalty Card's + # run-wide window (card.lua:3633) and DNA / Sixth Sense's current-round == 0 + # checks (card.lua:3501/2604). hands_left = cr["hands_left"] - 1 - hands_played = cr["hands_played"] + 1 # Step 4: per-card stats (game.py:528-535) — on copies. for card in played: @@ -142,7 +163,8 @@ def preview_play(gs: dict[str, Any], indices: tuple[int, ...]) -> ScoreResult: # deck + hand + discard_pile + played (Hook moves held→discard; len same). all_cards = [*live_deck, *held, *live_discard, *synth["discard_pile"], *played] synth["hands_left"] = hands_left - synth["current_round_hands_played"] = hands_played + synth["current_round_hands_played"] = cr.get("hands_played", 0) + synth["hands_played"] = gs.get("hands_played", 0) synth["discards_left"] = cr.get("discards_left", 0) synth["discards_used"] = cr.get("discards_used", 0) synth["money"] = synth["dollars"] @@ -157,6 +179,12 @@ def preview_play(gs: dict[str, Any], indices: tuple[int, ...]) -> ScoreResult: synth["idol_card"] = cr.get("idol_card") synth["ancient_suit"] = cr.get("ancient_card", {}).get("suit") synth["consumable_usage_tarot"] = gs.get("consumable_usage_total", {}).get("tarot", 0) + # Run-wide counters score_hand reads straight off the top level (game.py never + # re-derives them): Throwback's x0.25-per-skip (scoring.py:458) and Mr. Bones' + # cumulative-round-chips death test (scoring.py:535,908). Chips are accumulated + # after scoring (game.py:710), so the pre-play value is the one to pass. + synth["skips"] = gs.get("skips", 0) + synth["chips"] = gs.get("chips", 0) # Group-B keys score_hand reads from gs (run_init.py:46,93,292,327). synth["joker_slots"] = gs.get("joker_slots", 5) synth["starting_deck_size"] = gs.get("starting_deck_size", 52) diff --git a/src/jackhammer/playground/seeds.py b/src/jackhammer/playground/seeds.py index e1fe60c..ce8b905 100644 --- a/src/jackhammer/playground/seeds.py +++ b/src/jackhammer/playground/seeds.py @@ -8,9 +8,9 @@ The bank is split ``train`` (240) / ``val`` (60), disjoint and dup-free. The ``val`` slice is retired: it was consumed during development and is gated behind an -explicit CLI flag. v1 evaluations use ``train``. +explicit CLI flag. Headline evaluations use ``train``. -Protocol: ``docs/protocol-v1.md``. +Protocol: ``docs/protocol-v2.md``. """ from __future__ import annotations diff --git a/tests/test_exact_score.py b/tests/test_exact_score.py new file mode 100644 index 0000000..a557e56 --- /dev/null +++ b/tests/test_exact_score.py @@ -0,0 +1,274 @@ +"""``preview_play`` is only useful if it is the engine. + +``preview_play`` (``playground/exact_score.py``) rebuilds, by hand, the synthetic +``game_state`` that ``_handle_play_hand`` hands to ``score_hand``. Every key it +forgets, or sets to a value the engine would not hold yet, **defaults silently**: +no exception, no warning, just a preview that disagrees with the hand the engine +will actually run. ``GreedyTactical`` ranks every candidate play through it, so a +divergence reaches the reference agent's play selection. + +That failure mode is invisible to a scan-cap or determinism test, so it needs a +gate of its own, and the gate has to be *differential* — the preview compared +against the engine, at states real play reaches. Two forms, because they fail +independently: + +* :func:`test_preview_game_state_matches_the_engines_key_for_key` is the **class** + gate. It intercepts ``score_hand`` on both paths for the same (state, play) and + diffs the two ``game_state`` dicts over the exact key set the scorer reads. It + fails on a key that is missing *and* on a key that is present with the wrong + value, which a score comparison only catches when some card on the board happens + to read it. +* :func:`test_preview_total_matches_the_engine_for_every_legal_play` is the + **outcome** gate: over every play subset the tactical would enumerate, the + preview's total and hand type equal the engine's own dry-run. + +History: at engine ``4d6f19d`` this preview omitted ``hands_played``, ``skips`` and +``chips`` (jackhammer#9) and passed ``current_round_hands_played`` one ahead of the +engine, which increments both counters *after* ``score_hand`` returns. Four of the +nineteen keys the scorer reads. Measured effect of the four before the fix: 218 of +36,941 previewed plays across six battery seeds disagreed with the engine, every +one of them a Loyalty Card x4 the engine applied and the preview did not. + +Run with: ``uv run --no-sync python -m pytest tests/test_exact_score.py -q`` +""" + +import pytest +from jackdaw.env import BalatroEnvironment, DirectAdapter + +from jackhammer.bench import agents as agent_registry +from jackhammer.playground import exact_score +from jackhammer.playground.exact_score import preview_play +from jackhammer.playground.harness import _enumerate_play_combos, _legal_cards +from jackhammer.selfplay.runner import play_episode +from jackhammer.selfplay.tools import calculate_score + +# Two frozen-battery seeds, walked by the reference agent. Real positions rather +# than hand-built boards: the bug this file exists for was found in play and is +# about states the harness actually visits. +SEEDS = ("PVRQ4K5A", "4NNGD2DN") + +# Every key ``score_hand`` reads out of ``game_state``: the ScoringContext +# construction (``jackdaw/engine/scoring.py:437-458``) plus the two Mr. Bones +# death checks (``scoring.py:535,908``). A key added there and not here narrows +# this gate silently, which is why the count is asserted below. +READ_KEYS = ( + "ancient_suit", + "chips", + "consumable_usage_tarot", + "current_round_hands_played", + "deck_cards_remaining", + "discards_left", + "discards_used", + "enhanced_card_count", + "hands_left", + "hands_played", + "idol_card", + "joker_slots", + "mail_card_id", + "money", + "playing_cards_count", + "skips", + "starting_deck_size", + "steel_tally", + "stone_tally", +) + +MAX_POSITIONS = 6 + +# The jokers that read one of the four keys this file's history is about: Loyalty +# Card (``hands_played`` run-wide), Throwback (``skips``), Mr. Bones (``chips``), +# DNA and Sixth Sense (``current_round_hands_played`` == 0). +_KEY_READING_JOKERS = frozenset( + {"j_loyalty_card", "j_throwback", "j_mr_bones", "j_dna", "j_sixth_sense"} +) + + +class _Enough(Exception): + """Stop the walk once enough positions have been inspected.""" + + +def _walk(seed: str, visit, limit: int = MAX_POSITIONS, start: int = 0) -> int: + """Drive ``greedy-shop`` over *seed*, calling ``visit(env, mask)`` at each + SELECTING_HAND position from the *start*-th onward, for *limit* of them. + + ``visit`` runs *after* the agent has chosen, so nothing it does can change the + trajectory, and it reads ``env._adapter.raw_state`` — the pristine + post-restore dict, which is the same one ``harness.best_play_scan`` previews + against. + """ + env = BalatroEnvironment(adapter_factory=DirectAdapter) + decide_inner = agent_registry.get("greedy-shop").make_decider(env, seed) + reached = visited = 0 + + def decide(raw_state, mask, history): + nonlocal reached, visited + playable = str(raw_state.get("phase", "")).upper().endswith("SELECTING_HAND") + out = decide_inner(raw_state, mask, history) + if playable and _legal_cards(mask): + if reached >= start: + visit(env, mask) + visited += 1 + if visited >= limit: + raise _Enough + reached += 1 + return out + + try: + play_episode(env, seed, decide, None, None, max_steps=2000) + except _Enough: + pass + return visited + + +def test_read_keys_covers_everything_the_scorer_reads(): + """The audited key set is the scorer's, not a remembered subset of it.""" + import inspect + + from jackdaw.engine import scoring + + src = inspect.getsource(scoring) + reads = { + line.split('gs.get("', 1)[1].split('"', 1)[0] + for line in src.splitlines() + if 'gs.get("' in line + } + assert reads == set(READ_KEYS), f"scorer reads {sorted(reads - set(READ_KEYS))} unaudited" + + +@pytest.mark.parametrize("seed", SEEDS) +def test_preview_game_state_matches_the_engines_key_for_key(seed, monkeypatch): + real = exact_score.score_hand + captured: dict[str, dict] = {} + + def spy(tag): + def wrapper(*args, **kwargs): + captured[tag] = {k: kwargs["game_state"].get(k, "") for k in READ_KEYS} + return real(*args, **kwargs) + + return wrapper + + from jackdaw.engine import game as game_mod + from jackdaw.engine import scoring as scoring_mod + from jackdaw.env import FactoredAction + + compared = 0 + + def visit(env, mask): + nonlocal compared + combo = tuple(_enumerate_play_combos(_legal_cards(mask), mask, 300)[0]) + + # The engine's own game_state, from a dry-run that is rolled straight back. + snapshot = env.get_state() + monkeypatch.setattr(scoring_mod, "score_hand", spy("engine")) + try: + env.step(FactoredAction(action_type=0, card_target=combo)) + except Exception: # noqa: BLE001 — boss play restriction; nothing to compare + return + finally: + monkeypatch.setattr(scoring_mod, "score_hand", real) + env.load_state(snapshot) + + monkeypatch.setattr(exact_score, "score_hand", spy("preview")) + try: + preview_play(env._adapter.raw_state, combo) + finally: + monkeypatch.setattr(exact_score, "score_hand", real) + + assert captured["preview"] == captured["engine"], ( + f"seed {seed}, play {combo}: preview's game_state differs from the engine's" + ) + compared += 1 + + assert game_mod is not None # the dry-run path under test + _walk(seed, visit) + assert compared > 0, f"no comparable position reached on {seed}" + + +# The window of ``PVRQ4K5A`` that held the four-key divergence. Position 28 of that +# walk is a Loyalty Card board on which the engine applied x4 and the preview did +# not, on all 218 enumerated plays; positions 26-30 bracket it so the case survives +# a small trajectory shift. Deliberately pinned rather than sampled: a generic early +# position holds no joker that reads any of the four keys, which is exactly why the +# outcome gate alone would have passed through the original bug. +_DIVERGENCE_WINDOW = ("PVRQ4K5A", 26, 5) + + +@pytest.mark.parametrize("seed,start,limit", [_DIVERGENCE_WINDOW, ("4NNGD2DN", 0, 3)]) +def test_preview_total_matches_the_engine_for_every_legal_play(seed, start, limit): + checked = 0 + saw_key_reader = False + + def visit(env, mask): + nonlocal checked, saw_key_reader + raw = env._adapter.raw_state + if any( + getattr(j, "center_key", None) in _KEY_READING_JOKERS for j in (raw.get("jokers") or []) + ): + saw_key_reader = True + for combo in _enumerate_play_combos(_legal_cards(mask), mask, 300): + truth = calculate_score(env, list(combo)) + if "error" in truth: + continue # engine-illegal selection; the preview is not consulted + sr = preview_play(env._adapter.raw_state, tuple(combo)) + assert (int(sr.total), sr.hand_type) == (int(truth["score"]), truth["hand_type"]), ( + f"seed {seed}, play {list(combo)}: preview " + f"{int(sr.total)}/{sr.hand_type} != engine " + f"{int(truth['score'])}/{truth['hand_type']}" + ) + checked += 1 + + _walk(seed, visit, limit=limit, start=start) + assert checked > 0, f"no play scored on {seed}" + if (seed, start, limit) == _DIVERGENCE_WINDOW: + assert saw_key_reader, ( + f"{seed} positions {start}..{start + limit - 1} no longer hold a joker that " + f"reads one of the four keys, so this window has stopped covering the " + f"regression it was pinned to. Re-pin it: walk the seed, find a position " + f"whose board holds one of {sorted(_KEY_READING_JOKERS)}, and move the window." + ) + + +def test_the_repaired_keys_are_read_from_the_state_not_hard_coded(): + """Sentinel values must travel from the live state through to the scorer. + + The differential tests above cannot catch a constant. No baseline ever skips a + blind, so ``skips`` is 0 at every position this battery reaches; ``chips`` is 0 + on the first hand of every round and the counters are small. A ``preview_play`` + that wrote ``synth["skips"] = 0`` instead of reading ``gs`` would pass every + other test in this file. This one puts a value in the live state that the + battery never produces and asserts the scorer is handed it. + """ + import copy + + real = exact_score.score_hand + captured: dict[str, object] = {} + + def spy(*args, **kwargs): + captured.update(kwargs["game_state"]) + return real(*args, **kwargs) + + sentinels = {"skips": 7, "chips": 4321, "hands_played": 41} + checked = 0 + + def visit(env, mask): + nonlocal checked + if checked: + return + gs = copy.deepcopy(env._adapter.raw_state) + gs.update(sentinels) + gs["current_round"]["hands_played"] = 3 + combo = tuple(_enumerate_play_combos(_legal_cards(mask), mask, 300)[0]) + + exact_score.score_hand = spy + try: + preview_play(gs, combo) + finally: + exact_score.score_hand = real + + for key, value in sentinels.items(): + assert captured[key] == value, f"{key} did not reach the scorer" + assert captured["current_round_hands_played"] == 3 + checked = 1 + + _walk(SEEDS[0], visit, limit=1) + assert checked, f"no position reached on {SEEDS[0]}" diff --git a/tests/test_public_contracts.py b/tests/test_public_contracts.py index ff48e36..1fa3e73 100644 --- a/tests/test_public_contracts.py +++ b/tests/test_public_contracts.py @@ -39,14 +39,15 @@ def test_shop_baselines_record_their_tactical_layer(): def test_protocol_is_v2(): - """v1 numbers were produced by agents that could not skip a blind. + """v1 numbers were produced by agents that could not skip a blind, and v2.0 + numbers by a scorer that handed the engine four wrong ``game_state`` keys. The stamp is what tells a reader which apparatus produced a number. Pinning it here means demoting the protocol has to be deliberate. """ from jackhammer.bench.provenance import PROTOCOL - assert PROTOCOL == "jackhammer/v2" + assert PROTOCOL == "jackhammer/v2.1" def test_episode_loop_plays_no_phase_for_the_agent(): @@ -197,7 +198,7 @@ def test_a_swept_budget_is_not_stamped_as_a_v1_result(): assert provenance.TACTICAL_PROTOCOL != provenance.PROTOCOL # The frozen protocol itself is pinned by `test_protocol_is_v2`; what matters # here is that a sweep can never be stamped with it, whatever its version. - assert provenance.PROTOCOL == "jackhammer/v2" + assert provenance.PROTOCOL == "jackhammer/v2.1" def test_the_engine_dependency_pins_the_protocol_commit(): diff --git a/uv.lock b/uv.lock index 497719d..713b4cf 100644 --- a/uv.lock +++ b/uv.lock @@ -56,7 +56,7 @@ wheels = [ [[package]] name = "jackdaw" version = "0.1.0" -source = { git = "https://github.com/idIing/jackdaw-balatro.git?rev=4d6f19d9fe73f96603412a59ad5eab16d08937e7#4d6f19d9fe73f96603412a59ad5eab16d08937e7" } +source = { git = "https://github.com/idIing/jackdaw-balatro.git?rev=de733ebd494a5da71fb7049b3b6b18ecd039786c#de733ebd494a5da71fb7049b3b6b18ecd039786c" } dependencies = [ { name = "gymnasium" }, { name = "numpy" }, @@ -64,7 +64,7 @@ dependencies = [ [[package]] name = "jackhammer-benchmark" -version = "1.0.0" +version = "1.1.0" source = { editable = "." } dependencies = [ { name = "jackdaw" }, @@ -79,7 +79,7 @@ dev = [ [package.metadata] requires-dist = [ - { name = "jackdaw", git = "https://github.com/idIing/jackdaw-balatro.git?rev=4d6f19d9fe73f96603412a59ad5eab16d08937e7" }, + { name = "jackdaw", git = "https://github.com/idIing/jackdaw-balatro.git?rev=de733ebd494a5da71fb7049b3b6b18ecd039786c" }, { name = "numpy", specifier = ">=1.24" }, ]