protocol v2.1: re-pin the engine, and make preview_play the engine again - #11
Conversation
Two changes that each move published numbers, landed together so the battery is re-baselined once. Shipped as a point release: the contract (protocol items 1-7) is untouched; only the conditions under it moved. Engine pin 4d6f19d -> de733eb. The fork is rebased onto canonical Jackdaw 8712c1e, taking eleven upstream commits and dropping two it had absorbed. On its own the pin moves nothing: with the scorer untouched the battery returns greedy-shop 3.204, random-shop 1.637, +1.567 [+1.400, +1.729], 15,349 decisions -- the v2.0 headline to the digit. Everything that moves is the second change. preview_play now builds the scorer's game_state the way the engine does. It reconstructs by hand the dict _handle_play_hand passes to score_hand, and GreedyTactical ranks every candidate play through it, so it sits inside the reference agent's play selection. #9 reported three keys it never set -- hands_played, skips, chips. Diffing the engine's game_state against the preview's, key for key, at real positions found a fourth at 99 of 99 of them: current_round_hands_played was passed one ahead, because the engine increments both hand counters in the event queued after evaluate_play, which is exactly the value DNA and Sixth Sense test against zero. A missing key and a stale key fail identically -- silently, by defaulting -- so only the missing ones showed up in a set difference. Measured before the fix over six battery seeds: 218 of 36,941 previewed plays disagreed with the engine's own dry-run, every one a Loyalty Card x4 the engine applied and the preview did not. After: 0. tests/test_exact_score.py is the standing gate, in two forms that fail independently -- a key-for-key diff of the two game_state dicts, and preview total vs engine dry-run over every play the tactical enumerates, on a window pinned to the board that diverged. Both are red on the unfixed scorer. Re-baselined battery, 240-seed train split, attributable (de733eb, not dirty): greedy-shop 3.204 -> 3.196, random-shop 1.637 and random-legal 1.000 unchanged, paired +1.567 [+1.400, +1.729] -> +1.558 [+1.396, +1.721], greedy-shop decisions 15,349 -> 15,307, wins 0/240 throughout. Only greedy-shop moves: random-legal never runs the tactical layer, and random-shop's recorded inventories never hold any of the five jokers concerned, where greedy-shop's do in 23 of 240 runs. Every other figure in docs/known-limits.md was re-measured rather than assumed. The scan-cap audit re-derives unchanged -- +0.104 [+0.046, +0.175] and +0.004 [+0.000, +0.013], difference-of-differences +0.100 [+0.046, +0.167], 13/240 against 1/240, the 201/39 split with dod exactly zero at zero variance and +0.615 [+0.308, +0.974] on the exposed seeds, 576 truncated scans in 30/240 games against 151 in 14/240, max hand size 13, and the two Psychic lockups. What moved is what the scorer touches: greedy's scan count 8341 -> 8311, its decision histogram, and the raised-cap plateau 3.308 -> 3.300. Closes #9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h gap Adversarial review of the PR found four things. The fix itself and every re-baselined number held; what did not were three claims in my own prose and one coverage gap in the tests. The docstring claimed preview_play now hands the engine what the engine would hold, full stop. True of the scorer's own inputs, false as stated: synth is a partial mirror and engine code other than score_hand runs against it. Three reproducible Hook divergences -- money on a Mail-In Rebate board, a missing discard-time suit target for Castle, and joker_slots copied pre-press_play so a destroyed negative Ramen does not shrink it. All three predate the published v1 numbers and none is reached by the frozen battery, so they are logged as a known limit and a contribution surface, not fixed here. The versioning note said neither change touches a clause. It does: v2.1 edits item 1's commit and changes how greedy-shop plays. The document now says plainly that it amends the rule frozen at v2.0 -- which said any numbered item creates v3 -- splitting the rule by what moved rather than by whether a clause was edited, and names that amendment as this release's only change to the contract's text. "Both upstream commits read as exact-output" was scoped wrong. The stake-sticker commit is a real behaviour change at Gold Stake; it is a no-op at item 3's White Stake. The claim now attaches to the published configuration, not to the commits. The 23/240 joker figure was a union of two partial sources and matched neither. Replaying the streams: greedy-shop holds one of the five while playing a hand in 25 of 240 runs, random-shop in none -- and "dies too early to buy them" was false, it acquires one in 5 runs and sells each before a play decision. Coverage: every differential test would still pass if preview_play hard-coded skips to 0, because no baseline skips a blind and the battery never produces a nonzero value. Added a test that puts sentinels the battery cannot produce into the live state and asserts they reach the scorer; it fails on the hard-coded constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review, and four correctionsPut to a second model cold — isolated clone of the pushed branch, the six claims handed over as What held. The four key assignments; the pre-increment reading, cross-checked against the Lua What did not. Three claims in the prose and one coverage gap — all in the write-up, none in the
Coverage gap, closed. Every differential test still passed with No published number changed. |
Closes #9.
Two coupled changes that each move published numbers, landed together so the battery is
re-baselined once rather than twice, and shipped as protocol v2.1 — a point release: the
contract (items 1–7) is untouched, only the conditions under it moved.
1. Engine re-pin:
4d6f19d→de733ebThe fork is rebased onto canonical Jackdaw
8712c1e, taking eleven upstream commits (including theO(n²)
get_x_samehoist and stake-sticker flag handling) and dropping two of its own that upstreamhad absorbed.
The pin on its own moves nothing. Run with the new engine and the scorer untouched, the full
240-seed battery returns
greedy-shop3.204,random-shop1.637, paired+1.567 [+1.400, +1.729], 15,349 decisions — the v2.0 headline to the digit. Everything below is
therefore the scorer fix.
2.
preview_playbuilds the scorer'sgame_statethe way the engine does (#9)GreedyTacticalranks every candidate play throughpreview_play, so this is inside the referenceagent's play selection, not a diagnostic path. #9 reported three keys
score_handreads thatpreview_playnever set. Auditing the class rather than the instance — interceptingscore_handonboth paths at real positions and diffing the two
game_statedicts over the nineteen keys thescorer actually reads — found four defects, at 99 of 99 audited positions:
hands_playedscoring.py:447skipsscoring.py:458chipsscoring.py:535,908current_round_hands_playedscoring.py:446The fourth is not in #9. The engine increments both hand counters in the event queued after
evaluate_play(game.py:704-708), so every scoring context reads pre-increment values;preview_playpassed the post-increment one, which is exactly the value DNA and Sixth Sense testagainst zero.
Measured effect before the fix, over six battery seeds: 218 of 36,941 previewed plays disagreed
with the engine's own dry-run of the same play — all 218 a Loyalty Card ×4 the engine applied and
the preview did not. After: 0 of 36,941, and 0 key disagreements.
The re-baselined battery
Full 240-seed
trainsplit, attributable (engine.commit de733eb,dirty false, no"not attributable" warning), stamped
jackhammer/v2.1:greedy-shopmean highest anterandom-shoprandom-legalgreedy-shop−random-shop[+1.400, +1.729][+1.396, +1.721]greedy-shopdecisionsrandom-shop/random-legaldecisionsOnly
greedy-shopmoves.random-legalnever runs the tactical layer;random-shopdoes, but itsrecorded end-of-run inventories never hold any of the five jokers concerned (
greedy-shop's do in23 of 240 runs), and its per-seed outcomes and whole decision histogram are unchanged.
The regression gate
tests/test_exact_score.py, two forms because they fail independently:game_stateagainst the preview's, key for key, at realpositions. Catches a missing key and a present-but-wrong one.
tactical enumerates, on a window pinned to the board that actually diverged.
Both are red on the unfixed scorer and green after; 3.6s.
Everything in
docs/known-limits.mdwas re-measured, not assumedEvery figure in that file was re-derived, not assumed. Almost all of them come back unchanged,
which is the useful result — the scan-cap audit is independent of the scorer:
--score-budget 8000:greedy-shop+0.104 [+0.046, +0.175],random-shop+0.004 [+0.000, +0.013]; difference-of-differences+0.100 [+0.046, +0.167]; 13/240 vs 1/240outcomes changed
+0.615 [+0.308, +0.974]greedy-shop, 151 in 14/240 forrandom-shop; max handsize 13; 0 truncation at 8000
greedy-shopPsychic blinds, 0 of 26 forrandom-shop;657P5QGWandPM4RVISWstill 0/600 at ante 1 and still clear 720/600 at the raised capin 232/240, never a tactical one
What moved is only what the scorer touches: greedy's scan count
8341 → 8311(6.91% → 6.93%truncating), its decision histogram, the raised-cap plateau
3.308 → 3.300, and the sweep delta+1.667 → +1.658. The 300→8000 cost was re-timed on this machine: 9.6% more combos for 9.6% morewall clock (63.6s → 69.7s, 14 workers).
Why a point release and not v3
CONTRIBUTING.mdrequires a new protocol version for an engine-pin or evaluation-procedure change,and
docs/protocol-v2.mdforbade editing a numbered item in place. Both are satisfied by v2.1, andthe distinction is now written into the protocol and into
provenance.PROTOCOL: a major bumpmeans a clause changed meaning, so the two versions answer different questions; a point bump
means the contract is untouched and the conditions under it moved, so every figure is re-baselined
while still answering the same question. Without that rule a routine pin move would ship v3, v4, v5.
Checks
The headline reproduced identically across four independent full-battery runs.
Note for the merge
pyproject.toml,CITATION.cffand the README install snippet move to 1.1.0; thev1.1.0taghas to be created at merge for that snippet to resolve.