Make the tactical scan cap reproducible, and correct what it costs - #1
Conversation
known-limits.md quantifies what GreedyTactical's 300-subset scan cap costs, but the cap was a constructor default with no CLI surface, so checking that claim meant editing src/bench/agents.py and the identity string two contract tests assert. A limitation a reader cannot reproduce is a claim they have to take on faith. --score-budget defaults to the v1 300, leaving the benchmark path byte-identical. The risk it introduces is someone publishing a v1-labelled number at a changed cap, so three guardrails: - any non-default budget stamps jackhammer/tactical-sweep/v1 with scope=diagnostic, never jackhammer/v1 -- the same treatment --dataset already gets, rather than a new mechanism; - the result's agent.tactical records the budget that actually ran, so an artifact can never read score_budget=300 while 8000 produced it; - agents the flag cannot reach raise instead of silently no-opping. random-legal has no tactical layer and a submitted agent builds its own, so quietly accepting the flag would let a sweep report a budget it never applied. The budget travels in the pool task tuple rather than module state, so it survives every start method and not just fork. Side benefit: per-run artifacts now name the tactical layer under a sweep, which is what makes "both arms share the layer" checkable from the artifacts instead of only from prose. Verified: --score-budget 8000 over the full train split reproduces +1.667 [+1.504, +1.825], matching the figure known-limits.md now cites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring the cap rather than reasoning about it changed the story in three ways, and the shipped text was wrong or silent on each. Severity is a ladder, not a cliff. The old bullet stopped at "at hand size 9 only 45 of 126 five-card subsets; at 10, none". The runs reach hand size 13, where the budget is spent inside k=3 -- at most three cards selectable, so only high card, pair and three of a kind remain reachable at all. "Both shop arms share the layer, so the paired comparison stays valid" was a non-sequitur. Shared is not symmetric: a large hand is downstream of buying Jokers, so greedy-shop truncates 2.4x as often (576/8341 scans against 151/5176). Re-running at a raised cap moves the paired delta +1.567 -> +1.667, a paired difference-of-differences of +0.100 [+0.046, +0.167] that excludes zero. The published number slightly understates the shop contrast -- and +1.667 still sits inside the published interval, so no v1 conclusion changes. The worst case was undocumented. The Psychic zeroes any play under five cards, and at hand size >=10 the cap enumerates no five-card subset, so every reachable play scores zero. It bit 2 of 45 greedy-shop Psychic blinds and 0 of 26 random: 657P5QGW and PM4RVISW both lost at ante 1 with 0/600 and both clear 720/600 at a raised cap. Reproducible from the shipped records via scripts/inspect_run.py. Also corrects the count this document published at launch: 19/240 and 8/240 reproduce exactly, but from a terminal-state estimate that omitted Troubadour and could not see transient hand size. Said explicitly rather than swapped silently. README.md and docs/adding-an-agent.md both promised the paired difference isolates the shop policy -- "and nothing else", "alone". The evidence makes that false, and shipping the admission only in known-limits would leave the front door overclaiming. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Status update, and a correction to the reason this was being held. This PR was held on the expectation that the fidelity fix would invalidate the numbers it documents to three decimals. It did not. #6 lands protocol v2 — agents are now offered every action the engine makes legal — and all three baselines return byte-identical per-seed outcomes on the 240-seed Independently verified before that: this refactor is behaviourally inert. Over 24 games × 2 arms, Two things to fix before this merges:
|
Resolutions, all three where v2 and this branch touched the same lines: - `agents.py` keeps v2's `greedy-shop` description (it now declares never-skip and never-use-before-cash-out as policy) on top of this branch's `_greedy_shop_at(DEFAULT_SCORE_BUDGET)` factory. - `test_public_contracts.py` keeps both sets of new tests. The sweep test's `PROTOCOL == "jackhammer/v1"` assertion becomes v2; what it exists to check is that a sweep is never stamped with the frozen protocol, whichever version that is. `test_protocol_is_v2` pins the version itself. - `known-limits.md` takes this branch's rewritten scan-cap bullet wholesale. v2 had reworded the old bullet; this branch replaces it with a summary plus the measured § The tactical scan cap below, which supersedes the 19/240 and 8/240 figures v2's wording still carried. Its "raising the cap is a v2 question" now reads v3, since v2 shipped. The follow-up commit reconciles the statements this merge left stale elsewhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The merge left three statements false and one command self-destructive. - The `--score-budget` help, the DIAGNOSTIC banner, and two comments said "not a v1 result" about a stamp that now reads v2. All now name the frozen protocol rather than a version number, so the next bump does not restale them. - The sweep artifact field `v1_score_budget` becomes `frozen_score_budget` for the same reason. It has never shipped -- the flag that writes it is landing in this PR. - `agents.py` still claimed the paired difference measures the shop policy "and nothing else". That is the exact overclaim this PR removed from `README.md` and `adding-an-agent.md`; it survived in a comment. - The sweep command wrote to `--out-dir`'s default `data/bench/`, which is where the README's baseline run lands -- so following this page in order overwrote the records its next paragraph asks you to inspect, and after a raised-cap re-run seed 657P5QGW clears the blind instead of locking up. The command now names its own out-dir. The inspect command is also no longer described as reading "published records": `data/` is gitignored and ships empty, so it reads your own run. Verified: `uv run pytest tests/` 78 passed, 1 deselected; `ruff check src scripts tests` clean. Full 240-seed battery reproduces the published numbers -- greedy-shop 3.204, random-shop 1.637, paired +1.567 [+1.400, +1.729], random-legal 1.000. The cited lockup reproduces: `inspect_run.py --seed 657P5QGW` prints exactly three `High Card score=0` plays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped
docs/known-limits.mdconcedes thatGreedyTacticaltruncates itsscan above hand size 8, then defends the headline: "Both shop arms share the
layer, so the paired comparison stays valid."
Shared is not symmetric. A large hand is downstream of buying Jokers, so the
stronger arm meets the degraded regime more often. Measuring that changed the
story, and the cap had no CLI surface, so the claim was unverifiable either way.
--score-budgetDefaults to the v1
300, so the benchmark path is byte-identical. Any othervalue is stamped
jackhammer/tactical-sweep/v1withscope: diagnostic, neverjackhammer/v1, and the result'sagent.tacticalrecords the budget thatactually ran — a sweep cannot be read as a v1 number. Agents the flag cannot
reach (
random-legal, or any agent building its own tactical layer) raiserather than silently no-op.
What it costs
+1.567 [+1.400, +1.729]+1.667 [+1.504, +1.825]Paired difference-of-differences
+0.100 [+0.046, +0.167], excludes zero.greedy-shoptruncates on 576/8341 play scans (6.91%) against 151/5176 (2.92%)for
random-shop. The effect is concentrated: the 201 seeds that never truncatemove by exactly zero with zero variance; the 39 exposed seeds move
+0.615 [+0.308, +0.974].No published v1 number is restated.
+1.567remains the headline, and+1.667falls inside its interval, so no v1 conclusion changes. Raising the capstays a v2 question.
The worst case, previously undocumented
The Psychic scores any play of fewer than five cards as zero, and at hand size
Also corrected
budget is spent inside k=3 and only high card, pair and three of a kind stay
reachable. The old text stopped at hand 10.
estimate that omitted Troubadour and could not see transient hand size. Stated
explicitly rather than swapped silently.
README.mdanddocs/adding-an-agent.mdboth promised the paired differenceisolates the shop policy — "and nothing else", "alone". The evidence makes
that false, and shipping the admission only in
known-limits.mdwould leavethe front door overclaiming.
74 tests pass; three new ones cover the override's honesty and its refusals.