Skip to content

Make the tactical scan cap reproducible, and correct what it costs - #1

Merged
idIing merged 4 commits into
mainfrom
fix/scan-cap-known-limits
Sep 5, 2026
Merged

idIing merged 4 commits into
mainfrom
fix/scan-cap-known-limits

Conversation

@idIing

@idIing idIing commented Sep 4, 2026

Copy link
Copy Markdown
Owner

The shipped docs/known-limits.md concedes that GreedyTactical truncates its
scan above hand size 8, then defends the headline: "Both shop arms share the
layer, so the paired comparison stays valid."

Shared is not symmetric. A large hand is downstream of buying Jokers, so the
stronger arm meets the degraded regime more often. Measuring that changed the
story, and the cap had no CLI surface, so the claim was unverifiable either way.

--score-budget

uv run python scripts/evaluate.py --agent greedy-shop --vs random-shop --score-budget 8000

Defaults to the v1 300, so the benchmark path is byte-identical. Any other
value is stamped jackhammer/tactical-sweep/v1 with scope: diagnostic, never
jackhammer/v1, and the result's agent.tactical records the budget that
actually ran — a sweep cannot be read as a v1 number. Agents the flag cannot
reach (random-legal, or any agent building its own tactical layer) raise
rather than silently no-op.

What it costs

budget 300 budget 8000
paired delta +1.567 [+1.400, +1.729] +1.667 [+1.504, +1.825]

Paired difference-of-differences +0.100 [+0.046, +0.167], excludes zero.
greedy-shop truncates on 576/8341 play scans (6.91%) against 151/5176 (2.92%)
for random-shop. The effect is concentrated: the 201 seeds that never truncate
move by exactly zero with zero variance; the 39 exposed seeds move
+0.615 [+0.308, +0.974].

No published v1 number is restated. +1.567 remains the headline, and
+1.667 falls inside its interval, so no v1 conclusion changes. Raising the cap
stays a v2 question.

The worst case, previously undocumented

The Psychic scores any play of fewer than five cards as zero, and at hand size

= 10 the cap enumerates no five-card subset — so every reachable play scores
zero. It bit 2 of 45 greedy-shop Psychic blinds and 0 of 26 for random-shop:
657P5QGW and PM4RVISW both lost at ante 1 with 0/600, and both clear 720/600
at a raised cap. Visible in the shipped records:

uv run python scripts/inspect_run.py data/bench/greedy-shop.jsonl --seed 657P5QGW

Also corrected

  • Severity is a ladder, not a cliff: the runs reach hand size 13, where the
    budget is spent inside k=3 and only high card, pair and three of a kind stay
    reachable. The old text stopped at hand 10.
  • The launch's 19/240 and 8/240 reproduce exactly, but came from a terminal-state
    estimate that omitted Troubadour and could not see transient hand size. Stated
    explicitly rather than swapped silently.
  • README.md and docs/adding-an-agent.md both promised the paired difference
    isolates the shop policy — "and nothing else", "alone". The evidence makes
    that false, and shipping the admission only in known-limits.md would leave
    the front door overclaiming.

74 tests pass; three new ones cover the override's honesty and its refusals.

idIing and others added 2 commits September 3, 2026 20:47
known-limits.md quantifies what GreedyTactical's 300-subset scan cap costs, but
the cap was a constructor default with no CLI surface, so checking that claim
meant editing src/bench/agents.py and the identity string two contract tests
assert. A limitation a reader cannot reproduce is a claim they have to take on
faith.

--score-budget defaults to the v1 300, leaving the benchmark path byte-identical.
The risk it introduces is someone publishing a v1-labelled number at a changed
cap, so three guardrails:

- any non-default budget stamps jackhammer/tactical-sweep/v1 with
  scope=diagnostic, never jackhammer/v1 -- the same treatment --dataset already
  gets, rather than a new mechanism;
- the result's agent.tactical records the budget that actually ran, so an
  artifact can never read score_budget=300 while 8000 produced it;
- agents the flag cannot reach raise instead of silently no-opping. random-legal
  has no tactical layer and a submitted agent builds its own, so quietly
  accepting the flag would let a sweep report a budget it never applied.

The budget travels in the pool task tuple rather than module state, so it
survives every start method and not just fork.

Side benefit: per-run artifacts now name the tactical layer under a sweep, which
is what makes "both arms share the layer" checkable from the artifacts instead of
only from prose.

Verified: --score-budget 8000 over the full train split reproduces
+1.667 [+1.504, +1.825], matching the figure known-limits.md now cites.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring the cap rather than reasoning about it changed the story in three ways,
and the shipped text was wrong or silent on each.

Severity is a ladder, not a cliff. The old bullet stopped at "at hand size 9 only
45 of 126 five-card subsets; at 10, none". The runs reach hand size 13, where the
budget is spent inside k=3 -- at most three cards selectable, so only high card,
pair and three of a kind remain reachable at all.

"Both shop arms share the layer, so the paired comparison stays valid" was a
non-sequitur. Shared is not symmetric: a large hand is downstream of buying
Jokers, so greedy-shop truncates 2.4x as often (576/8341 scans against
151/5176). Re-running at a raised cap moves the paired delta +1.567 -> +1.667, a
paired difference-of-differences of +0.100 [+0.046, +0.167] that excludes zero.
The published number slightly understates the shop contrast -- and +1.667 still
sits inside the published interval, so no v1 conclusion changes.

The worst case was undocumented. The Psychic zeroes any play under five cards,
and at hand size >=10 the cap enumerates no five-card subset, so every reachable
play scores zero. It bit 2 of 45 greedy-shop Psychic blinds and 0 of 26 random:
657P5QGW and PM4RVISW both lost at ante 1 with 0/600 and both clear 720/600 at a
raised cap. Reproducible from the shipped records via scripts/inspect_run.py.

Also corrects the count this document published at launch: 19/240 and 8/240
reproduce exactly, but from a terminal-state estimate that omitted Troubadour and
could not see transient hand size. Said explicitly rather than swapped silently.

README.md and docs/adding-an-agent.md both promised the paired difference
isolates the shop policy -- "and nothing else", "alone". The evidence makes that
false, and shipping the admission only in known-limits would leave the front door
overclaiming.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@idIing

idIing commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Status update, and a correction to the reason this was being held.

This PR was held on the expectation that the fidelity fix would invalidate the numbers it documents to three decimals. It did not. #6 lands protocol v2 — agents are now offered every action the engine makes legal — and all three baselines return byte-identical per-seed outcomes on the 240-seed train split. random-shop and greedy-shop have zero differing summary fields; the headline paired delta is unchanged at +1.567 [+1.400, +1.729]. So this PR's scan-cap numbers still stand.

Independently verified before that: this refactor is behaviourally inert. Over 24 games × 2 arms, main vs this branch with no flag, and this branch with an explicit --score-budget 300, produce raw JSONL identical apart from run_id/started_at. --score-budget is genuinely wired (12 seeds: greedy-shop 3.000 → 3.083 at budget 8000, random-shop unchanged), stamps agent.tactical = GreedyTactical(score_budget=8000), and fails fast on random-legal. The Psychic claim reproduces exactly — 657P5QGW ante 1→3 with three High Card score=0 plays, PM4RVISW ante 1→4 with four — and it is not a sim artifact: bl_psychic … debuff = {h_size_ge = 5} (Balatro/game.lua:280), Balatro/blind.lua:527 returns true below five cards, and Balatro/functions/state_events.lua:614 skips the whole scoring block when debuffed, so the live hand scores zero too.

Two things to fix before this merges:

  1. Rebase onto v2. protocol v2: give agents the blind-select and cash-out decisions #6 touches known-limits.md (the scan-cap bullet's closing sentence, which said raising the cap "is a v2 question, not a v1 patch" — now stale phrasing) and renumbers the protocol doc. Expect a small conflict there.
  2. One doc command contradicts itself. known-limits.md instructs the reader to run --score-budget 8000, then four paragraphs later to inspect data/bench/greedy-shop.jsonl for the Psychic lock. Both use the default out-dir, so the first command overwrites the artifact the second reads — verified, 3 score=0 lines before, 0 after. Give one of them an explicit --out-dir.

idIing and others added 2 commits September 4, 2026 17:23
Resolutions, all three where v2 and this branch touched the same lines:

- `agents.py` keeps v2's `greedy-shop` description (it now declares
  never-skip and never-use-before-cash-out as policy) on top of this
  branch's `_greedy_shop_at(DEFAULT_SCORE_BUDGET)` factory.
- `test_public_contracts.py` keeps both sets of new tests. The sweep
  test's `PROTOCOL == "jackhammer/v1"` assertion becomes v2; what it
  exists to check is that a sweep is never stamped with the frozen
  protocol, whichever version that is. `test_protocol_is_v2` pins the
  version itself.
- `known-limits.md` takes this branch's rewritten scan-cap bullet
  wholesale. v2 had reworded the old bullet; this branch replaces it
  with a summary plus the measured § The tactical scan cap below, which
  supersedes the 19/240 and 8/240 figures v2's wording still carried.
  Its "raising the cap is a v2 question" now reads v3, since v2 shipped.

The follow-up commit reconciles the statements this merge left stale
elsewhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The merge left three statements false and one command self-destructive.

- The `--score-budget` help, the DIAGNOSTIC banner, and two comments
  said "not a v1 result" about a stamp that now reads v2. All now name
  the frozen protocol rather than a version number, so the next bump
  does not restale them.
- The sweep artifact field `v1_score_budget` becomes
  `frozen_score_budget` for the same reason. It has never shipped -- the
  flag that writes it is landing in this PR.
- `agents.py` still claimed the paired difference measures the shop
  policy "and nothing else". That is the exact overclaim this PR removed
  from `README.md` and `adding-an-agent.md`; it survived in a comment.
- The sweep command wrote to `--out-dir`'s default `data/bench/`, which
  is where the README's baseline run lands -- so following this page in
  order overwrote the records its next paragraph asks you to inspect,
  and after a raised-cap re-run seed 657P5QGW clears the blind instead
  of locking up. The command now names its own out-dir. The inspect
  command is also no longer described as reading "published records":
  `data/` is gitignored and ships empty, so it reads your own run.

Verified: `uv run pytest tests/` 78 passed, 1 deselected; `ruff check
src scripts tests` clean. Full 240-seed battery reproduces the published
numbers -- greedy-shop 3.204, random-shop 1.637, paired +1.567
[+1.400, +1.729], random-legal 1.000. The cited lockup reproduces:
`inspect_run.py --seed 657P5QGW` prints exactly three
`High Card score=0` plays.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@idIing
idIing merged commit 8de6822 into main Sep 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant