Jackhammer

Heavy machinery for a card game.

Jackhammer is a small, reproducible benchmark for Balatro agents. It runs agents on the same published seeds, pins the exact Jackdaw simulator commit, joins results by seed, and reports uncertainty. It is a referee, not a claim that these are strong agents or that Jackdaw is identical to live Balatro.

The useful loop is deliberately short:

install -> run a baseline -> add an agent -> compare on identical seeds -> inspect failures

The baselines

Display name	Stable agent ID	What it does	Mean highest ante
Random Legal	random-legal	Samples a legal action type uniformly, then a legal target; no tactical layer	1.000
Random Shop	random-shop	GreedyTactical card play; random legal shop actions	1.637
Cheapest-Joker Shop	greedy-shop	GreedyTactical card play; buys the cheapest affordable Joker	3.196

Those are the complete 240-seed v2.1 results on the idIing/jackdaw-balatro fork of Jackdaw at de733eb, the exact commit the lockfile installs; all three went 0/240 on wins. The stable ID greedy-shop predates the clearer display name and is retained so existing artifacts stay comparable. The paired Random Shop -> Cheapest-Joker Shop difference is +1.558 ante with a bootstrap-95 interval of [+1.396, +1.721].

v2.1 re-pinned the engine and fixed four game_state keys preview_play handed the scorer inside GreedyTactical's ranking; only greedy-shop moved, and by 0.008 ante. Artifacts stamped jackhammer/v2 were produced at the previous pin and are not directly comparable — see what changed.

The shared card play: GreedyTactical

random-shop and greedy-shop both play their cards with GreedyTactical (src/jackhammer/playground/harness.py) and differ only in what they do in the shop. In one sentence: on every play decision it exact-scores up to 300 legal card subsets, plays the best hand if that clears the blind, and otherwise discards to dig for a better one.

Holding it fixed is the point — it plays every hand both agents play, so the paired difference between them is a shop-policy contrast, not two different card players. random-legal does not use it at all, which is what makes it an honest floor rather than a third variation on the same player.

The 300 cap is exhaustive for a standard 8-card hand, where the complete set of ≤5-card subsets is 218. Above 8 cards it truncates, and it drops the largest subsets first. It also does not bind equally on the two arms: shopping grows the hand, so greedy-shop truncates ~2.4x as often, and the +1.558 understates the contrast by about 0.1 ante — see known limits.

This ladder is intentionally weak. greedy-shop does not understand Joker text, quality, rarity, or synergy; it never rerolls or sells, it does not buy vouchers or consumables, and it never opens a booster pack. Over the whole battery it emits 6 of the engine's 21 action types; random-shop emits 16. So the +1.558 prices buying the cheapest Joker at all against an arm that touches most of the shop at random — the narrowest agent on the slate beating the widest one. Every run records that histogram and the evaluator checks it against each baseline's published description (declared repertoires). That gap is an open contribution surface, not something hidden behind a flattering name.

Install

Prerequisites: Git and uv. Python 3.12 and the exact simulator source are resolved by the lockfile.

git clone https://github.com/idIing/jackhammer.git
cd jackhammer
uv sync --locked
uv run python scripts/evaluate.py --list

To use the kit as a library from your own project instead, install it — the frozen seed battery and the pinned simulator commit come with it, so a benchmark run from an install is attributable:

pip install "jackhammer-benchmark @ git+https://github.com/idIing/jackhammer.git@v1.1.0"

from jackhammer.bench import agents, provenance
from jackhammer.playground.harness import run_battery_with
from jackhammer.playground.seeds import load_battery

spec = agents.get("greedy-shop")
results = run_battery_with(load_battery("train")[:8], spec.make_decider, out_path="runs.jsonl",
                           config_label=spec.name, slot1=spec.slot1, slot2=spec.slot2)
print(provenance.kit_pin())   # {'version': '1.1.0', 'commit': None, 'dirty': None}

scripts/ is not installed; the CLI below runs from a clone. An installed copy reports kit.commit: null because it has no checkout — kit.version is the field that identifies it.

Expected list output:

  greedy-shop      Buys the cheapest affordable joker; fixed greedy tactics; never skips a blind, never uses a consumable before cash-out.
  random-legal     Uniformly-random legal action in every phase; unlike the shop baselines it may skip a blind and may act at cash-out. The floor.
  random-shop      Uniformly-random legal shop action; fixed greedy tactics; never skips a blind, never uses a consumable before cash-out.

Run the benchmark

A quick smoke run takes only the first eight public training seeds:

uv run python scripts/evaluate.py \
  --agent greedy-shop --vs random-shop --limit 8 --workers 4

Remove --limit 8 for the reportable comparison over all 240 seeds. The command writes raw JSONL decision records, one result artifact per arm, and a paired comparison under data/bench/. The reportable run prints (artifact paths omitted here):

greedy-shop: 240 runs
  mean highest ante: 3.196
random-shop: 240 runs
  mean highest ante: 1.637
paired (240 seeds): greedy-shop - random-shop = +1.558 ante [+1.396, +1.721] boot-95
  interval excludes zero

See the frozen protocol for what that claim means and the artifact contract for the machine-readable output.

Inspect a failure

uv run python scripts/inspect_run.py data/bench/greedy-shop.jsonl --worst
uv run python scripts/inspect_run.py data/bench/greedy-shop.jsonl --seed PVRQ4K5A

The viewer prints the selected run's outcome, final resources, Jokers, and decision timeline. It does not require the game client.

Add an agent

The public interface is AgentSpec(name, description, make_decider). A decider receives the live environment once per seed and returns a function that selects factored legal actions. Start with the agent-porting guide, register a new stable name, then compare it to the closest baseline with --vs.

Add a seed dataset

The headline protocol always means the committed 240-seed train split. New coverage, stress, curriculum, or seed-difficulty questions belong in versioned sidecar manifests:

uv run python scripts/evaluate.py \
  --agent greedy-shop \
  --dataset examples/datasets/coverage-example.json \
  --split sample

Custom datasets are stamped as jackhammer/dataset-eval/v2.1, never jackhammer/v2.1. That makes future questions such as “how does seed coverage affect measured agent strength?” additive without silently moving the headline benchmark. See Adding datasets.

Contributing

Read CONTRIBUTING.md and the project's known limits. The short version: preserve benchmark meaning, pair comparisons on identical seeds, report uncertainty, and describe simulator-only evidence as simulator-only evidence.

Jackhammer is MIT licensed. Balatro is by LocalThunk/Playstack; this project is unaffiliated and does not distribute game assets or source.