Adding an agent

An agent is registered with jackhammer.bench.agents.AgentSpec:

from jackhammer.bench.agents import AgentSpec, register


def make_decider(env, seed):
    policy = MyPolicy(env=env, seed=seed)

    def decide(raw_state, mask, history):
        action = policy.choose(raw_state, mask)
        return action, "my-policy", "choose", {}, False

    return decide


register(
    AgentSpec(
        name="my-agent-v1",
        description="One honest line shown by evaluate.py --list.",
        make_decider=make_decider,
        deterministic=True,
    )
)

Place the implementation in src/jackhammer/bench/agents.py for a small baseline, or import and register it there from a separate module. Registration happens when the evaluator imports the registry.

Decider contract

make_decider(env, seed) runs once per game. The returned callable receives:

raw_state: Jackdaw's current state dictionary;
mask: the current legal-action mask; and
history: prior non-fallback decision summaries.

It returns (FactoredAction, reasoning, method, params, was_fallback). The harness records this packet and the state transition. Always choose through the mask. A policy that previews actions may use env.get_state() / env.load_state() but must restore the environment before returning.

Randomized policies must derive their random generator from seed. If exact repeated replay is not expected, set deterministic=False; do not publish it as reproducible.

Your decider is called for every phase

Changed in protocol v2. The episode loop holds no policy of its own, so decide is called for every phase the engine offers an action in — including BLIND_SELECT and ROUND_EVAL, which the v1 loop played on the agent's behalf. An agent ported from v1 that only handles SELECTING_HAND and SHOP will now be asked something it has no branch for.

The two phases carry real decisions. SkipBlind is legal on Small and Big blinds only; it takes a tag and advances to the next blind. A consumable used at ROUND_EVAL frees its key before the next shop is rolled, so it can change what the shop offers — one step later, in the shop, is too late.

If you do not want to make those decisions, decline them explicitly, the way the shop baselines do:

_SELECT_BLIND = int(ActionType.SelectBlind)
_CASHOUT = int(ActionType.CashOut)

phase = str(raw_state.get("phase", "")).upper()
if "BLIND_SELECT" in phase and mask.type_mask[_SELECT_BLIND]:
    return FactoredAction(action_type=_SELECT_BLIND), "always-select-blind", "SelectBlind", {}, False
if "ROUND_EVAL" in phase and mask.type_mask[_CASHOUT]:
    return FactoredAction(action_type=_CASHOUT), "always-cash-out", "CashOut", {}, False

Declining is a legitimate policy and costs nothing measurable on the current slate. Say so in your description: the string is stamped into every artifact you publish, and a description that claims a broader action set than the agent uses is the exact defect v2 exists to fix.

Declaring what you actually do (optional, recommended)

Prose descriptions drift from policies, and no test catches it — the reference agent shipped for months described as taking pack cards it can never reach. So AgentSpec takes an optional declared_actions: the action types you claim appear in your decision stream on this battery.

register(
    AgentSpec(
        name="my-agent-v1",
        description="One honest line shown by evaluate.py --list.",
        make_decider=make_decider,
        declared_actions=("PlayHand", "Discard", "SelectBlind", "CashOut", "BuyCard", "NextRound"),
    )
)

scripts/evaluate.py then reads it back out of the recorded runs. Emitting an action you did not declare fails the evaluation at any sample size; declaring one your complete-battery run never contains fails it too (on a --limit smoke run that is reported as a sample-size caveat instead). Omit the field and your repertoire is reported and never enforced — every artifact still carries summary.repertoire, so a reader can see what your agent did either way. Names come from jackhammer.bench.repertoire.ALL_ACTIONS; a name that is not an action type is rejected at registration.

Leave was_fallback false for decisions your policy actually made. It means "the harness substituted for the agent", not "the agent had no preference" — a policy that internally reaches for a default still returns an ordinary action and is recorded as having decided.

The shared tactical layer

The two shop baselines are not written from scratch. Both compose build_decider(env, GreedyTactical(), <shop policy>, MarginValue()), where GreedyTactical (src/jackhammer/playground/harness.py) is a fixed in-blind policy: it exact-scores up to score_budget=300 legal play-card subsets, then clinches or digs. Holding it constant is what makes a paired comparison a shop-policy contrast — imperfectly, though: the 300-subset cap binds more often on greedy-shop, worth about 0.1 ante, so a margin measured against it carries that handicap on the affected seeds. --score-budget re-runs either baseline at another cap if you want to size that against your own agent. See known limits.

You are free to replace it — an agent that plays cards better is a legitimate submission — but say so, because a comparison against greedy-shop then measures both layers at once, not just the shop. Set AgentSpec.tactical to a label for your in-blind policy: it is written into the result artifact's agent identity, so a reader can see which layers differed.

Compare it

First test registration, then run a smoke:

uv run python scripts/evaluate.py --list
uv run python scripts/evaluate.py \
  --agent my-agent-v1 --vs greedy-shop --limit 8 --workers 4

Audit raw runs with scripts/inspect_run.py, and read the repertoire line the evaluator prints — an agent doing far less than you think is not visible in its mean. Only after that should you remove --limit for the full 240-seed comparison. Report negative or null intervals as results, not invitations to tune on the same public battery until the sign changes.

Never change an existing stable agent ID's behavior. A behavioral revision gets a new ID.

Watching a run

Sometimes the question is not "how deep did it get" but "what happened on the way". Pass an observer= to run_battery or run_battery_with and you get told, without changing the run:

from jackhammer.playground.harness import run_battery_with

class LegalityCoverage:
    def __init__(self):
        self.offered = []

    def observe_decision(self, state, mask, action):
        self.offered.append((sorted(mask.type_mask.nonzero()[0].tolist()),
                             int(action.action_type)))

observer = LegalityCoverage()
run_battery_with(seeds, spec.make_decider, "runs.jsonl", "coverage", observer=observer)

Two hooks, at two layers. They are independent — implement whichever one you need, both, and inherit nothing: the kit matches them by name, not by base class. (NullObserver is available as a base with both as no-ops if you would rather subclass.)

observe(state, *, event=None) — the raw engine state after the reset and after every step, with a label for the action that produced it. This is the layer that sees both sides of a step, so it is where a state-transition check belongs.
observe_decision(state, mask, action) — the state and legal-action mask your decider was handed, and the action it returned. The engine layer cannot reconstruct the mask.

An object with neither hook raises TypeError rather than running silently: a misspelled hook would otherwise produce a run that looks instrumented and records nothing.

Neither hook sees the checkpointed previews an agent runs while deciding. A search agent get_state/load_states and re-steps the engine dozens of times per decision; counting those would make the trace a record of what the agent considered rather than of what happened, so the kit suspends observation for the duration of each decider call.

The event labels

event is either "run_reset" — the one label that is not an action, so it can never collide with one — or an action name from the same vocabulary as AgentSpec.declared_actions and the repertoire report: PlayHand, SelectBlind, SellJoker, SortHandRank, and so on. One label carries a qualifier after a colon, RedeemVoucher:v_hieroglyph, because the voucher's identity is an index into the shop that the redemption itself removes.

Match on event.split(":", 1)[0] if you want the action and not the qualifier. Do not match on the engine's class names: jackdaw spells two actions with a discriminating field where this vocabulary spells four (SellCard.area, SortHand.mode), so SellCard is not a name this kit uses anywhere.

Pure-read, and what that costs you

An observer is pure-read instrumentation. It is called for side effects, its return value is ignored, and a run with one produces the same numbers as a run without — mutating the state you are handed corrupts the run you were measuring.

The state you are handed is the live engine dict, not a snapshot. DirectAdapter is zero-copy by design: raw_state is the engine's own object, and the engine keeps stepping it in place. Reading inside the call is correct and free. Storing it is not — consecutive observations are often literally the same object, so a list of stored states resolves, later, to whatever the engine last wrote. If you need to keep one, copy it:

from jackdaw.engine.fastcopy import fast_deepcopy

def observe(self, state, *, event=None):
    self.kept.append(fast_deepcopy(state))   # ~0.2 ms; a few percent of a battery run

That cost is charged only to the observers that need it, which is why the kit does not copy for you.

FilesExpand file tree

adding-an-agent.md

Latest commit

History