Commit 28dc3a1
committed
Replace the eval tasks with four multi-turn tasks
The three tasks were single-request contracts in toy domains, and the
one that looked saturated (project-doc, 8/10) was failing on a verifier
that rejected a literal reading of its own prompt. None was
long-horizon, only one had a second turn, and nothing checked whether
the agent's words matched the data.
Four tasks replace them, all multi-turn, verified through the Gadget's
RPC against references the verifier owns:
- change-calendar: a maintenance-window calendar built to an exact
contract, written up as a Document from its own data, changed under
new rules, committed and reloaded, then asked a question whose
answer is in the seeded schedule.
- worker-logs: a Workers request-log analyser fed a seeded day of
events with one planted bad hour; filters and rankings are added,
then the agent is asked which Worker had the worst hour, in a fixed
reply shape checked against the data.
- incident-desk: an on-call desk where twenty responders acknowledge
at once and exactly one may win, then escalation and metrics.
- chess: a complete engine written without a library, then PGN, then
draw detection, each checked differentially against chess.js on
curated positions and seeded random games. Either en passant FEN
convention is accepted.
The verifier now sees the agent's chat replies for the question turns,
and the verification budget grows to four minutes: these checks make
hundreds of RPC calls a turn.1 parent de83fa0 commit 28dc3a1
14 files changed
Lines changed: 1625 additions & 996 deletions
This file was deleted.
0 commit comments