Skip to content

Commit 28dc3a1

Browse files
committed
Replace the eval tasks with four multi-turn tasks
The three tasks were single-request contracts in toy domains, and the one that looked saturated (project-doc, 8/10) was failing on a verifier that rejected a literal reading of its own prompt. None was long-horizon, only one had a second turn, and nothing checked whether the agent's words matched the data. Four tasks replace them, all multi-turn, verified through the Gadget's RPC against references the verifier owns: - change-calendar: a maintenance-window calendar built to an exact contract, written up as a Document from its own data, changed under new rules, committed and reloaded, then asked a question whose answer is in the seeded schedule. - worker-logs: a Workers request-log analyser fed a seeded day of events with one planted bad hour; filters and rankings are added, then the agent is asked which Worker had the worst hour, in a fixed reply shape checked against the data. - incident-desk: an on-call desk where twenty responders acknowledge at once and exactly one may win, then escalation and metrics. - chess: a complete engine written without a library, then PGN, then draw detection, each checked differentially against chess.js on curated positions and seeded random games. Either en passant FEN convention is accepted. The verifier now sees the agent's chat replies for the question turns, and the verification budget grows to four minutes: these checks make hundreds of RPC calls a turn.
1 parent de83fa0 commit 28dc3a1

14 files changed

Lines changed: 1625 additions & 996 deletions

‎packages/workshop-evals/evals/appointment-desk.eval.ts‎

Lines changed: 0 additions & 242 deletions
This file was deleted.

0 commit comments

Comments
 (0)