Replace the eval tasks with four multi-turn tasks

[AshishKumar4]

This PR replaces the current existing toy tasks with more relevant, multi-turn tasks.
The set of tasks are designed to sit on a spectrum of trivially solvable to medium-hard, so we can have a set of tasks that can act as regression tests that fail if harness quality or prompt degrade, or tasks that have a high margin of improvement which can help us evaluate improvements to harness.

All are multi-turn and verified through the Gadget's RPC against references the verifier owns, so any correct implementation passes.

change-calendar (4 turns). A maintenance-window calendar for five services, built to an exact contract with five error codes. Turn 2 writes the week up as a Document, and the check walks the document's headings and bullets against the calendar's own seeded windows. Turn 3 changes two rules; existing windows must survive, and verifyAfterAccept commits, reloads, and checks both data and rules persisted. Turn 4 asks how many hours one service has that week, answer as a bare number.

worker-logs (3 turns). A Workers request-log analyser. The verifier pushes a seeded day of 5,869 events for six Workers across four colos, with one planted worst hour and one decoy, and computes every reference answer itself (nearest-rank p95, half-open ranges, hourly buckets with empty hours). Turn 2 adds colo and route filters, a per-colo breakdown and a slowest-routes ranking. Turn 3 asks which Worker had the worst hour by error rate, in a fixed three-line reply shape; the check parses it and compares with the planted answer. That is the fabricated-answer failure made deterministic.

incident-desk (2 turns). An on-call desk where twenty responders acknowledge the same page at once: exactly one wins, everyone else is told who. Durable Object RPCs interleave at every await, so a check-then-write implementation fails. Turn 2 adds escalation and metrics computed from the board's own timestamps.

chess (3 turns). A complete engine with no library available in the sandbox, then PGN import/export, then draw detection. Verification is differential against chess.js: legal move sets, resulting positions and status must agree on twelve curated positions (castling through check, en passant that would expose the king, promotion by capture, pins, mate, stalemate) and on seeded random games with illegal-move probes; PGN import covers three miniatures, the Opera Game, and two seeded random games; export must replay in the oracle to the same position. Both en passant FEN conventions are accepted by canonicalising through the oracle, and once a game is drawn with pieces still mobile either an empty or a full move list is accepted.

Two small harness changes: the verifier receives the agent's chat replies for the turn (EvalVerifier.replies), and the verification budget is four minutes rather than two, since these checks make hundreds of RPC calls per turn. chess.js is a dev dependency as the oracle.

First live run (GPT 5.6 Luna, 10 trials each, 25 minutes, $1.21, no infrastructure failures, verification under 3 s per turn):

Task	Pass	What the misses were
change-calendar	10/10	all ten answered 6.5
incident-desk	9/10	time-to-resolve measured from acknowledgement; the prompt now says from openedAt
worker-logs	6/10	in turn 2, four agents called reset() while self-testing and destroyed the user's loaded data, against an explicit instruction
chess	5/10	castling rights dropped after 1. Nh3 and after an opponent's Rxc1; O-O-O with a knight on b1; king stepping into a pawn's attack; en passant not generated

Every verifier was run against a minimal correct implementation and a broken one before this was opened; that caught a key-order comparison bug in worker-logs that would have failed every agent. The probes are not committed.

[github-actions]

Preview: pr525-evals-tasks

https://pr525-evals-tasks-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[ask-bonk]

Posted 9 actionable inline findings.

github run

[ask-bonk]

Performance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 83.3% to 75.0%, while mean model turns rose from 8.3 to 18.4 and tool calls from 9.4 to 19.7. Candidate trajectories carried state across turns, notably creating a document from calendar data before editing rules and querying results. Different workloads and 10 stochastic trials per cohort could explain the apparent decline.

github run

[ask-bonk]

• [P1] change-calendar.eval.ts:148: The duplicate probe is also an overlap. A valid overlap-first implementation returns OVERLAP; exact-record deduplication also passes. Use the same ID with non-overlapping fields.
• [P1] chess.eval.ts:456: Turn 3 never rechecks loadPgn() or pgn(), so draw-detection changes can break or remove PGN support while the overall task passes.
• [P1] src/harness.ts:111: Replies are sliced after the previous snapshot, not after the current user prompt. Late callbacks from the prior activation can therefore be attributed to the current turn.
• [P2] change-calendar.eval.ts:131: All range queries contain each returned window’s start. An implementation filtering by start time instead of interval overlap passes.
• [P2] change-calendar.eval.ts:268: Spacing tests cover only later windows and not the exact 24-hour boundary, allowing one-directional or off-by-one implementations.
• [P2] incident-desk.eval.ts:161: lateAck and unknown-incident checks validate only error codes, not the contractually required owner: null.
• [P2] incident-desk.eval.ts:246: Optional escalations plus ?? 0 lets unchanged incidents omit the newly required field.
• [P2] worker-logs.eval.ts:354: byColo() is never called with its required route filter, so implementations may ignore it.
• [P2] change-calendar.eval.ts:347: The “number only” verifier accepts 6.5 hours..
• [P2] worker-logs.eval.ts:377: The fixed three-line format accepts code fences, reordered labels, arbitrary parseable dates, omitted %, and unrestricted decimal precision.

github run

[ask-bonk]

Performance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 86.7% to 67.5%, while mean model turns rose from 9.8 to 17.7 and tool calls from 12.1 to 19.8. Candidate trajectories sustained state across turns, notably querying calendar data, creating a document from it, then editing and rechecking the Gadget. Different workloads and only 10 stochastic trials per cohort could explain the apparent regression.

github run

[github-actions]

Eval runs comparison

Baseline 87e09feb vs candidate e53cab91 · gpt-5.6-luna · 10 trials per task.

No cohort is comparable.

Not compared:

• only in baseline: appointment-desk 9/10, expense-ledger 5/10, project-doc 9/10
• only in candidate: change-calendar 10/10, chess 7/10, incident-desk 7/10, worker-logs 6/10

Run · trajectories and raw results

[ask-bonk]

• [P1] incident-desk.eval.ts:143: Contract restricts severity to 1 | 2 | 3, but line 162 sends 4. A correctly validated RPC rejects before returning INVALID_SEVERITY. Accept number and validate in the implementation.
• [P1] incident-desk.eval.ts:267: Filtered metric expectations derive services from the Gadget’s own board. An implementation storing every incident under edge-cache passes. Compare stored services against the original open() inputs.
• [P1] change-calendar.eval.ts:189: Overlap verification only tests a window starting inside another. An implementation accepting a window that fully contains an existing window passes. Add a containment probe.
• [P1] chess.eval.ts:468: Threefold repetition is tested only while castling rights remain unchanged. Counting board layout and side-to-move alone therefore passes, despite castling rights being part of repetition identity.
• [P2] chess.eval.ts:403: Invalid-PGN rollback checks only FEN. An attempted import can corrupt move history while restoring the board and still pass. Compare canonical history before and after.
• [P2] worker-logs.eval.ts:40: Events are ingested in alphabetical worker order, so unsorted summary() output passes. Ingest workers in a non-alphabetical order.

github run

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

⚠️ 1 issue in files not directly in the diff

⚠️ Reconnects can undercount turn cost

A reconnect before delayed gateway accounting makes #reconnectedDuringTurn skip the cost wait. The completed turn then reports stale cumulative cost.

[ask-bonk]

• [P1] chess.eval.ts:181: Castling is never tested while in check or into check. An engine validating only the transit square passes. Add both positions.
• [P1] incident-desk.eval.ts:160: Stored severity is never compared with the requested severity. An implementation storing every incident as severity 3 passes.
• [P1] worker-logs.eval.ts:154: The planted worst-rate hour also has the most errors. Selecting by error count instead of rate therefore passes.
• [P2] change-calendar.eval.ts:77: All windows are inserted chronologically, so returning insertion order passes without implementing the required sorting or equal-start ID tie-break.
• [P2] change-calendar.eval.ts:315: No accepted non-billing window exceeds five hours. An incorrect five-hour maximum passes the eight-hour contract.
• [P2] change-calendar.eval.ts:337: Post-reload checks verify only the spacing rule, not persistence of the new billing cap.
• [P2] chess.eval.ts:83: FEN canonicalization erases any non-capturable en-passant square. A wrong square such as a3 after e2e4 passes as equivalent to -.
• [P2] chess.eval.ts:447: PGN export accepts any result token. The check permits the Opera Game to end in 0-1 or 1/2-1/2.
• [P2] incident-desk.eval.ts:80: The successful acknowledgement response’s owner is never checked. It may disagree with the stored winner and still pass.
• [P2] worker-logs.eval.ts:345: summary() never combines colo and route, so implementations that apply only one of those filters pass.

github run

[ask-bonk]

Eval runs review

Performance is inconclusive because every task cohort is non-comparable. Across unmatched cohorts, pass rate fell from 70.0% to 42.5%, while mean turns rose from 7.1 to 14.5 and tool calls from 9.0 to 17.3. Candidate runs sustained multi-turn state, notably querying calendar data, creating a document, then editing and rechecking rules. Replaced workloads and only 10 stochastic trials per cohort could explain the apparent decline.

github run

[ask-bonk]

Eval runs review

Performance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 76.7% to 40.0%, while mean turns rose from 7.8 to 15.4 and tool calls from 9.5 to 16.8. Candidate runs sustained state across turns, notably querying calendar data, creating a document, editing rules, and re-querying results. Replaced workloads and only 10 stochastic trials per cohort could explain the apparent decline.

github run

[ask-bonk]

• [P1] worker-logs.eval.ts:361: byColo() and slowestRoutes() only receive FULL_DAY; implementations may ignore both range fields and pass. Add partial-range probes.
• [P2] chess.eval.ts:442: chess.js accepts coordinate moves and missing move numbers. Comparing replay history therefore permits non-PGN output like e2e4 e7e5, violating the export contract. Validate the exported notation itself.
• [P2] incident-desk.eval.ts:282: The final board check verifies only ordering. board() may omit newly created race-6 and esc-1, while self-derived metrics still pass. Assert the complete expected ID set.
• [P2] change-calendar.eval.ts:257: Literal clock matching rejects valid forms such as 11:00 PM UTC, but accepts 23:00 PST. Parse the stated time or require and verify an explicit UTC format.
• [P2] change-calendar.eval.ts:197: conflicts() never tests touching ranges, allowing inclusive-boundary overlap logic despite the contract.
• [P2] incident-desk.eval.ts:29: Timestamp fields accept any string. Values such as "0" are parseable and can satisfy ordering and metrics despite not being ISO 8601.
• [P2] incident-desk.eval.ts:276: Unknown-service metrics do not verify acknowledged or resolved; leaked global values pass when count is zero.
• [P2] src/verifier.ts:42: agentReplies() includes empty tool-only messages, contrary to its documented semantics. Because reply checks use .at(-1), a trailing empty agent record can hide a valid textual answer.
• [P3] worker-logs.eval.ts:243: Only the aggregate accepted count is checked. Individual ingest() calls may return incorrect compensating counts. Validate each response against its batch length.

workshop-evals tests and TypeScript checks pass.

github run

[ask-bonk]

• [P1] worker-logs.eval.ts:253: Preservation checks compare only full-day aggregates. Moving existing error timestamps can change the true worst hour while every check and the static answer still pass. Verify timestamp-sensitive hourly data after turn 2.
• [P1] incident-desk.eval.ts:101: Metrics expectations trust the Gadget’s timestamps. An acknowledge implementation that incorrectly sets resolvedAt passes races, then its false resolved count becomes the expected value. Assert lifecycle timestamp consistency independently.
• [P1] chess.eval.ts:489: Fifty-move verification tests only crossing 99 to 100. An implementation whose counter never resets after pawn moves or captures passes but declares premature draws. Add both reset cases.
• [P2] worker-logs.eval.ts:364: Every seeded route belongs to one Worker, so grouping only by route passes despite the required worker-route grouping. Seed a shared route across Workers.
• [P2] change-calendar.eval.ts:157: Operating-hour boundaries are untested. Implementations rejecting valid 03:xx starts or accepting 04:xx pass. Add boundary probes.
• [P2] change-calendar.eval.ts:197: conflicts() can ignore service; the only cross-service overlap begins exactly at the query’s exclusive end. Add a genuinely overlapping different-service window.

github run

[ask-bonk]

Eval runs review

Measured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist.
Trajectories. In change-calendar turn 2, the candidate queried the calendar and generated a document from the returned windows.
Caveats. All tasks were replaced, and each cohort has only ten stochastic trials.
Verdict. inconclusive — no task cohort is comparable

github run

[ask-bonk]

Posted 8 new actionable inline findings.

github run

[ask-bonk]

• [P1] chess.eval.ts:165: Illegal probes only move current-player pieces. move() may incorrectly allow moving an opponent’s piece and still pass.
• [P1] chess.eval.ts:400: Imported games are never continued. An implementation rejecting every move after loadPgn() passes.
• [P1] incident-desk.eval.ts:228: Turn 2 never rechecks resolve(), allowing the update to break or remove resolution support.
• [P2] chess.eval.ts:63: Concatenating move fields makes malformed output such as {from:"e", to:"2e4"} equivalent to e2-e4.
• [P2] change-calendar.eval.ts:160: No zero-duration range is tested. Implementations checking end < start instead of end <= start pass.
• [P2] incident-desk.eval.ts:164: resolve() is never called with an unknown ID, so the required UNKNOWN_INCIDENT response is unverified.
• [P2] worker-logs.eval.ts:304: Loaded-data queries never end with an empty hour, allowing trailing empty buckets to be omitted.
• [P2] worker-logs.eval.ts:364: Only limits 3 and 5 are tested. An implementation permanently capping results at five passes.

github run

[ask-bonk]

Eval runs review

Measured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist.
Trajectories. In change-calendar turn 2, the candidate queried the calendar and generated a document from the returned windows.
Caveats. All tasks were replaced, and each cohort has only ten stochastic trials.
Verdict. inconclusive — no task cohort is comparable

github run

[ask-bonk]

• [P1] change-calendar.eval.ts:294: Every newly accepted turn-3 window is canceled before listing or reload. A transient, non-persisting schedule() implementation can pass. Query accepted windows before cancellation.
• [P1] chess.eval.ts:477: Threefold repetition never varies en-passant rights. A board-and-turn-only repetition key passes despite incorrectly counting positions where en passant was previously available.
• [P1] incident-desk.eval.ts:168: The only successful resolution uses owner alice. Hard-coding Alice as the authorized resolver passes; resolve a raced incident using its actual winner.
• [P1] worker-logs.eval.ts:293: sorted[Math.floor(0.95 * n)] produces every asserted p95 value. Groups exposing this nearest-rank off-by-one exist but fall outside the tested top-N results.
• [P2] chess.eval.ts:288: No imported PGN contains a promotion, so loadPgn() may reject all promotion SAN while passing.
• [P2] chess.eval.ts:392: No import tests [SetUp]/[FEN] tags. An implementation may honor a custom starting position despite the contract requiring the standard start.
• [P2] chess.eval.ts:243: Promotion execution only supplies valid q/r/b/n values. Treating an invalid value such as "k" as queen promotion passes.
• [P2] incident-desk.eval.ts:267: Exactly one incident is resolved, making sum and mean resolution duration identical. Add another resolved incident with a distinct duration.
• [P2] change-calendar.eval.ts:262: Heading substring matching lets api-gateway / auth pass, despite auth being required to remain omitted.
• [P2] worker-logs.eval.ts:58: All timestamps use identical second-precision formatting. Lexicographic timestamp comparison passes but mishandles valid fractional timestamps such as 14:00:00.500Z.
• [P2] worker-logs.eval.ts:346: Loaded-data summaries always match at least one request. Returning zero-count rows for a no-match filter therefore passes despite the contract requiring no rows.

github run

[ask-bonk]

• [P1] change-calendar.eval.ts:277: Spacing probes do not distinguish end-to-start from start-to-start spacing. An implementation measuring between starts passes all checks despite violating the rule.
• [P2] change-calendar.eval.ts:153: Duplicate IDs are tested only within the same service. Per-service ID uniqueness passes despite IDs and cancellation being global.
• [P2] worker-logs.eval.ts:302: Loaded hourly() queries never place events exactly at outer range boundaries. Inclusive-end filtering can pass; querying loaded AFTERNOON data would exercise exact 12:00 and 16:00 events.
• [P2] chess.eval.ts:509: Material fixtures omit opposite-colored bishops. An engine declaring every K+B versus K+B position insufficient passes, although chess.js reports opposite-colored bishops as non-drawn.

workshop-evals build and 62 tests pass.

github run

[ask-bonk]

• [P1] worker-logs.eval.ts:341: Turn 2 never rechecks ingest(). The extension can break future ingestion while retained-data queries still pass.
• [P1] chess.eval.ts:194: Promotion is only exercised for White. An engine without Black promotion passes.
• [P2] chess.eval.ts:189: Queenside castling is never tested with b1/b8 occupied while c and d are clear.
• [P2] chess.eval.ts:191: No test captures an unmoved corner rook. Castling rights may incorrectly survive that capture.
• [P2] change-calendar.eval.ts:260: The document’s internal title is never compared with PLAN; an “Untitled document” can pass.

github run

[ask-bonk]

• [P1] pnpm-lock.yaml:9690: The removed Vitest snapshot remains referenced by packages/ui. Frozen install fails with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY.

Posted 3 additional actionable inline findings.

github run

[ask-bonk]

Eval runs review

Measured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist.
Trajectories. In change-calendar turn 2, the candidate queried the calendar and generated a document from the returned windows.
Caveats. All tasks were replaced, and each cohort has only ten stochastic trials.
Verdict. inconclusive — no task cohort is comparable

github run

[ask-bonk]

• [P1] chess.eval.ts:367: The shared-game check is sequential. Two concurrent legal White moves can both succeed and race to overwrite storage. Test simultaneous moves over separate connections and require exactly one success.
• [P1] chess.eval.ts:448: PGN export is read through the connection that created the history. Persisting only FEN while keeping PGN history connection-local passes. Reconnect before exporting.
• [P1] chess.eval.ts:485: Threefold repetition is built and checked through one connection. Session-local repetition counts pass despite the shared-game requirement. Reconnect before checking final status.
• [P1] incident-desk.eval.ts:283: Turn 2 lacks post-accept verification. Escalation state can be transient and disappear after reload while all checks pass. Add verifyAfterAccept coverage for severity and escalation count.
• [P1] worker-logs.eval.ts:30: checkout-api has both the highest full-day error rate and worst hour. An incorrect algorithm that first selects the worst daily Worker, then checks only its hours, passes. Seed another Worker with a higher distributed daily rate.
• [P2] worker-logs.eval.ts:296: The reset check validates only bucket count and zero requests. Stale error counts or incorrect hourIso values pass. Compare against referenceHourly(AFTERNOON).

github run

[ask-bonk]

• [P1] change-calendar.eval.ts:297: Spacing is never tested between two newly added same-service windows. Enforcing TOO_CLOSE only against seeded windows passes.

• [P1] chess.eval.ts:273: Special moves validate only the returned FEN and status, not persisted FEN. Castling can return the correct FEN while storing an unmoved rook; the next fixture reload hides corruption.

• [P1] incident-desk.eval.ts:275: State is not read after the second escalation. It may succeed without mutation, while the third mutates to severity 1 but returns AT_MAX_SEVERITY; all assertions pass.

• [P2] incident-desk.eval.ts:265: Several turn-two open() results are ignored. Returning an error while still persisting race-6, esc-1, or billing incidents passes downstream checks.

• [P2] worker-logs.eval.ts:340: Every range boundary is on an exact hour. Implementations that round arbitrary ISO endpoints to whole hours pass despite violating half-open range semantics.

github run

[ask-bonk]

Eval runs review

Measured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist.
Trajectories. In change-calendar turn 2, the candidate queried the calendar and generated a document from its windows.
Caveats. All tasks were replaced, and each cohort has only ten stochastic trials.
Verdict. inconclusive — no task cohort is comparable

github run

[ask-bonk]

• [P1] incident-desk.eval.ts:94: Turn 2 ignores race-open severity. Its summary identifies the winning attempt and expected severity, so a migration that changes severity still passes.
• [P1] chess.eval.ts:515,526: Draw-triggering move() results are ignored. Returning ILLEGAL_MOVE while committing the move passes both repetition and fifty-move checks.
• [P2] worker-logs.eval.ts:397: Full-day and SIN top-three route pairs are identical. Selecting candidates globally before applying the colo filter passes incorrectly.
• [P2] change-calendar.eval.ts:239: conflicts() results match insertion order, so the required ID sorting is not exercised.
• [P2] change-calendar.eval.ts:188: All timestamps use Z; textual-hour validation passes without converting offset timestamps to UTC.
• [P2] incident-desk.eval.ts:287: Escalation checks omit immutable fields such as summary and service, allowing escalation to corrupt them.
• [P2] chess.eval.ts:542: Material fixtures omit K+NN versus K, allowing overly broad insufficient-material detection to pass.

github run

[AshishKumar4]

All twelve read and handled in 4c5f0527, in one pass rather than one patch per item, since any verifier change after merge marks every cohort non-comparable and forces a fresh baseline. Ten as raised, one loosened, one declined.

• change-calendar.eval.ts:297 spacing between two new windows: agreed; two auth windows added in the same turn 22 hours apart, the second must be TOO_CLOSE.
• chess.eval.ts:273 persisted FEN after special moves: agreed; compareHere now reads fen() first, so the stored position is checked wherever moves and status are, including after every special move.
• incident-desk.eval.ts:275 second escalation not read back: agreed; every escalate is read back, each must change severity and the count and nothing else (escalatedOnce), and the refused third must leave the record identical.
• incident-desk.eval.ts:265 ignored open() results: agreed; the opens a check depends on go through mustOpen, which fails the check on a refusal.
• worker-logs.eval.ts:340 whole-hour boundaries: agreed for summary, whose contract is on the events; one range now runs 13:30 to 14:45. Not for hourly, where the prompt defines buckets as whole hours and a non-aligned range has no single right answer.
• incident-desk.eval.ts:94 race-open severity: agreed; its summary names the winning attempt, and the severity must be the one that attempt carried.
• chess.eval.ts:515,526 draw-triggering move() results: agreed; parsed and required ok, along with the fifty-move loadFen.
• worker-logs.eval.ts:397 full-day and SIN top three identical: agreed, and my doing; the planted tail now sits in FRA only, so a colo-filtered ranking is a different list.
• change-calendar.eval.ts:239 sorted equals insertion order: agreed; the touching window is mw-100, scheduled after mw-101 and sorting before it.
• incident-desk.eval.ts:287 escalation corrupting other fields: agreed; covered by escalatedOnce.
• change-calendar.eval.ts:188 offset timestamps: agreed on the probe, not on the code. 23:00+05:00 is 18:00 UTC and must be refused; a textual-hour parser accepts it. But the prompt defines no error for unsupported input, and an implementation that refuses offsets outright is not wrong, so the check asserts only that scheduling fails and nothing was added.
• chess.eval.ts:542 K+NN vs K: declined. chess.js, python-chess and Lichess call it sufficient; the rule summaries agents commonly follow call it insufficient; the prompt does not define it. A check there measures which convention the model recalled, not the harness. If we want it, the prompt has to define insufficient material, and that is a task change with its own cost.

One rule for all of these: a check that rejects a defensible reading of the prompt turns the task into a prompt-reading lottery, and with ten trials that noise is indistinguishable from a regression. Every check above is satisfiable by any correct reading of its prompt.