Replace the eval tasks with four multi-turn tasks - #525
AshishKumar4 wants to merge 11 commits into
Conversation
Preview:
|
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 9 actionable inline findings. |
|
Performance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 83.3% to 75.0%, while mean model turns rose from 8.3 to 18.4 and tool calls from 9.4 to 19.7. Candidate trajectories carried state across turns, notably creating a document from calendar data before editing rules and querying results. Different workloads and 10 stochastic trials per cohort could explain the apparent decline. |
51cb312 to
28dc3a1
Compare
|
|
Performance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 86.7% to 67.5%, while mean model turns rose from 9.8 to 17.7 and tool calls from 12.1 to 19.8. Candidate trajectories sustained state across turns, notably querying calendar data, creating a document from it, then editing and rechecking the Gadget. Different workloads and only 10 stochastic trials per cohort could explain the apparent regression. |
28dc3a1 to
b92b3f4
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Eval runs comparisonBaseline
No cohort is comparable. Not compared:
|
|
b92b3f4 to
b1d342d
Compare
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 1 new potential issue.
⚠️ 1 issue in files not directly in the diff
⚠️ Reconnects can undercount turn cost
A reconnect before delayed gateway accounting makes #reconnectedDuringTurn skip the cost wait. The completed turn then reports stale cumulative cost.
|
Eval runs reviewPerformance is inconclusive because every task cohort is non-comparable. Across unmatched cohorts, pass rate fell from 70.0% to 42.5%, while mean turns rose from 7.1 to 14.5 and tool calls from 9.0 to 17.3. Candidate runs sustained multi-turn state, notably querying calendar data, creating a document, then editing and rechecking rules. Replaced workloads and only 10 stochastic trials per cohort could explain the apparent decline. |
b1d342d to
9eacc4b
Compare
Eval runs reviewPerformance is inconclusive because no task cohort is comparable. Across unmatched cohorts, pass rate fell from 76.7% to 40.0%, while mean turns rose from 7.8 to 15.4 and tool calls from 9.5 to 16.8. Candidate runs sustained state across turns, notably querying calendar data, creating a document, editing rules, and re-querying results. Replaced workloads and only 10 stochastic trials per cohort could explain the apparent decline. |
|
|
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
|
One pass over the four verifiers with three rules: after a mutation, assert the return and re-read the whole record; give every parser one input shape a lazy parser gets wrong; for rules that interact, one case where the order matters. Each check stays satisfiable by any correct reading of its prompt: the offset-timestamp probe asserts only that scheduling fails, since the prompt fixes no code for it, and K+NN vs K is left out because the prompt does not define insufficient material and engines disagree. incident-desk: the turn-one predicate covers inc-1 after resolve too; escalation must change severity and the count and nothing else, and is read back after every call including the refused one; race-open's severity is checked against the attempt its summary names; the opens a check depends on fail the check when refused. chess: compareHere also reads fen(), so the stored position is checked wherever moves and status are, including after each special move; the PGN self-comparison strips tag pairs so a Gadget that adds a Date tag is not failed for it; the repetition and fifty-move moves are asserted. change-calendar: the touching window is mw-100 so conflicts() has to sort; two new windows in one turn 22 hours apart; a +05:00 start that is outside hours in UTC must fail; a bullet may state its length as hours and minutes. worker-logs: the planted slow tail sits in one colo, so a colo-filtered ranking differs from the day's; one summary range has endpoints inside an hour.
|
All twelve read and handled in
One rule for all of these: a check that rejects a defensible reading of the prompt turns the task into a prompt-reading lottery, and with ten trials that noise is indistinguishable from a regression. Every check above is satisfiable by any correct reading of its prompt. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
incident-desk: a refused escalation of a resolved incident must leave it unchanged, and one of an unknown id must not create it. change-calendar: seed a six-hour billing window, valid under turn 1's cap and over turn 2's, so a migration that drops windows the new rules would refuse fails "everything already scheduled stays". chess: after the start position has occurred three times, loading it again must not report a repetition, and the loaded fifty-move position must match the oracle before the move that draws.
|
Handled in
|
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 4 actionable inline findings. |
A run took as long as the sum of two lanes of slowest trials, because two eval files shared one runner at a time: chess 316 s plus change-calendar 135 s made the last run 451 s. Each eval file of each measured revision now gets its own runner, so a run takes as long as its slowest task, and a runner hosts 10 Workshops instead of 20. A plan job lists the eval files at each revision from a sparse checkout, and an assemble job per revision joins the per-task reports. It fails unless it has exactly the planned files, since a missing task would otherwise read as removed and be cached that way. The push baseline goes through the same jobs, so main and pull requests are measured the same way, and the cache key records the layout so older baselines are measured again rather than compared against. A failed candidate no longer stops a complete baseline from being cached.
|
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
Adds GPT 6 Luna to the suggested models and makes it the model eval baselines are measured on. It has GPT-5.6 Luna's window, output cap and doubled input price above 272K, so it gets the same compaction budget, and like GPT-5.6 it is not in pi's catalog, so both take the same synthesized openai-responses path. The suggested models are also the Workshop's model picker, so users can choose it too.
github-actions
Bot
added
kernel
|
LGTM! |
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
The probe resubmitted mw-101 unchanged, so it also overlapped mw-101, and the prompt does not say whether DUPLICATE_ID or OVERLAP wins. GPT 6 Luna answered OVERLAP in 3 of 10 trials, a defensible reading the check failed. The probe now reuses mw-101's id at a time that overlaps nothing.
|
LGTM! |
Eval runs reviewMeasured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist. |
chess: turn 3 checks that malformed FENs are still refused, and reaches insufficient material by a capture as well as by loading; PGN export must replay to the played position, not only the same moves; imports include games ending 0-1 and 1/2-1/2. Before turn 3 defines draws, only a dead position may list no legal moves. incident-desk: a turn-two open reads back exactly as submitted. change-calendar: no bullet anywhere in the plan, including before the first heading, may list a window outside the week. worker-logs: events arrive in a seeded random order, so ordering by first appearance fails.
The comparison comment opens with every cohort's pass rate as a ten-cell bar, green for passed and red for failed, compared or not, and marks each pass delta green, red or grey. The Bonk review gets a coloured verdict, marks deltas as rising or falling, and copies the bars from the comparison instead of drawing its own, so its numbers stay the comparison's.
|
The six items from the
|
|
LGTM! |
🔬 Eval runs review📊 Measured. No cohort is comparable, so no valid pass, duration, tool-error, or cost deltas exist.
|
1 participant
This PR replaces the current existing toy tasks with more relevant, multi-turn tasks.
The set of tasks are designed to sit on a spectrum of trivially solvable to medium-hard, so we can have a set of tasks that can act as regression tests that fail if harness quality or prompt degrade, or tasks that have a high margin of improvement which can help us evaluate improvements to harness.
All are multi-turn and verified through the Gadget's RPC against references the verifier owns, so any correct implementation passes.
change-calendar (4 turns). A maintenance-window calendar for five services, built to an exact contract with five error codes. Turn 2 writes the week up as a Document, and the check walks the document's headings and bullets against the calendar's own seeded windows. Turn 3 changes two rules; existing windows must survive, and
verifyAfterAcceptcommits, reloads, and checks both data and rules persisted. Turn 4 asks how many hours one service has that week, answer as a bare number.worker-logs (3 turns). A Workers request-log analyser. The verifier pushes a seeded day of 5,869 events for six Workers across four colos, with one planted worst hour and one decoy, and computes every reference answer itself (nearest-rank p95, half-open ranges, hourly buckets with empty hours). Turn 2 adds colo and route filters, a per-colo breakdown and a slowest-routes ranking. Turn 3 asks which Worker had the worst hour by error rate, in a fixed three-line reply shape; the check parses it and compares with the planted answer. That is the fabricated-answer failure made deterministic.
incident-desk (2 turns). An on-call desk where twenty responders acknowledge the same page at once: exactly one wins, everyone else is told who. Durable Object RPCs interleave at every
await, so a check-then-write implementation fails. Turn 2 adds escalation and metrics computed from the board's own timestamps.chess (3 turns). A complete engine with no library available in the sandbox, then PGN import/export, then draw detection. Verification is differential against chess.js: legal move sets, resulting positions and status must agree on twelve curated positions (castling through check, en passant that would expose the king, promotion by capture, pins, mate, stalemate) and on seeded random games with illegal-move probes; PGN import covers three miniatures, the Opera Game, and two seeded random games; export must replay in the oracle to the same position. Both en passant FEN conventions are accepted by canonicalising through the oracle, and once a game is drawn with pieces still mobile either an empty or a full move list is accepted.
Two small harness changes: the verifier receives the agent's chat replies for the turn (
EvalVerifier.replies), and the verification budget is four minutes rather than two, since these checks make hundreds of RPC calls per turn.chess.jsis a dev dependency as the oracle.First live run (GPT 5.6 Luna, 10 trials each, 25 minutes, $1.21, no infrastructure failures, verification under 3 s per turn):
6.5openedAtreset()while self-testing and destroyed the user's loaded data, against an explicit instruction1. Nh3and after an opponent'sRxc1;O-O-Owith a knight on b1; king stepping into a pawn's attack; en passant not generatedEvery verifier was run against a minimal correct implementation and a broken one before this was opened; that caught a key-order comparison bug in
worker-logsthat would have failed every agent. The probes are not committed.