refactor(eval): invokeDataset + per-type example classes - #2
Merged
Conversation
added 4 commits
August 18, 2026 20:20
…ype example classes
Split the monolithic core.eval.simulate into two composable EvalClient calls the
handler orchestrates — invokeDataset (replay) then startBatchEvaluation (grade) —
and model each dataset type as a class that owns its parse + run + ground truth.
- src/core/eval/dataset/: types (Example interface, RunContext), predefined +
simulated example classes, DatasetLoader (shape-classify → switch → new), and a
generic runExamples pool. Replaces src/core/eval/simulate.ts.
- core.eval.invokeDataset returns { sessions, invoked, failed } with neutral inline
ground truth; the batch-evaluation simulate handler wraps it as sessionMetadata and
submits startBatchEvaluation. No Core method calls a sibling.
- readDatasetText reuses readLocalDatasetFile + downloadDatasetToTemp (drops the
combined loadDatasetFile); dispatch is a build() switch whose non-exhaustiveness on a
new DatasetSchemaType fails the build (no map, no assertNever).
- Carries the simulate fixes: sparse multi-turn ground truth keeps turn position
(input.prompt per turn), both-row refusal, NotImplementedError for simulated,
per-example failure isolation, 180s ingestion wait, AbortSignal, JSON-only.
- Tests: dataset unit tests + a golden snapshot of the inline ground-truth shape;
handler test asserts the invokeDataset→startBatchEvaluation composition.
… crashing A JSONL line that is valid JSON but not an object (e.g. bare `null`) threw a raw TypeError (`null is not an object`) when DatasetLoader dereferenced `row.example_id`; the same happened in PredefinedExample for a non-object turn entry (`turns: [null]`). Both now surface a clear InputValidationError at the parse boundary. Adds guard tests plus behavior locks for CRLF endings, unicode ids, empty assertions/trajectory omission, and `expected_response: ""`.
…data golden
runExamples: lock per-example failure isolation (one worker throws, the rest run
and are returned; firstError captured and non-Error throws wrapped), exactly-once
processing, the concurrency bound, and the empty-input case. Adds a golden
snapshot of the sessionMetadata the simulate handler builds, pinning the
`{ inline: gt }` wrapping and the omitted-member case for a session with no GT.
jariy17
marked this pull request as ready for review
August 18, 2026 20:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Refactors the monolithic
core.eval.simulateinto two composableEvalClientcalls the handler orchestrates, and models each dataset type as a class that owns its own parse + run + ground truth.Stacked on aws#2000 (
feat/eval-batch-simulate) — base that branch, so this diff is only the refactor.core.eval.invokeDatasetreplays a dataset (invoke per example, one session each) and returns{ sessions, invoked, failed }with neutral inline ground truth.invokeDataset→startBatchEvaluation. No Core method calls a sibling.PredefinedExample/SimulatedExample) implement anExampleinterface;DatasetLoaderclassifies each row by shape andnews the right class. Dispatch is aswitch, not a map.CLI command structure (unchanged for users)
Output:
{ batchEvaluationId, status, examplesInvoked, examplesFailed }.What changed
src/core/eval/dataset/types.ts(Example,RunContext),predefined.ts,simulated.ts,load.ts(DatasetLoader),run.ts(runExamples)src/core/eval/simulate.tsScenario→Example,runScenarios→runExamplessrc/core/eval.tsxsimulatemethod →invokeDataset+ privatereadDatasetTexthandlers/eval/types.tsxSimulateInput/Result→InvokeDatasetInput/ResultonCoreEvalClienthandlers/eval/.../simulate/index.tsxinvokeDataset+startBatchEvaluationtesting/TestCoreClient.tsxsetSimulateResponse/simulate→setInvokeDatasetResponse/invokeDatasetDesign decisions
build()switch, not a registry map. A newDatasetSchemaTypemember makes the switch non-exhaustive →buildlacks an ending return → compile error (TS2366). NoassertNever, nosatisfies Record.readDatasetTextreuses the existing read path —readLocalDatasetFile(already used byupdateDatasetExamples) for local files,downloadDatasetToTempfor dataset ids — and drops the combinedloadDatasetFile.invokeDatasetreturnsInlineGroundTruth; the handler wraps it intoSessionMetadataShapefor batch. A futureondemand simulatehandler adapts the same output toEvaluationReferenceInput.SimulatedExample's constructor throwsNotImplementedErrorat load, so the user gets a clear "not replayable yet" instead of a per-row "has no turns" misdiagnosis.Deliberately not in this PR (follow-up): folding
invokeRuntime/normalizeRuntimeInvokeRequestinto aRuntimeInvokerstatic class. Kept the existing free functions to keep the blast radius on the dataset seam.Bug fixes carried from the simulate work
GroundTruthTurnper turn carryinginput.prompt, so a 3-turn example whose only expectation is on turn 3 is graded against turn 3, not turn 1. (The old.filter(...)renumbered turns.)turns-and-actor_profilerow is refused (AWS's SDK silently drops the actor profile).{ inline: {} }).Testing
New tests:
src/core/eval/dataset/dataset.test.ts— loader classification (predefined/simulated/both/neither/dup-id/missing-id/bad-JSON/empty/blank-lines), constructor validation, turn-replay order, sparse-turn position, no-GT, and a golden snapshot of the full inline ground-truth shape ([golden]).simulate.test.tsx— asserts theinvokeDataset→startBatchEvaluationcomposition: runtime-level flags reachinvokeDataset(no evaluator/name leak), sessions + wrapped ground truth reachstartBatchEvaluation, and "nothing invoked" refuses to grade.Mock-free: example classes are tested by constructing them and passing a fake
invokeOnce.