Skip to content

refactor(eval): invokeDataset + per-type example classes - #2

Merged
jariy17 merged 4 commits into
feat/eval-batch-simulatefrom
feat/eval-invoke-dataset
Aug 18, 2026
Merged

refactor(eval): invokeDataset + per-type example classes#2
jariy17 merged 4 commits into
feat/eval-batch-simulatefrom
feat/eval-invoke-dataset

Conversation

@jariy17

@jariy17 jariy17 commented Aug 18, 2026

Copy link
Copy Markdown
Owner

What & why

Refactors the monolithic core.eval.simulate into two composable EvalClient calls the handler orchestrates, and models each dataset type as a class that owns its own parse + run + ground truth.

Stacked on aws#2000 (feat/eval-batch-simulate) — base that branch, so this diff is only the refactor.

  • core.eval.invokeDataset replays a dataset (invoke per example, one session each) and returns { sessions, invoked, failed } with neutral inline ground truth.
  • The handler composes invokeDatasetstartBatchEvaluation. No Core method calls a sibling.
  • Per-type example classes (PredefinedExample / SimulatedExample) implement an Example interface; DatasetLoader classifies each row by shape and news the right class. Dispatch is a switch, not a map.

CLI command structure (unchanged for users)

agentcore eval batch-evaluation simulate \
  --runtime-id <id> --payload-template '{"prompt":"{input}"}' \
  --dataset <path.jsonl | dataset-id> [--dataset-version <v>] \
  --evaluator <id...> --name <name> \
  [--qualifier PROD] [--header 'x-a: 1'] [--bearer-token -] [--user-id u] \
  [--description ...] [--kms-key-arn ...]

Output: { batchEvaluationId, status, examplesInvoked, examplesFailed }.

What changed

Area Change
src/core/eval/dataset/ newtypes.ts (Example, RunContext), predefined.ts, simulated.ts, load.ts (DatasetLoader), run.ts (runExamples)
src/core/eval/simulate.ts deleted — split into the above; ScenarioExample, runScenariosrunExamples
src/core/eval.tsx simulate method → invokeDataset + private readDatasetText
handlers/eval/types.tsx SimulateInput/ResultInvokeDatasetInput/Result on CoreEvalClient
handlers/eval/.../simulate/index.tsx composes invokeDataset + startBatchEvaluation
testing/TestCoreClient.tsx setSimulateResponse/simulatesetInvokeDatasetResponse/invokeDataset

Design decisions

  1. Dispatch is a build() switch, not a registry map. A new DatasetSchemaType member makes the switch non-exhaustive → build lacks an ending return → compile error (TS2366). No assertNever, no satisfies Record.
  2. readDatasetText reuses the existing read pathreadLocalDatasetFile (already used by updateDatasetExamples) for local files, downloadDatasetToTemp for dataset ids — and drops the combined loadDatasetFile.
  3. Ground truth stays neutralinvokeDataset returns InlineGroundTruth; the handler wraps it into SessionMetadataShape for batch. A future ondemand simulate handler adapts the same output to EvaluationReferenceInput.
  4. Simulated is scoped outSimulatedExample's constructor throws NotImplementedError at load, so the user gets a clear "not replayable yet" instead of a per-row "has no turns" misdiagnosis.

Deliberately not in this PR (follow-up): folding invokeRuntime/normalizeRuntimeInvokeRequest into a RuntimeInvoker static class. Kept the existing free functions to keep the blast radius on the dataset seam.

Bug fixes carried from the simulate work

  • Sparse multi-turn ground truth kept in position — one GroundTruthTurn per turn carrying input.prompt, so a 3-turn example whose only expectation is on turn 3 is graded against turn 3, not turn 1. (The old .filter(...) renumbered turns.)
  • Both-turns-and-actor_profile row is refused (AWS's SDK silently drops the actor profile).
  • Empty ground truth omits the union member (no { inline: {} }).

Testing

$ bun run typecheck        # clean
$ bunx oxlint src/core/eval/dataset   # clean
$ SIMULATE_INGESTION_WAIT_MS=0 bun test
  1401 pass, 0 fail, 2 snapshots

New tests:

  • src/core/eval/dataset/dataset.test.ts — loader classification (predefined/simulated/both/neither/dup-id/missing-id/bad-JSON/empty/blank-lines), constructor validation, turn-replay order, sparse-turn position, no-GT, and a golden snapshot of the full inline ground-truth shape ([golden]).
  • simulate.test.tsx — asserts the invokeDatasetstartBatchEvaluation composition: runtime-level flags reach invokeDataset (no evaluator/name leak), sessions + wrapped ground truth reach startBatchEvaluation, and "nothing invoked" refuses to grade.

Mock-free: example classes are tested by constructing them and passing a fake invokeOnce.

jariy17 added 4 commits August 18, 2026 20:20
…ype example classes

Split the monolithic core.eval.simulate into two composable EvalClient calls the
handler orchestrates — invokeDataset (replay) then startBatchEvaluation (grade) —
and model each dataset type as a class that owns its parse + run + ground truth.

- src/core/eval/dataset/: types (Example interface, RunContext), predefined +
  simulated example classes, DatasetLoader (shape-classify → switch → new), and a
  generic runExamples pool. Replaces src/core/eval/simulate.ts.
- core.eval.invokeDataset returns { sessions, invoked, failed } with neutral inline
  ground truth; the batch-evaluation simulate handler wraps it as sessionMetadata and
  submits startBatchEvaluation. No Core method calls a sibling.
- readDatasetText reuses readLocalDatasetFile + downloadDatasetToTemp (drops the
  combined loadDatasetFile); dispatch is a build() switch whose non-exhaustiveness on a
  new DatasetSchemaType fails the build (no map, no assertNever).
- Carries the simulate fixes: sparse multi-turn ground truth keeps turn position
  (input.prompt per turn), both-row refusal, NotImplementedError for simulated,
  per-example failure isolation, 180s ingestion wait, AbortSignal, JSON-only.
- Tests: dataset unit tests + a golden snapshot of the inline ground-truth shape;
  handler test asserts the invokeDataset→startBatchEvaluation composition.
… crashing

A JSONL line that is valid JSON but not an object (e.g. bare `null`) threw a
raw TypeError (`null is not an object`) when DatasetLoader dereferenced
`row.example_id`; the same happened in PredefinedExample for a non-object turn
entry (`turns: [null]`). Both now surface a clear InputValidationError at the
parse boundary. Adds guard tests plus behavior locks for CRLF endings, unicode
ids, empty assertions/trajectory omission, and `expected_response: ""`.
…data golden

runExamples: lock per-example failure isolation (one worker throws, the rest run
and are returned; firstError captured and non-Error throws wrapped), exactly-once
processing, the concurrency bound, and the empty-input case. Adds a golden
snapshot of the sessionMetadata the simulate handler builds, pinning the
`{ inline: gt }` wrapping and the omitted-member case for a session with no GT.
@jariy17
jariy17 marked this pull request as ready for review August 18, 2026 20:53
@jariy17
jariy17 merged commit 7d1f9b1 into feat/eval-batch-simulate Aug 18, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant