feat(eval): batch-evaluation simulate — each example owns its invoker

[jariy17]

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.ts
export type TurnResult = { text: string };
export type RunContext = { invokeOnce(input: string): Promise<TurnResult> };

export interface Example {
  readonly exampleId: string;
  readonly schemaType: DatasetSchemaType;
  // Replay this example against the runtime `ctx` reaches, return neutral ground truth.
  run(ctx: RunContext): Promise<InlineGroundTruth | undefined>;
}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

const results = await runExamples(examples, async (example) => {
  const ctx: RunContext = {
    invokeOnce: async (input) => {
      const res = await invokeRuntime(
        deps,
        { /* resolved runtime, session, and rendered payload */ },
        options,
        signal,
      );
      return { text: /* drained response body */ };
    },
  };

  return example.run(ctx);
});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts                 DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts                  runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts   end-to-end golden coverage of the whole path
└── example/
    ├── types.ts            Example interface, RunContext (the invoker), TurnResult
    ├── predefined.ts       PredefinedExample — replays scripted turns, builds ground truth
    └── simulated.ts        SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

• bun run typecheck passes.
• bun test: 1544 pass, 0 fail.
• invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
• simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.51838% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.10%. Comparing base (ee854eb) to head (6a2915e).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.10%	14 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2032      +/-   ##
============================================
- Coverage     97.13%   97.10%   -0.04%     
============================================
  Files           381      388       +7     
  Lines         22786    23160     +374     
============================================
+ Hits          22134    22490     +356     
- Misses          652      670      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.