feat(eval): add batch-evaluation simulate (dataset simulations)

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate     (existing)
├── simulate     ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a runtime, then batch-evaluate the resulting sessions

Options:
  --runtime-id <runtime-id>              runtime id to invoke per scenario
  --qualifier <qualifier>                runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>  JSON payload template; {input} is the scenario input, e.g.
                                         {"prompt":"{input}"}
  --header <header...>                   an ordered application header (repeatable)
  --bearer-token <bearer-token>          CUSTOM_JWT bearer token (for JWT-auth runtimes)
  --user-id <user-id>                    runtime user id
  --dataset <dataset>                    dataset source: local JSONL path or a dataset id
  --dataset-version <dataset-version>    dataset version (with a dataset id)
  --evaluator <evaluator...>             evaluator id(s) to apply
  --name <name>                          batch evaluation name (unique in the account)
  --description <description>            optional description
  --kms-key-arn <kms-key-arn>            KMS key to encrypt evaluation data at rest
  -h, --help                             display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2000      +/-   ##
============================================
- Coverage     96.96%   96.12%   -0.84%     
============================================
  Files           364      370       +6     
  Lines         20758    21393     +635     
============================================
+ Hits          20127    20563     +436     
- Misses          631      830     +199     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.