feat: August 12 release — evaluator OpenResponses, insights KMS, recommendation trace sources - #1981
Merged
Conversation
…mmendation trace sources Squashed release of the bug-bash-august-10 feature set: - feat(evaluator): add OpenResponses (OpenAI) model support for LLM-as-a-judge evaluators, incl. bug-bash fixes (#362, #366). Requires agentcore-l3-cdk-constructs #314 to be merged first. - feat: add optional KMS key for on-demand insights jobs (#363). - feat(run): add --online-evaluation-arn trace source for recommendations, plus reuse-eval CLI fixes for inherited evaluators, conflicting-source rejection, and TUI-vs-headless gating (B1/B2/B3/B6, #365).
github-actions
Bot
added
size/xl
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
Package TarballHow to installgh release download pr-1981-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.26.0.tgz |
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
github-actions
Bot
removed
the
agentcore-harness-reviewing
Contributor
Coverage Report
|
nborges-aws
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Merge blocker:
aws/agentcore-l3-cdk-constructs#314 (feat(evaluator): support OpenResponses models) must be merged first. The OpenResponses evaluator feature below depends on the CDK L3 construct support it adds. As of this writing #314 is still open.Summary
Squashed release of the
bug-bash-august-10feature set. Three new features plus their bug-bash follow-up fixes. Purely additive — cherry-picked onto currentmain, so nothing existing is removed.Features
1. OpenResponses (OpenAI) model support for evaluators — #362, #366
LLM-as-a-judge evaluators can now use OpenAI models via OpenResponses in addition to Bedrock. Adds an optional
⚠️ Depends on CDK constructs PR #314.
modelProvider('Bedrock' | 'OpenAI', defaults toBedrock) toLlmAsAJudgeConfig, with wiring through the evaluator primitive, import path, TUI add-evaluator wizard, telemetry, and docs. Includes bug-bash fixes (#366).2. KMS key option for on-demand insights — #363
On-demand insights jobs accept an optional customer-managed KMS key for encryption, surfaced as a
runflag and threaded through the insights handler and shared job types.3.
--online-evaluation-arntrace source for recommendationsRecommendations can source traces from an online evaluation ARN. Bundled with reuse-eval CLI fixes (#365, B1/B2/B3/B6):
evaluationConfig.--json.Changes
37 files, +1254 / -253 (features + tests + docs).
Testing
npm test/npm run test:integNotes
The generated
src/schema/llm-compacted/agentcore.ts(a read-only LLM-context asset) had a squash conflict againstmain'sratingScalerefactor; resolved by keepingmain's inline-union shape and adding the newmodelProviderfield.