Skip to content

feat: August 12 release — evaluator OpenResponses, insights KMS, recommendation trace sources #1981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jariy17 merged 1 commit into from
Aug 12, 2026
Merged

feat: August 12 release — evaluator OpenResponses, insights KMS, recommendation trace sources #1981

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 22 additions & 13 deletions docs/commands.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -600,6 +600,14 @@ agentcore add evaluator \
--instructions "Evaluate the response quality. Context: {context}" \
--rating-scale 1-5-quality

# OpenAI through Bedrock Mantle/OpenResponses
agentcore add evaluator \
--name OpenAIQuality \
--level SESSION \
--model-provider OpenResponses \
--model openai.gpt-5.4 \
--instructions "Evaluate the response quality. Context: {context}"

# Code-based (existing Lambda)
agentcore add evaluator \
--name LatencyCheck \
Expand All @@ -609,19 +617,20 @@ agentcore add evaluator \
--timeout 60
```

| Flag | Description |
| ------------------------- | ----------------------------------------------------------------------------- |
| `--name <name>` | Evaluator name |
| `--type <type>` | `llm-as-a-judge` (default) or `code-based` |
| `--level <level>` | `SESSION`, `TRACE`, or `TOOL_CALL` |
| `--model <model>` | [LLM] Bedrock model ID for the LLM judge |
| `--instructions <text>` | [LLM] Evaluation prompt with placeholders (e.g. `{context}`) |
| `--rating-scale <preset>` | [LLM] `1-5-quality`, `1-3-simple`, `pass-fail`, `good-neutral-bad`, or custom |
| `--lambda-arn <arn>` | [Code-based] Existing Lambda function ARN |
| `--timeout <seconds>` | [Code-based] Lambda timeout in seconds (1–300) |
| `--kms-key-arn <arn>` | KMS key ARN for evaluator encryption (optional) |
| `--config <path>` | Config JSON file (overrides `--model`, `--instructions`, `--rating-scale`) |
| `--json` | JSON output |
| Flag | Description |
| ----------------------------- | ----------------------------------------------------------------------------- |
| `--name <name>` | Evaluator name |
| `--type <type>` | `llm-as-a-judge` (default) or `code-based` |
| `--level <level>` | `SESSION`, `TRACE`, or `TOOL_CALL` |
| `--model <model>` | [LLM] Bedrock model ID/profile or OpenAI model ID |
| `--model-provider <provider>` | [LLM] `Bedrock` (default) or `OpenResponses` |
| `--instructions <text>` | [LLM] Evaluation prompt with placeholders (e.g. `{context}`) |
| `--rating-scale <preset>` | [LLM] `1-5-quality`, `1-3-simple`, `pass-fail`, `good-neutral-bad`, or custom |
| `--lambda-arn <arn>` | [Code-based] Existing Lambda function ARN |
| `--timeout <seconds>` | [Code-based] Lambda timeout in seconds (1–300) |
| `--kms-key-arn <arn>` | KMS key ARN for evaluator encryption (optional) |
| `--config <path>` | Config JSON file; cannot be combined with `--model-provider` |
| `--json` | JSON output |

### add online-eval

Expand Down
30 changes: 25 additions & 5 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,11 +329,31 @@ See [Evaluations](evals.md) for the full guide.

### LLM-as-a-Judge Config

| Field | Required | Description |
| -------------- | -------- | ------------------------------------------------------ |
| `model` | Yes | Bedrock model ID or cross-region inference profile |
| `instructions` | Yes | Evaluation prompt with placeholders (e.g. `{context}`) |
| `ratingScale` | Yes | Either `numerical` or `categorical` array (not both) |
| Field | Required | Description |
| --------------- | -------- | ------------------------------------------------------ |
| `modelProvider` | No | `"Bedrock"` (default) or `"OpenResponses"` |
| `model` | Yes | Bedrock model ID/profile or OpenAI model ID |
| `instructions` | Yes | Evaluation prompt with placeholders (e.g. `{context}`) |
| `ratingScale` | Yes | Either `numerical` or `categorical` array (not both) |

OpenAI evaluators run through the OpenResponses API on Bedrock Mantle with maximum output tokens set to `4096`,
temperature set to `0`, and top P left unset.

```json
{
"llmAsAJudge": {
"modelProvider": "OpenResponses",
"model": "openai.gpt-5.4",
"instructions": "Evaluate the response quality. Context: {context}",
"ratingScale": {
"categorical": [
{ "label": "Pass", "definition": "Meets expectations" },
{ "label": "Fail", "definition": "Does not meet expectations" }
]
}
}
}
```

### Rating Scale

Expand Down
62 changes: 39 additions & 23 deletions docs/evals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ agentcore add evaluator \
--instructions "Evaluate the agent response quality. Context: {context}" \
--rating-scale 1-5-quality

# Non-interactive — OpenAI through Bedrock Mantle/OpenResponses
agentcore add evaluator \
--name OpenAIQuality \
--level SESSION \
--model-provider OpenResponses \
--model openai.gpt-5.4 \
--instructions "Evaluate the agent response quality. Context: {context}" \
--rating-scale 1-5-quality

# Non-interactive — Code-based (existing Lambda)
agentcore add evaluator \
--name LatencyCheck \
Expand All @@ -50,22 +59,24 @@ agentcore add evaluator \
--timeout 60
```

| Flag | Description |
| ------------------------- | --------------------------------------------------------------------------------------- |
| `--name <name>` | Evaluator name (alphanumeric + underscore, max 48 chars) |
| `--type <type>` | Evaluator type: `llm-as-a-judge` (default) or `code-based` |
| `--level <level>` | Evaluation level: `SESSION`, `TRACE`, `TOOL_CALL` |
| `--model <model>` | [LLM] Bedrock model ID for the LLM judge |
| `--instructions <text>` | [LLM] Evaluation prompt (must include level-appropriate placeholders — see below) |
| `--rating-scale <preset>` | [LLM] Rating scale preset or custom format (default: `1-5-quality`) |
| `--lambda-arn <arn>` | [Code-based] Existing Lambda function ARN |
| `--timeout <seconds>` | [Code-based] Lambda timeout in seconds, 1–300 (default: 60) |
| `--kms-key-arn <arn>` | KMS key ARN for evaluator encryption (optional) |
| `--config <path>` | Path to evaluator config JSON (overrides `--model`, `--instructions`, `--rating-scale`) |
| `--json` | JSON output |
| Flag | Description |
| ----------------------------- | --------------------------------------------------------------------------------- |
| `--name <name>` | Evaluator name (alphanumeric + underscore, max 48 chars) |
| `--type <type>` | Evaluator type: `llm-as-a-judge` (default) or `code-based` |
| `--level <level>` | Evaluation level: `SESSION`, `TRACE`, `TOOL_CALL` |
| `--model <model>` | [LLM] Bedrock model ID/profile or OpenAI model ID |
| `--model-provider <provider>` | [LLM] `Bedrock` (default) or `OpenResponses` |
| `--instructions <text>` | [LLM] Evaluation prompt (must include level-appropriate placeholders — see below) |
| `--rating-scale <preset>` | [LLM] Rating scale preset or custom format (default: `1-5-quality`) |
| `--lambda-arn <arn>` | [Code-based] Existing Lambda function ARN |
| `--timeout <seconds>` | [Code-based] Lambda timeout in seconds, 1–300 (default: 60) |
| `--kms-key-arn <arn>` | KMS key ARN for evaluator encryption (optional) |
| `--config <path>` | Path to evaluator config JSON; cannot be combined with `--model-provider` |
| `--json` | JSON output |

> **Note**: For LLM-as-a-Judge, `--instructions` is required in non-interactive mode unless `--config` is provided. For
> code-based evaluators, `--lambda-arn` is required.
> code-based evaluators, `--lambda-arn` is required. OpenAI evaluators use maximum output tokens `4096`, temperature
> `0`, and leave top P unset.

### Instruction Placeholders

Expand Down Expand Up @@ -143,15 +154,20 @@ Evaluators are stored in the `evaluators` array of `agentcore.json`:

### Model Selection

Model availability varies by AWS region. Recommended models:

| Model | Description |
| ----------------- | ------------------------------------------- |
| Claude Sonnet 4.5 | Recommended — balanced speed and accuracy |
| Claude Opus 4.5 | Most capable — best for complex evaluations |
| Claude Haiku 4.5 | Fastest — good for high-volume evaluations |
| Amazon Nova Pro | Strong reasoning |
| Amazon Nova Lite | Fast and cost-effective |
Model availability varies by AWS region. Bedrock models use the Converse API; OpenAI models use the OpenResponses API
through Bedrock Mantle.

| Provider | Model | Description |
| -------- | ----------------- | ------------------------------------------- |
| Bedrock | Claude Sonnet 4.5 | Recommended — balanced speed and accuracy |
| Bedrock | Claude Opus 4.5 | Most capable — best for complex evaluations |
| Bedrock | Claude Haiku 4.5 | Fastest — good for high-volume evaluations |
| Bedrock | Amazon Nova Pro | Strong reasoning |
| Bedrock | Amazon Nova Lite | Fast and cost-effective |
| OpenAI | GPT-5.4 | High-quality reasoning and evaluation |
| OpenAI | GPT-5.5 | Most capable OpenAI model |
| OpenAI | GPT OSS 120B | Open-weight model for complex evaluations |
| OpenAI | GPT OSS 20B | Open-weight model for faster evaluations |

---

Expand Down
14 changes: 14 additions & 0 deletions docs/recommendations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,22 @@ agentcore run recommendation ... --session-id <id-1> <id-2>

# From a local spans file (OTEL format)
agentcore run recommendation ... --spans-file ./traces.json

# Reuse an existing evaluation run's scores
agentcore run recommendation ... --batch-evaluation-arn <arn>
agentcore run recommendation ... --online-evaluation-arn <arn>
```

When the trace source is a batch or online evaluation (`--from-insights`, `--batch-evaluation-arn`, or
`--online-evaluation-arn`), the recommendation inherits that evaluation's evaluator, so `--evaluator` is not required.
If you do pass an `--evaluator` that differs from the referenced evaluation's, the two source types behave differently:

- **Batch evaluation:** falls back to running a fresh evaluation with the evaluator you specified.
- **Online evaluation:** fails — an online config's evaluator cannot be overridden.

For online and CloudWatch sources, `--lookback` bounds the window of scores/traces considered
(`startTime = now − lookback`, `endTime = now`).

## Encrypting Results with KMS

By default, recommendation results are encrypted with an AWS-managed key. To encrypt them with your own customer managed
Expand Down
112 changes: 112 additions & 0 deletions integ-tests/add-remove-evaluator.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import { createTestProject, parseJsonOutput, readProjectConfig, runCLI } from '../src/test-utils/index.js';
import type { TestProject } from '../src/test-utils/index.js';
import { createTelemetryHelper } from '../src/test-utils/telemetry-helper.js';
import { writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';

const telemetry = createTelemetryHelper();
Expand Down Expand Up @@ -67,6 +69,7 @@ describe('integration: add and remove evaluators and online eval configs', () =>
const found = config.evaluators.find(e => e.name === evalName);
expect(found).toBeDefined();
expect(found!.level).toBe('SESSION');
expect(found!.config.llmAsAJudge?.modelProvider).toBeUndefined();
});

it('rejects duplicate evaluator name', async () => {
Expand Down Expand Up @@ -140,6 +143,39 @@ describe('integration: add and remove evaluators and online eval configs', () =>
});

describe('error cases', () => {
it('adds an OpenResponses evaluator', async () => {
const name = `OpenResponsesEval${Date.now().toString().slice(-6)}`;
const json = await runSuccess(
[
'add',
'evaluator',
'--name',
name,
'--level',
'SESSION',
'--model-provider',
'OpenResponses',
'--model',
'openai.gpt-5.4',
'--instructions',
'Evaluate the session quality. Context: {context}',
'--json',
],
project.projectPath
);
expect(json.evaluatorName).toBe(name);

const config = await readProjectConfig(project.projectPath);
expect(config.evaluators.find(e => e.name === name)?.config.llmAsAJudge).toEqual(
expect.objectContaining({
modelProvider: 'OpenResponses',
model: 'openai.gpt-5.4',
})
);

await runSuccess(['remove', 'evaluator', '--name', name, '--json'], project.projectPath);
});

it('fails to remove non-existent evaluator', async () => {
const json = await runFailure(['remove', 'evaluator', '--name', 'NonExistent', '--json'], project.projectPath);
expect(json.error).toContain('not found');
Expand All @@ -165,6 +201,82 @@ describe('integration: add and remove evaluators and online eval configs', () =>
expect(json.error).toContain('--config');
});

it('rejects the obsolete OpenAI provider discriminator', async () => {
const json = await runFailure(
[
'add',
'evaluator',
'--name',
'ObsoleteProvider',
'--level',
'SESSION',
'--model-provider',
'OpenAI',
'--model',
'openai.gpt-5.4',
'--instructions',
'Evaluate {context}',
'--json',
],
project.projectPath
);
expect(json.error).toContain('Bedrock, OpenResponses');
});

it('rejects invalid model IDs before writing config', async () => {
const json = await runFailure(
[
'add',
'evaluator',
'--name',
'InvalidModel',
'--level',
'SESSION',
'--model-provider',
'OpenResponses',
'--model',
' ',
'--instructions',
'Evaluate {context}',
'--json',
],
project.projectPath
);
expect(json.error).toContain('Model ID is required');
});

it('rejects --model-provider when --config is used', async () => {
const configPath = join(project.projectPath, 'evaluator-config.json');
await writeFile(
configPath,
JSON.stringify({
llmAsAJudge: {
model: 'openai.gpt-5.4',
instructions: 'Evaluate {context}',
ratingScale: { categorical: [{ label: 'Pass', definition: 'Meets expectations' }] },
},
})
);

const json = await runFailure(
[
'add',
'evaluator',
'--name',
'ConfigProviderConflict',
'--level',
'SESSION',
'--config',
configPath,
'--model-provider',
'OpenResponses',
'--json',
],
project.projectPath
);
expect(json.error).toContain('--model-provider cannot be used with --config');
});

it('rejects evaluator with instructions missing required placeholders', async () => {
const json = await runFailure(
[
Expand Down
30 changes: 30 additions & 0 deletions integ-tests/run-insights.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,36 @@ describe('integration: run insights command validation', () => {
expect(json.error).not.toContain('--session-ids');
});

it('accepts a valid --kms-key ARN', async () => {
const result = await runCLI(
[
'run',
'insights',
'--runtime',
project.agentName,
'--kms-key',
'arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012',
'--json',
],
project.projectPath,
{ env: telemetry.env }
);
expect(result.exitCode).toBe(1);
const json = parseJsonOutput(result.stdout) as Record<string, unknown>;
expect(json.error).not.toContain('--kms-key');
});

it('rejects an invalid --kms-key ARN', async () => {
const result = await runCLI(
['run', 'insights', '--runtime', project.agentName, '--kms-key', 'not-a-kms-arn', '--json'],
project.projectPath,
{ env: telemetry.env }
);
expect(result.exitCode).toBe(1);
const json = parseJsonOutput(result.stdout) as Record<string, unknown>;
expect(json.error).toContain('--kms-key must be a valid KMS key ARN');
});

it('accepts --online-eval-config-arn as data source (no --runtime needed)', async () => {
const result = await runCLI(
[
Expand Down
Loading
Loading