Skip to content

Commit d54599e

Browse files
committed
feat(evaluate): add TypeScript HR assistant agent evaluation sample
Demonstrates the full AgentCore Evaluations lifecycle for a LangGraph TypeScript agent deployed as a custom container on AgentCore Runtime: on-demand (EvaluationClient), dataset (OnDemandEvaluationDatasetRunner), batch (StartBatchEvaluation) and online (CreateOnlineEvaluationConfig). - agent.ts: stamp ADOT-style resource attributes (service.name, cloud.resource_id, ...) on exported span docs so batch and online evaluation can discover sessions; strip Nova <thinking> tags; remove unused helpers and dependencies - deploy.py: inject OTEL_LOG_GROUP_NAME, OTEL_SERVICE_NAME and AGENT_RUNTIME_ARN after runtime creation; refactor into functions - evaluate.py: restructure into per-stage functions with --modes and --session-id; add 4-scenario ground-truth dataset; scope batch jobs to the run's session IDs; move HRPolicyAccuracy to the dataset stage (plain-string expected_response only covers the last trace) - cleanup.py: track and delete batch jobs, online configs, evaluators and IAM roles from results/cleanup_state.json - READMEs: document the lifecycle, per-stage evaluators and results - Pass ruff check/format and pylint (10.00/10)
1 parent 2747451 commit d54599e

14 files changed

Lines changed: 4306 additions & 2 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,7 @@ plan/
288288
!01-features/02-host-your-agent/01-runtime/04-coding-agents/07-claude-code-with-gateway-inference-targets/Dockerfile
289289
!01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/agent/Dockerfile
290290
!01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/mcp-server/Dockerfile
291+
!01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/typescript-agents/hr-assistant/Dockerfile
291292

292293
# Agent state and runtime files
293294
**/.agent_arn*

‎01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/README.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,9 @@ Three evaluation interfaces are available depending on your use case:
124124
| [`custom-code-based-evaluation/`](custom-code-based-evaluation/) | Lambda-backed deterministic evaluators (code-based) for exact data validation, mixed with built-in LLM evaluators; on-demand and online modes. Also includes Jev and Strands Decider decision-model evaluators (opt-in via `--with-jev` / `--with-decider`) |
125125
| [`skills-evaluation/`](skills-evaluation/) | Native Strands Agent Skills evaluated with `Builtin.SkillSelectionAccuracy` and `Builtin.SkillInstructionFollowing` |
126126
| [`supported-frameworks/`](supported-frameworks/) | The same HR Assistant re-implemented in other supported frameworks (OpenAI Agents SDK, LlamaIndex), each deployed and evaluated with built-in and custom evaluators |
127+
| [`typescript-agents/`](typescript-agents/) | A TypeScript (LangGraph.js) HR Assistant in a custom container, evaluated across the full lifecycle: on-demand, dataset, batch, and online evaluation |
127128

128-
The `ground-truth-based-evaluation/`, `llm-as-a-judge-evaluation/`, and `custom-code-based-evaluation/` samples share the default HR Assistant agent deployed from `utils/`. The `skills-evaluation/` sample uses the same agent source with an opt-in skills directory and a separate runtime config. The `supported-frameworks/` samples re-implement that agent in each framework and deploy it from their own folders.
129+
The `ground-truth-based-evaluation/`, `llm-as-a-judge-evaluation/`, and `custom-code-based-evaluation/` samples share the default HR Assistant agent deployed from `utils/`. The `skills-evaluation/` sample uses the same agent source with an opt-in skills directory and a separate runtime config. The `supported-frameworks/` samples re-implement that agent in each framework and deploy it from their own folders. The `typescript-agents/` sample ports the agent to TypeScript and deploys it as a custom Docker container with its own `deploy.py`.
129130

130131
## Agent Architecture
131132

@@ -245,4 +246,10 @@ cd skills-evaluation
245246
pip install -r requirements.txt
246247
python ../utils/deploy.py --skills-dir skills --config-output agent_config.json
247248
python evaluate.py
249+
250+
# TypeScript agent: on-demand, dataset, batch, and online evaluation (from 02-evaluate/)
251+
cd typescript-agents
252+
pip install -r requirements.txt
253+
python deploy.py
254+
python evaluate.py
248255
```
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Runtime-generated config with account IDs and ARNs — never commit
2+
agent_config.json
3+
4+
# Evaluation output (may contain ARNs and session IDs)
5+
results/
6+
7+
# Node.js build artifacts
8+
hr-assistant/node_modules/
9+
hr-assistant/dist/
10+
11+
# Python cache
12+
__pycache__/
13+
*.pyc
Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
# TypeScript HR Assistant — AgentCore Runtime + Evaluations
2+
3+
A human-resources assistant built with [LangGraph](https://langchain-ai.github.io/langgraphjs/)
4+
(TypeScript) and deployed on Amazon Bedrock AgentCore Runtime. The sample shows that
5+
AgentCore Evaluations works the same way for TypeScript agents as for Python agents:
6+
the evaluators score OpenTelemetry spans in CloudWatch, so the agent can be written in
7+
any language while the evaluation workflow uses the standard Python SDK.
8+
9+
It covers the full evaluation lifecycle:
10+
11+
| Stage | API | What it answers |
12+
|:------|:----|:----------------|
13+
| 1. On-demand | `EvaluationClient` | "How did this one session go?" — spot-check while developing |
14+
| 2. Dataset | `OnDemandEvaluationDatasetRunner` | "Does the agent still pass my ground-truth test set?" — regression testing / CI |
15+
| 3. Batch | `StartBatchEvaluation` | "How are all recorded sessions scoring?" — service-side scoring of the log group |
16+
| 4. Online | `CreateOnlineEvaluationConfig` | "How is production traffic scoring right now?" — continuous monitoring |
17+
18+
Highlights:
19+
20+
- **LangGraph TypeScript agent on AgentCore Runtime** — `createReactAgent` backed by
21+
Amazon Bedrock Nova Lite via `ChatBedrockConverse` (`@langchain/aws`)
22+
- **Custom CloudWatch Logs span exporter** — the ADOT sidecar is not injected for
23+
custom Docker container deployments, so the agent ships a `CloudWatchLogsSpanExporter`
24+
that writes flat-format OTel span documents directly to the runtime log group
25+
- **OpenInference instrumentation** — `@arizeai/openinference-instrumentation-langchain`
26+
auto-instruments LangGraph tool calls and LLM invocations
27+
- **Built-in and custom evaluators** — `Builtin.GoalSuccessRate`, `Builtin.Correctness`,
28+
`Builtin.Helpfulness`, plus two custom LLM-as-a-judge evaluators:
29+
- **HRPolicyAccuracy** — checks each answer against its expected HR facts (needs ground truth)
30+
- **HRResponseQuality** — rates helpfulness and specificity per turn (reference-free)
31+
32+
---
33+
34+
## Architecture
35+
36+
```
37+
User request
38+
│
39+
▼
40+
AgentCore Runtime ──────────────────────────────────────────────────┐
41+
│ hr-assistant/src/agent.ts │
42+
│ ├── LangGraph ReAct agent (createReactAgent) │
43+
│ │ └── ChatBedrockConverse (Nova Lite) │
44+
│ │ ├── get_pto_balance │
45+
│ │ ├── submit_pto_request │
46+
│ │ ├── lookup_hr_policy │
47+
│ │ ├── get_benefits_summary │
48+
│ │ └── get_pay_stub │
49+
│ └── CloudWatchLogsSpanExporter │
50+
│ └── @arizeai/openinference-instrumentation-langchain │
51+
└─────────────────────────────────────────────────────────────────────┘
52+
│ │
53+
▼ ▼
54+
Response CloudWatch Logs
55+
(flat OTel span docs)
56+
│
57+
▼
58+
AgentCore Evaluations
59+
├── 1. On-demand (EvaluationClient)
60+
├── 2. Dataset (OnDemandEvaluationDatasetRunner)
61+
├── 3. Batch (StartBatchEvaluation)
62+
└── 4. Online (online evaluation config)
63+
```
64+
65+
> **Note — Docker containers and ADOT:** The ADOT collector sidecar is only
66+
> auto-injected for `agentcore-cli` / code-zip deployments. Custom Docker
67+
> containers must export spans themselves. This sample implements
68+
> `CloudWatchLogsSpanExporter` in `agent.ts`, which writes span documents in the
69+
> compact flat format expected by `CloudWatchAgentSpanCollector`.
70+
>
71+
> Each span document also carries an ADOT-style `resource.attributes` block
72+
> (`service.name = <runtime-name>.DEFAULT`, `cloud.resource_id`, and so on).
73+
> Batch and online evaluation use `service.name` to discover a runtime's
74+
> sessions — without it they find zero sessions. The log group name, service
75+
> name, and runtime ARN are only known after the runtime is created, so
76+
> `deploy.py` injects `OTEL_LOG_GROUP_NAME`, `OTEL_SERVICE_NAME`, and
77+
> `AGENT_RUNTIME_ARN` with `update_agent_runtime`.
78+
79+
---
80+
81+
## Files
82+
83+
| File / Directory | Description |
84+
|-----------------|-------------|
85+
| `hr-assistant/src/agent.ts` | TypeScript LangGraph agent (deployed to runtime) |
86+
| `hr-assistant/Dockerfile` | Multi-stage Docker build for `linux/arm64` |
87+
| `hr-assistant/package.json` | Node.js dependencies |
88+
| `hr-assistant/tsconfig.json` | TypeScript compiler settings |
89+
| `deploy.py` | Creates IAM role, ECR repo, builds/pushes image, creates runtime, injects span-export env vars; saves `agent_config.json` |
90+
| `evaluate.py` | Creates custom evaluators and runs the four evaluation stages |
91+
| `cleanup.py` | Deletes all AWS resources created by this sample |
92+
| `requirements.txt` | Local Python dependencies for the scripts |
93+
94+
---
95+
96+
## Quick start
97+
98+
### Prerequisites
99+
100+
- Docker (with `buildx` and `linux/arm64` support; Colima or Docker Desktop)
101+
- Python 3.10+
102+
- AWS credentials with permissions for ECR, IAM, Bedrock, CloudWatch Logs, and AgentCore
103+
- Access to `us.amazon.nova-lite-v1:0` (agent) and `us.amazon.nova-pro-v1:0` (custom judges)
104+
105+
### 1. Install local Python dependencies
106+
107+
```bash
108+
pip install -r requirements.txt
109+
```
110+
111+
### 2. Deploy
112+
113+
```bash
114+
python deploy.py [--region us-east-1]
115+
```
116+
117+
This:
118+
1. Creates an IAM execution role for the runtime
119+
2. Creates an ECR repository and pushes the Docker image (`linux/arm64`)
120+
3. Creates the AgentCore Runtime (container deployment) and waits for `READY`
121+
4. Calls `update_agent_runtime` to inject `OTEL_LOG_GROUP_NAME`, `OTEL_SERVICE_NAME`, and
122+
`AGENT_RUNTIME_ARN` (enables span export with the resource attributes evaluations need)
123+
5. Waits for `READY` again, then writes `agent_config.json`
124+
125+
### 3. Evaluate
126+
127+
```bash
128+
python evaluate.py [--region us-east-1] [--modes on-demand dataset batch online]
129+
```
130+
131+
`evaluate.py` first creates the `HRPolicyAccuracy` and `HRResponseQuality` evaluators,
132+
then runs the selected stages in lifecycle order (all four by default):
133+
134+
| Stage | What happens | Evaluators | Output |
135+
|:------|:-------------|:-----------|:-------|
136+
| `on-demand` | Invokes a 5-turn session (PTO, remote work, health, 401k, PTO request), waits 150s for span ingestion, then scores that session | GoalSuccessRate (with assertions), Correctness, Helpfulness, HRResponseQuality | `results/on_demand_results.json` |
137+
| `dataset` | Runs a 4-scenario ground-truth dataset (one multi-turn); every turn has its own `expected_response` and each scenario has assertions | GoalSuccessRate, Correctness, HRPolicyAccuracy | `results/dataset_runner_results.json` |
138+
| `batch` | Invokes a fresh 3-turn session, then starts a service-side batch job over the runtime log group, scoped to the session IDs from this run (`filterConfig.sessionIds`), and polls until it finishes | GoalSuccessRate, Helpfulness, HRResponseQuality (all reference-free) | `results/batch_eval_results.json` |
139+
| `online` | Creates an IAM role and an online evaluation config that samples 100% of new sessions | GoalSuccessRate, Helpfulness, HRResponseQuality | `results/online_eval_config.json` |
140+
141+
To re-score a session that already exists without invoking the agent again:
142+
143+
```bash
144+
python evaluate.py --modes on-demand --session-id <SESSION_ID>
145+
```
146+
147+
> **Choosing where to put ground truth:** a plain-string `expected_response` in
148+
> `ReferenceInputs` applies to the *last* trace of a session only. Evaluators that
149+
> need an expected answer for every turn (such as `HRPolicyAccuracy`) belong in the
150+
> dataset stage, where each `Turn` carries its own `expected_response`.
151+
> `expected_trajectory` is not used: trajectory reference inputs are rejected for
152+
> LangGraph/OpenInference spans.
153+
154+
### 4. Cleanup
155+
156+
```bash
157+
python cleanup.py [--region us-east-1]
158+
```
159+
160+
Deletes the online evaluation config, custom evaluators, runtime, ECR repository, and
161+
IAM roles recorded in `agent_config.json` and `results/cleanup_state.json`.
162+
163+
---
164+
165+
## Evaluation results
166+
167+
Scores from a run against the deployed agent (LLM judges vary slightly between runs):
168+
169+
| Stage | Evaluator | Level | Score |
170+
|-------|-----------|-------|-------|
171+
| On-demand | Builtin.GoalSuccessRate | SESSION | 1.0 |
172+
| On-demand | Builtin.Correctness | TRACE | 0.5–1.0 per turn (4/5 turns at 1.0) |
173+
| On-demand | Builtin.Helpfulness | TRACE | 0.83 (5/5 turns) |
174+
| On-demand | HRResponseQuality (custom) | TRACE | 1.0 (5/5 turns) |
175+
| Dataset | Builtin.GoalSuccessRate | SESSION | 1.0 (4/4 scenarios) |
176+
| Dataset | Builtin.Correctness | TRACE | 1.0 (5/5 turns) |
177+
| Dataset | HRPolicyAccuracy (custom) | TRACE | 0.90 average (5 turns) |
178+
| Batch | Builtin.GoalSuccessRate | SESSION | 1.0 (1/1 session) |
179+
| Batch | Builtin.Helpfulness | TRACE | 0.83 |
180+
| Batch | HRResponseQuality (custom) | TRACE | 1.0 |
181+
182+
Batch results are written to the
183+
`/aws/bedrock-agentcore/evaluations/batch-evaluations/results/default` log group, and
184+
online results to `/aws/bedrock-agentcore/evaluations/results/<config-id>`.
185+
186+
---
187+
188+
## AgentCore CLI usage
189+
190+
```bash
191+
# Invoke a deployed runtime
192+
agentcore invoke \
193+
--agent-runtime-id <AGENT_ID> \
194+
--payload '{"prompt": "What is the PTO balance for employee EMP-001?"}' \
195+
--region us-east-1
196+
```
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
"""Clean up all AWS resources created by deploy.py and evaluate.py.
2+
3+
Reads resource IDs from agent_config.json and results/cleanup_state.json and deletes:
4+
- Batch evaluations (stopped if still running)
5+
- Online evaluation configs
6+
- Custom evaluators
7+
- AgentCore Runtime
8+
- ECR repository + images
9+
- IAM roles (online evaluation role and runtime execution role)
10+
11+
Usage:
12+
python cleanup.py [--region REGION] [--config PATH]
13+
"""
14+
15+
import argparse
16+
import json
17+
import sys
18+
import time
19+
from pathlib import Path
20+
21+
import boto3
22+
from boto3.session import Session
23+
from botocore.exceptions import ClientError
24+
25+
_SCRIPT_DIR = Path(__file__).parent
26+
_DEFAULT_CONFIG = _SCRIPT_DIR / "agent_config.json"
27+
_CLEANUP_STATE_PATH = _SCRIPT_DIR / "results" / "cleanup_state.json"
28+
29+
_TERMINAL_BATCH_STATUSES = ("COMPLETED", "FAILED", "STOPPED")
30+
31+
errors = []
32+
33+
34+
def _try(label: str, fn) -> None:
35+
try:
36+
fn()
37+
print(f" OK {label}")
38+
except ClientError as exc:
39+
print(f" ERR {label}: {exc}")
40+
errors.append(f"{label}: {exc}")
41+
42+
43+
def _delete_role(iam, role_name: str) -> None:
44+
for policy in iam.list_role_policies(RoleName=role_name).get("PolicyNames", []):
45+
iam.delete_role_policy(RoleName=role_name, PolicyName=policy)
46+
iam.delete_role(RoleName=role_name)
47+
48+
49+
def _stop_batch_if_running(agentcore, batch_id: str) -> None:
50+
if agentcore.get_batch_evaluation(batchEvaluationId=batch_id).get("status") not in _TERMINAL_BATCH_STATUSES:
51+
agentcore.stop_batch_evaluation(batchEvaluationId=batch_id)
52+
53+
54+
def _wait_runtime_deleted(ctrl, agent_id: str, timeout: int = 120) -> None:
55+
for _ in range(0, timeout, 10):
56+
try:
57+
ctrl.get_agent_runtime(agentRuntimeId=agent_id)
58+
except ClientError:
59+
return
60+
time.sleep(10)
61+
62+
63+
def main() -> None:
64+
"""Delete every resource recorded by deploy.py and evaluate.py."""
65+
parser = argparse.ArgumentParser(description="Clean up TypeScript HR Assistant resources")
66+
parser.add_argument("--region", default=None)
67+
parser.add_argument("--config", default=str(_DEFAULT_CONFIG))
68+
args = parser.parse_args()
69+
70+
config_path = Path(args.config)
71+
if not config_path.exists():
72+
print("No agent_config.json found — nothing to clean up.")
73+
sys.exit(0)
74+
75+
cfg = json.loads(config_path.read_text(encoding="utf-8"))
76+
state = json.loads(_CLEANUP_STATE_PATH.read_text(encoding="utf-8")) if _CLEANUP_STATE_PATH.exists() else {}
77+
region = args.region or cfg.get("region") or Session().region_name or "us-east-1"
78+
delete_resources(cfg, state, region)
79+
80+
if errors:
81+
print(f"\n{len(errors)} error(s) during cleanup:")
82+
for err in errors:
83+
print(f" - {err}")
84+
else:
85+
print("\nAll resources cleaned up successfully.")
86+
87+
88+
def delete_resources(cfg: dict, state: dict, region: str) -> None:
89+
"""Delete resources in dependency order: evaluations first, IAM roles last."""
90+
agentcore = boto3.client("bedrock-agentcore", region_name=region)
91+
ctrl = boto3.client("bedrock-agentcore-control", region_name=region)
92+
iam = boto3.client("iam")
93+
ecr = boto3.client("ecr", region_name=region)
94+
95+
for batch_id in state.get("batch_evaluation_ids", []):
96+
_try(f"StopBatchEvaluation {batch_id}", lambda b=batch_id: _stop_batch_if_running(agentcore, b))
97+
98+
for config_id in state.get("online_evaluation_config_ids", []):
99+
_try(
100+
f"DeleteOnlineEvaluationConfig {config_id}",
101+
lambda c=config_id: ctrl.delete_online_evaluation_config(onlineEvaluationConfigId=c),
102+
)
103+
104+
for evaluator_id in state.get("custom_evaluator_ids", []):
105+
_try(f"DeleteEvaluator {evaluator_id}", lambda e=evaluator_id: ctrl.delete_evaluator(evaluatorId=e))
106+
107+
agent_id = cfg.get("agent_id", "")
108+
if agent_id:
109+
_try(f"DeleteAgentRuntime {agent_id}", lambda: ctrl.delete_agent_runtime(agentRuntimeId=agent_id))
110+
_wait_runtime_deleted(ctrl, agent_id)
111+
112+
ecr_repo = cfg.get("ecr_repo", "")
113+
if ecr_repo:
114+
_try(
115+
f"DeleteECRRepository {ecr_repo}",
116+
lambda: ecr.delete_repository(repositoryName=ecr_repo, force=True),
117+
)
118+
119+
role_names = list(state.get("evaluation_role_names", []))
120+
if cfg.get("role_arn"):
121+
role_names.append(cfg["role_arn"].split("/")[-1])
122+
for role_name in role_names:
123+
_try(f"DeleteIAMRole {role_name}", lambda n=role_name: _delete_role(iam, n))
124+
125+
126+
if __name__ == "__main__":
127+
main()

0 commit comments

Comments
 (0)