|
| 1 | +# TypeScript HR Assistant — AgentCore Runtime + Evaluations |
| 2 | + |
| 3 | +A human-resources assistant built with [LangGraph](https://langchain-ai.github.io/langgraphjs/) |
| 4 | +(TypeScript) and deployed on Amazon Bedrock AgentCore Runtime. The sample shows that |
| 5 | +AgentCore Evaluations works the same way for TypeScript agents as for Python agents: |
| 6 | +the evaluators score OpenTelemetry spans in CloudWatch, so the agent can be written in |
| 7 | +any language while the evaluation workflow uses the standard Python SDK. |
| 8 | + |
| 9 | +It covers the full evaluation lifecycle: |
| 10 | + |
| 11 | +| Stage | API | What it answers | |
| 12 | +|:------|:----|:----------------| |
| 13 | +| 1. On-demand | `EvaluationClient` | "How did this one session go?" — spot-check while developing | |
| 14 | +| 2. Dataset | `OnDemandEvaluationDatasetRunner` | "Does the agent still pass my ground-truth test set?" — regression testing / CI | |
| 15 | +| 3. Batch | `StartBatchEvaluation` | "How are all recorded sessions scoring?" — service-side scoring of the log group | |
| 16 | +| 4. Online | `CreateOnlineEvaluationConfig` | "How is production traffic scoring right now?" — continuous monitoring | |
| 17 | + |
| 18 | +Highlights: |
| 19 | + |
| 20 | +- **LangGraph TypeScript agent on AgentCore Runtime** — `createReactAgent` backed by |
| 21 | + Amazon Bedrock Nova Lite via `ChatBedrockConverse` (`@langchain/aws`) |
| 22 | +- **Custom CloudWatch Logs span exporter** — the ADOT sidecar is not injected for |
| 23 | + custom Docker container deployments, so the agent ships a `CloudWatchLogsSpanExporter` |
| 24 | + that writes flat-format OTel span documents directly to the runtime log group |
| 25 | +- **OpenInference instrumentation** — `@arizeai/openinference-instrumentation-langchain` |
| 26 | + auto-instruments LangGraph tool calls and LLM invocations |
| 27 | +- **Built-in and custom evaluators** — `Builtin.GoalSuccessRate`, `Builtin.Correctness`, |
| 28 | + `Builtin.Helpfulness`, plus two custom LLM-as-a-judge evaluators: |
| 29 | + - **HRPolicyAccuracy** — checks each answer against its expected HR facts (needs ground truth) |
| 30 | + - **HRResponseQuality** — rates helpfulness and specificity per turn (reference-free) |
| 31 | + |
| 32 | +--- |
| 33 | + |
| 34 | +## Architecture |
| 35 | + |
| 36 | +``` |
| 37 | +User request |
| 38 | + │ |
| 39 | + ▼ |
| 40 | +AgentCore Runtime ──────────────────────────────────────────────────┐ |
| 41 | +│ hr-assistant/src/agent.ts │ |
| 42 | +│ ├── LangGraph ReAct agent (createReactAgent) │ |
| 43 | +│ │ └── ChatBedrockConverse (Nova Lite) │ |
| 44 | +│ │ ├── get_pto_balance │ |
| 45 | +│ │ ├── submit_pto_request │ |
| 46 | +│ │ ├── lookup_hr_policy │ |
| 47 | +│ │ ├── get_benefits_summary │ |
| 48 | +│ │ └── get_pay_stub │ |
| 49 | +│ └── CloudWatchLogsSpanExporter │ |
| 50 | +│ └── @arizeai/openinference-instrumentation-langchain │ |
| 51 | +└─────────────────────────────────────────────────────────────────────┘ |
| 52 | + │ │ |
| 53 | + ▼ ▼ |
| 54 | + Response CloudWatch Logs |
| 55 | + (flat OTel span docs) |
| 56 | + │ |
| 57 | + ▼ |
| 58 | + AgentCore Evaluations |
| 59 | + ├── 1. On-demand (EvaluationClient) |
| 60 | + ├── 2. Dataset (OnDemandEvaluationDatasetRunner) |
| 61 | + ├── 3. Batch (StartBatchEvaluation) |
| 62 | + └── 4. Online (online evaluation config) |
| 63 | +``` |
| 64 | + |
| 65 | +> **Note — Docker containers and ADOT:** The ADOT collector sidecar is only |
| 66 | +> auto-injected for `agentcore-cli` / code-zip deployments. Custom Docker |
| 67 | +> containers must export spans themselves. This sample implements |
| 68 | +> `CloudWatchLogsSpanExporter` in `agent.ts`, which writes span documents in the |
| 69 | +> compact flat format expected by `CloudWatchAgentSpanCollector`. |
| 70 | +> |
| 71 | +> Each span document also carries an ADOT-style `resource.attributes` block |
| 72 | +> (`service.name = <runtime-name>.DEFAULT`, `cloud.resource_id`, and so on). |
| 73 | +> Batch and online evaluation use `service.name` to discover a runtime's |
| 74 | +> sessions — without it they find zero sessions. The log group name, service |
| 75 | +> name, and runtime ARN are only known after the runtime is created, so |
| 76 | +> `deploy.py` injects `OTEL_LOG_GROUP_NAME`, `OTEL_SERVICE_NAME`, and |
| 77 | +> `AGENT_RUNTIME_ARN` with `update_agent_runtime`. |
| 78 | +
|
| 79 | +--- |
| 80 | + |
| 81 | +## Files |
| 82 | + |
| 83 | +| File / Directory | Description | |
| 84 | +|-----------------|-------------| |
| 85 | +| `hr-assistant/src/agent.ts` | TypeScript LangGraph agent (deployed to runtime) | |
| 86 | +| `hr-assistant/Dockerfile` | Multi-stage Docker build for `linux/arm64` | |
| 87 | +| `hr-assistant/package.json` | Node.js dependencies | |
| 88 | +| `hr-assistant/tsconfig.json` | TypeScript compiler settings | |
| 89 | +| `deploy.py` | Creates IAM role, ECR repo, builds/pushes image, creates runtime, injects span-export env vars; saves `agent_config.json` | |
| 90 | +| `evaluate.py` | Creates custom evaluators and runs the four evaluation stages | |
| 91 | +| `cleanup.py` | Deletes all AWS resources created by this sample | |
| 92 | +| `requirements.txt` | Local Python dependencies for the scripts | |
| 93 | + |
| 94 | +--- |
| 95 | + |
| 96 | +## Quick start |
| 97 | + |
| 98 | +### Prerequisites |
| 99 | + |
| 100 | +- Docker (with `buildx` and `linux/arm64` support; Colima or Docker Desktop) |
| 101 | +- Python 3.10+ |
| 102 | +- AWS credentials with permissions for ECR, IAM, Bedrock, CloudWatch Logs, and AgentCore |
| 103 | +- Access to `us.amazon.nova-lite-v1:0` (agent) and `us.amazon.nova-pro-v1:0` (custom judges) |
| 104 | + |
| 105 | +### 1. Install local Python dependencies |
| 106 | + |
| 107 | +```bash |
| 108 | +pip install -r requirements.txt |
| 109 | +``` |
| 110 | + |
| 111 | +### 2. Deploy |
| 112 | + |
| 113 | +```bash |
| 114 | +python deploy.py [--region us-east-1] |
| 115 | +``` |
| 116 | + |
| 117 | +This: |
| 118 | +1. Creates an IAM execution role for the runtime |
| 119 | +2. Creates an ECR repository and pushes the Docker image (`linux/arm64`) |
| 120 | +3. Creates the AgentCore Runtime (container deployment) and waits for `READY` |
| 121 | +4. Calls `update_agent_runtime` to inject `OTEL_LOG_GROUP_NAME`, `OTEL_SERVICE_NAME`, and |
| 122 | + `AGENT_RUNTIME_ARN` (enables span export with the resource attributes evaluations need) |
| 123 | +5. Waits for `READY` again, then writes `agent_config.json` |
| 124 | + |
| 125 | +### 3. Evaluate |
| 126 | + |
| 127 | +```bash |
| 128 | +python evaluate.py [--region us-east-1] [--modes on-demand dataset batch online] |
| 129 | +``` |
| 130 | + |
| 131 | +`evaluate.py` first creates the `HRPolicyAccuracy` and `HRResponseQuality` evaluators, |
| 132 | +then runs the selected stages in lifecycle order (all four by default): |
| 133 | + |
| 134 | +| Stage | What happens | Evaluators | Output | |
| 135 | +|:------|:-------------|:-----------|:-------| |
| 136 | +| `on-demand` | Invokes a 5-turn session (PTO, remote work, health, 401k, PTO request), waits 150s for span ingestion, then scores that session | GoalSuccessRate (with assertions), Correctness, Helpfulness, HRResponseQuality | `results/on_demand_results.json` | |
| 137 | +| `dataset` | Runs a 4-scenario ground-truth dataset (one multi-turn); every turn has its own `expected_response` and each scenario has assertions | GoalSuccessRate, Correctness, HRPolicyAccuracy | `results/dataset_runner_results.json` | |
| 138 | +| `batch` | Invokes a fresh 3-turn session, then starts a service-side batch job over the runtime log group, scoped to the session IDs from this run (`filterConfig.sessionIds`), and polls until it finishes | GoalSuccessRate, Helpfulness, HRResponseQuality (all reference-free) | `results/batch_eval_results.json` | |
| 139 | +| `online` | Creates an IAM role and an online evaluation config that samples 100% of new sessions | GoalSuccessRate, Helpfulness, HRResponseQuality | `results/online_eval_config.json` | |
| 140 | + |
| 141 | +To re-score a session that already exists without invoking the agent again: |
| 142 | + |
| 143 | +```bash |
| 144 | +python evaluate.py --modes on-demand --session-id <SESSION_ID> |
| 145 | +``` |
| 146 | + |
| 147 | +> **Choosing where to put ground truth:** a plain-string `expected_response` in |
| 148 | +> `ReferenceInputs` applies to the *last* trace of a session only. Evaluators that |
| 149 | +> need an expected answer for every turn (such as `HRPolicyAccuracy`) belong in the |
| 150 | +> dataset stage, where each `Turn` carries its own `expected_response`. |
| 151 | +> `expected_trajectory` is not used: trajectory reference inputs are rejected for |
| 152 | +> LangGraph/OpenInference spans. |
| 153 | +
|
| 154 | +### 4. Cleanup |
| 155 | + |
| 156 | +```bash |
| 157 | +python cleanup.py [--region us-east-1] |
| 158 | +``` |
| 159 | + |
| 160 | +Deletes the online evaluation config, custom evaluators, runtime, ECR repository, and |
| 161 | +IAM roles recorded in `agent_config.json` and `results/cleanup_state.json`. |
| 162 | + |
| 163 | +--- |
| 164 | + |
| 165 | +## Evaluation results |
| 166 | + |
| 167 | +Scores from a run against the deployed agent (LLM judges vary slightly between runs): |
| 168 | + |
| 169 | +| Stage | Evaluator | Level | Score | |
| 170 | +|-------|-----------|-------|-------| |
| 171 | +| On-demand | Builtin.GoalSuccessRate | SESSION | 1.0 | |
| 172 | +| On-demand | Builtin.Correctness | TRACE | 0.5–1.0 per turn (4/5 turns at 1.0) | |
| 173 | +| On-demand | Builtin.Helpfulness | TRACE | 0.83 (5/5 turns) | |
| 174 | +| On-demand | HRResponseQuality (custom) | TRACE | 1.0 (5/5 turns) | |
| 175 | +| Dataset | Builtin.GoalSuccessRate | SESSION | 1.0 (4/4 scenarios) | |
| 176 | +| Dataset | Builtin.Correctness | TRACE | 1.0 (5/5 turns) | |
| 177 | +| Dataset | HRPolicyAccuracy (custom) | TRACE | 0.90 average (5 turns) | |
| 178 | +| Batch | Builtin.GoalSuccessRate | SESSION | 1.0 (1/1 session) | |
| 179 | +| Batch | Builtin.Helpfulness | TRACE | 0.83 | |
| 180 | +| Batch | HRResponseQuality (custom) | TRACE | 1.0 | |
| 181 | + |
| 182 | +Batch results are written to the |
| 183 | +`/aws/bedrock-agentcore/evaluations/batch-evaluations/results/default` log group, and |
| 184 | +online results to `/aws/bedrock-agentcore/evaluations/results/<config-id>`. |
| 185 | + |
| 186 | +--- |
| 187 | + |
| 188 | +## AgentCore CLI usage |
| 189 | + |
| 190 | +```bash |
| 191 | +# Invoke a deployed runtime |
| 192 | +agentcore invoke \ |
| 193 | + --agent-runtime-id <AGENT_ID> \ |
| 194 | + --payload '{"prompt": "What is the PTO balance for employee EMP-001?"}' \ |
| 195 | + --region us-east-1 |
| 196 | +``` |
0 commit comments