feat(tel): populate base attributes for cli.command_run metrics. by Hweinstock · Pull Request #1870 · aws/agentcore-cli

Hweinstock · 2026-07-30T11:49:44Z

Problem

Attributes cannot yet be populated in the handlers since it isn't available in the context yet.
We are missing checks to verify there are no free-form values being emitted.

Solution

Passing the Recorder Down

We pass the attributes recorder into the router directly instead of the root handler. This allows us to add the command path as an attribute BEFORE flags/args are validated. Note that middleware is evaluated after flags/args are validated, so we can't inject it there.

For detecting TUI behavior, we add the attribute at the root TUI entrypoint leveraging the already injected recorder from the context.

Defining Shapes

For best security practice, we want protections against bugs in the CLI leaking PII into telemetry. Therefore, we define a factory function safeSchema that adds a compile time check to our schema that all fields are explicitly enumerated (boolean, enum, literal, etc.).

However, we declare two exceptions to this rule:

error names: since all errors at the top level become instances of AgentCoreCLIErrors, we know that their names must describe a subclass of the base error and therefore be an error we defined. We still provide minimal validation to our input in case the name field is overwritten with user input by mistake.
command_path: we rely on a pattern match to avoid enumerating all paths explicitly. The alternatives are to either determine the command paths at runtime by importing the root handler (creates a circular dependency and introduces complexity), or by explicitly enumerating the paths (susceptible to drift). The simplest approach is to simply check that the path is of the correct form, and limit its length.

Both fields provide fallback values that be used to detect bugs in the implementation.

Testing

added tests at the router level for the command_path flow.

also tested e2e with audit enabled:

rm -rf ~/.agentcore/telemetry

> bun run start config
...
> bun run start harness get --id k
...
> bun run start harness list --max-results 10
... 
> bun run start 
...

Then cat ~/.agentcore/telemetry/* | jq gives us:

{
  "metricName": "cli.command_run",
  "value": 1876,
  "attrs": {
    "service.name": "agentcore-cli",
    "service.version": "0.0.0",
    "agentcore-cli.installation_id": "42edb4ee-1d41-42cf-96ab-195b0356e05a",
    "agentcore-cli.session_id": "4b0f557f-aa08-4543-af41-c036c09749f4",
    "os.type": "Linux",
    "os.version": "6.12.94-123.190.amzn2023.x86_64",
    "host.arch": "x64",
    "node.version": "v24.3.0",
    "exit_reason": "success",
    "command_path": "/agentcore",
    "is_tui": true
  }
}
{
  "metricName": "cli.command_run",
  "value": 102,
  "attrs": {
    "service.name": "agentcore-cli",
    "service.version": "0.0.0",
    "agentcore-cli.installation_id": "42edb4ee-1d41-42cf-96ab-195b0356e05a",
    "agentcore-cli.session_id": "5f471674-be32-488b-922f-ba800d48fcbf",
    "os.type": "Linux",
    "os.version": "6.12.94-123.190.amzn2023.x86_64",
    "host.arch": "x64",
    "node.version": "v24.3.0",
    "exit_reason": "failure",
    "command_path": "/agentcore/harness/get",
    "error_name": "ExpiredTokenException",
    "error_source": "user",
    "is_tui": false
  }
}
{
  "metricName": "cli.command_run",
  "value": 103,
  "attrs": {
    "service.name": "agentcore-cli",
    "service.version": "0.0.0",
    "agentcore-cli.installation_id": "42edb4ee-1d41-42cf-96ab-195b0356e05a",
    "agentcore-cli.session_id": "752a393a-668d-4a94-bc28-207cc6146ca9",
    "os.type": "Linux",
    "os.version": "6.12.94-123.190.amzn2023.x86_64",
    "host.arch": "x64",
    "node.version": "v24.3.0",
    "exit_reason": "failure",
    "command_path": "/agentcore/harness/list",
    "error_name": "ExpiredTokenException",
    "error_source": "user",
    "is_tui": false
  }
}
{
  "metricName": "cli.command_run",
  "value": 33,
  "attrs": {
    "service.name": "agentcore-cli",
    "service.version": "0.0.0",
    "agentcore-cli.installation_id": "42edb4ee-1d41-42cf-96ab-195b0356e05a",
    "agentcore-cli.session_id": "769086c9-0dd3-432e-b24e-b2cf97be40f1",
    "os.type": "Linux",
    "os.version": "6.12.94-123.190.amzn2023.x86_64",
    "host.arch": "x64",
    "node.version": "v24.3.0",
    "exit_reason": "success",
    "command_path": "/agentcore/config",
    "is_tui": false
  }
}

Notes

If the TUI fails to fetch data, that error doesn't hit the root handler, and therefore marks the result as success. This is reasonable because the TUI did open, and the command isn't valid. Opened a thread on this, and will change behavior in a follow-up if we think this doesn't make sense.

codecov-commenter · 2026-07-30T11:51:19Z

Codecov Report

❌ Patch coverage is 97.61905% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.86%. Comparing base (c7f857b) to head (0139beb).

Files with missing lines	Patch %	Lines
src/router/router.tsx	80.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1870      +/-   ##
============================================
- Coverage     95.87%   95.86%   -0.01%     
============================================
  Files           198      198              
  Lines          9379     9417      +38     
============================================
+ Hits           8992     9028      +36     
- Misses          387      389       +2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tejaskash

Only one comment worth looking at

tejaskash · 2026-07-30T18:52:00Z

  c.action(async (...actionArgs: unknown[]) => {
    const command = actionArgs[actionArgs.length - 1] as Command;
    const merged = command.optsWithGlobals();

+    const telemetryAttributesRecorder = ctx.value(TelemetryAttributesRecorderKey);
+    const commandPath = ctx.require(PathKey);
+
+    telemetryAttributesRecorder?.record({
+      command_path: commandPath,
+    });
+


This records the command path inside Commander’s action callback. Commander rejects missing mandatory flags, unknown options, and help requests before invoking that callback. Verified this as below:

agentcore config telemetry.audit true agentcore runtime invoke || true jq '.attrs' "$(ls -t ~/.agentcore/telemetry/*.jsonl | head -n 1)" Verified output includes: { "exit_reason": "failure", "command_path": "unknown" }

Good catch, was able to handle this by intercepting the error on the way up. Since double recording is a no-op, this should be a safe approach. I wasn't able to find a single place to move this such that it always worked, but lmk if you have any ideas.

aidandaly24 · 2026-07-30T21:02:13Z

+/** Adds a compile time check that the given object has no free-form fields to prevent PII leakage **/
+function safeSchema<T extends Record<string, EnumeratedField>>(shape: T) {
+  return z.object(shape);
+}


I think this only protects the declared schema at compile time, so it can still be bypassed at the sink boundary. I tested a typed attributes value containing prompt: "review-secret" through DefaultTelemetryClient.emit, and the sink received prompt because emit never parses attributeSchema. I think emit should probably parse and forward the validated attributes, what do you think? This is obviously a two way door.

Yeah I think the issue is that there is an implicit assumption that the client's emit is called with the recorder's output. The choices are to make that assumption explicit, or to parse the schema twice.

I think I'm going to make it explicit by passing the recorder in directly to avoid this.

refactored this one more time. I moved emit onto the recorder (which is now called the metric event). I think this is much simpler since the interface becomes, createMetricEvent on the client then emit on the event for callers, instead of having to pass the metric/recorder around.

aidandaly24 · 2026-07-30T21:03:33Z

-      // TODO: add error details to telemetry recorder;
-      commandRunTelemetryRecorder.record({ exit_reason: "failure" });
-
+      commandRunTelemetryRecorder.record({


--help currently gets recorded as a failed command even though it exits 0. With audit enabled I got exit_reason: "failure" and error_source: "internal". Should CommanderError be handled using its exit code so help remains successful and nonzero usage errors are classified as user failures? Unless is there a reason that you didn't go with that?

yeah good catch. I don't think we handle commander errors at the root level classification so they end up being unknown. I can pick that up as a follow-up. I think we just need some logic in AgentCoreCLIError.fromError to check if its a commander error and map its fields to our fields.

meant to just leave as comment since these are really just questions

… client

…ting

tejaskash

LGTM

github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Jul 30, 2026

Hweinstock force-pushed the feat/telemetry-shapes-setup branch from a8615b3 to adaf6b5 Compare July 30, 2026 14:45

Hweinstock changed the title ~~wip(tel): populate base attributes for cli.command_run metrics.~~ feat(tel): populate base attributes for cli.command_run metrics. Jul 30, 2026

Hweinstock marked this pull request as ready for review July 30, 2026 18:30

tejaskash requested changes Jul 30, 2026

View reviewed changes

aidandaly24 previously requested changes Jul 30, 2026

View reviewed changes

Hweinstock force-pushed the feat/telemetry-shapes-setup branch from 03aa1dc to da404c0 Compare July 30, 2026 22:40

tejaskash previously approved these changes Jul 30, 2026

View reviewed changes

Hweinstock added 9 commits July 31, 2026 12:31

feat(errors): allow callers to override error name on base error

afc0fb0

feat(tel): add command path shapes explicitly

287b57f

feat(tel): route telemetry through the router context

d728cf8

refactor(tel): switch to shape check for command path

ba52e73

feat(tel): wire up is_tui flag

15ac834

docs(tel): remove some noisy comments

a154f73

refactor(tel): move parsing into emit by coupling the recorder to the…

b9973ee

… client

fix(tel): ensure commander errors recieve the command path before exi…

e8f3d7d

…ting

refactor(tel): move to metric event instead of attributes recorder

da68951

Hweinstock dismissed tejaskash’s stale review via da68951 July 31, 2026 12:32

Hweinstock force-pushed the feat/telemetry-shapes-setup branch from da404c0 to da68951 Compare July 31, 2026 12:32

Hweinstock added 2 commits July 31, 2026 12:37

refactor(tel): rename end to emit

f01e5ff

refactor(tel): rename start --> create

0139beb

Hweinstock force-pushed the feat/telemetry-shapes-setup branch from 6459e65 to 0139beb Compare July 31, 2026 13:43

tejaskash approved these changes Jul 31, 2026

View reviewed changes

aidandaly24 approved these changes Jul 31, 2026

View reviewed changes

Hweinstock merged commit bb14e85 into aws:refactor Jul 31, 2026
8 checks passed

Hweinstock deleted the feat/telemetry-shapes-setup branch July 31, 2026 16:14

Navigation Menu

Uh oh!

feat(tel): populate base attributes for cli.command_run metrics. - #1870

feat(tel): populate base attributes for cli.command_run metrics. #1870
Hweinstock merged 11 commits into
aws:refactorfrom
Hweinstock:feat/telemetry-shapes-setup

Hweinstock commented Jul 30, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 30, 2026 •

edited

Loading

Uh oh!

tejaskash left a comment

Uh oh!

tejaskash Jul 30, 2026

Uh oh!

Hweinstock Jul 30, 2026

Uh oh!

aidandaly24 Jul 30, 2026

Uh oh!

Hweinstock Jul 30, 2026 •

edited

Loading

Uh oh!

Hweinstock Jul 31, 2026 •

edited

Loading

Uh oh!

aidandaly24 Jul 30, 2026

Uh oh!

Hweinstock Jul 30, 2026 •

edited

Loading

Uh oh!

tejaskash left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Hweinstock commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Passing the Recorder Down

Defining Shapes

Testing

Notes

Uh oh!

codecov-commenter commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tejaskash left a comment

Choose a reason for hiding this comment

Uh oh!

tejaskash Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Hweinstock Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

aidandaly24 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Hweinstock Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aidandaly24 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Hweinstock Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Hweinstock commented Jul 30, 2026 •

edited

Loading

codecov-commenter commented Jul 30, 2026 •

edited

Loading

Hweinstock Jul 30, 2026 •

edited

Loading

Hweinstock Jul 31, 2026 •

edited

Loading

Hweinstock Jul 30, 2026 •

edited

Loading