feat(aws-agents): update agent skills for AgentCore CLI v1 by notgitika · Pull Request #502 · aws/agent-toolkit-for-aws

notgitika · 2026-10-26T20:18:30Z

Description

Updates the aws-agents skills owned by this half of the AgentCore CLI v1 split. Every agentcore command, flag, config key and behaviour claim now matches AgentCore CLI v1.0.0 (@aws/agentcore@latest).

Skills and files covered

agents-get-started:
- v1 templates (create --name --template)
- Harness vs Runtime choice
- new project layout
- dev ports and behaviour
- first deploy and invoke for both Runtime and Harness
agents-deploy and references/versioning.md:
- build plus a real cdk diff preview
- targets and status
- Runtime and Harness versions and endpoints, rollback, canary options
- extending the vended CDK app
- export
- a teardown that is scoped to one target, asks for confirmation, and cleans up what survives stack deletion
agents-debug and references/doctor.md:
- v1 status, logs, traces and invoke forms
- finding startup crashes
- preflight checks that work with assumed-role and SSO credentials and a region taken from env vars
- routing v0 projects to agentcore migrate
agents-optimize (evals, observability, cost):
- the eval command tree: on-demand, batch, online, datasets, recommendations
- working code-evaluator recipes
- the AWS/Bedrock-AgentCore metrics
- realistic trace and online-eval timing
- current pricing
agents-harden and references/limits.md:
- agentcore.json v2 and harness.yaml keys
- auth defaults, including gateways defaulting to NONE
- least-privilege IAM that works with cross-region inference profiles
- Harness coverage
- quotas with quota codes
CloudWatch Omni (omni-agents-instrumentation.md), AgentCore path only:
- v1 detection ("version": 2, including Harness-only projects)
- agentcore dev --no-traces with .env.local
- streaming-response instrumentation
- removed the "Python-only" claim
plugins/aws-agents/README.md:
- prerequisites updated to v1.0.0+, Node.js 20.12+ and git/jq
- references column matches the files on disk

Version policy: all updated skills declare requires-cli: ">=1.0.0". If a user has an older CLI, the skills tell them to upgrade. For v0 projects ("version": 1) they say to commit, then review the agentcore migrate plan, then apply it with --yes.

Not in this PR (owned separately): agents-build, agents-connect, agents-pay and amazon-bedrock. agents-build/references/teardown.md still describes v0, so agents-deploy no longer links to it.

How this was validated

Every agentcore invocation in the changed files was checked against the v1 CLI's --help and source.
Agents using only these skills ran two rounds of end-to-end tests with real deploys, covering:
- scaffold, dev, deploy and invoke;
- Harness endpoints and teardown;
- debugging a startup crash;
- every eval path;
- Omni local traces;
- migrating a v0 project.
Independent reviews followed each round; the findings from both rounds are applied.

Type of Change

Checklist

I have read the CONTRIBUTING guide
My changes pass python3 tools/validate.py
I have updated relevant documentation

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nborges-aws

A few callouts, but all minor changes. Overall LGTM

nborges-aws · 2026-10-27T19:20:58Z

+cp agentcore/agentcore.json /tmp/agentcore.json.bak
+cp agentcore/.env.local /tmp/env.local.bak 2>/dev/null
+agentcore remove all --yes
+agentcore deploy --yes


should this specify --target the example above uses staging as the target. If staging is not the default, this command would target the wrong stack

Good catch — fixed in 48b40fd. Step 0 is now agentcore status --json --target <target> and step 2 is agentcore deploy --yes --target <target>, with a note that without --target deploy uses the default target, which may be a different stack.

nborges-aws · 2026-10-27T19:21:07Z

-
-score=$(echo "$result" | jq -r '.run.results[0].aggregateScore // empty')
+# Recent-traffic gate:
+result=$(agentcore eval ondemand evaluate \


shouldn't this pass a session-id? The text on L#313 says "score the sessions your integration tests just produced", but this would include results even outside of that session

Agreed — fixed in 48b40fd. The recent-traffic gate now takes SESSION_IDS (the IDs the integration tests passed to agentcore invoke --session-id) and runs eval ondemand evaluate --session-ids instead of --lookback-days 1, so it only scores the sessions from that CI run.

nborges-aws · 2026-10-27T19:22:31Z

+#   --payload-template '{"prompt":"{input}"}' \
+#   --evaluators "$EVALUATOR") || { echo "⚠️  Dataset replay failed" >&2; exit 1; }
+
+score=$(echo "$result" | jq -r '[.results[] | select(.value != null) | .value] | if length == 0 then empty else add / length end')


I see we're dropping null in this calculation. Not sure exactly how this works, but would the value be null if the evaluation errored out? Because if so, we're gonna filter out all the errors and the result will be a false-positive success

Yes, that was a false-pass risk. A result that fails to evaluate comes back with errorCode/errorMessage and no value, so filtering out nulls hid errors. Fixed in 48b40fd: the gate now counts results with an errorCode or a missing value, prints them, and exits 1 before computing the average. I checked the script against a stubbed result with one errored evaluation, and it fails as expected.

arnewouters · 2026-11-06T19:40:59Z

+[tasks."test:payment-scripts"]
+description = "Run payment skill script tests"
+run = """
+uv run --with pytest python -m pytest \
+  plugins/aws-agents/skills/agents-build/scripts/test_setup_payment_user.py \
+  plugins/aws-agents/skills/agents-pay/scripts/test_resource_resolution.py
+"""


If you want custom ci workflows, please do something like startup-advisor so they are limited to changes in your plugin folder

Update agents-get-started, agents-deploy, agents-debug, agents-optimize, agents-harden, the CloudWatch Omni instrumentation reference, and the plugin README for AgentCore CLI v1.0.0 (templates, Harness vs Runtime, build/deploy/targets, logs/traces/status, eval subcommands, migrate).

Harness-endpoint-aware teardown and leftover cleanup, startup-crash debugging, inference-profile IAM scoping, preflight fixes for assumed-role creds and env regions, code-evaluator recipes, trace timing, metrics namespace, cost rates, dataset schema, and migrate git precondition.

Scope teardown to one target with explicit confirmation, fix startup-crash log query, consistent migrate approval flow, --json for non-TTY list commands, gateway auth table, CreateEvent quotas, CDK/ECR IAM naming, and trace timing consistency.

Judge model that accepts temperature, online-eval and trace timing, commit- before-plan migrate flow, complete BYO ECR/KMS and deploy IAM checks, target-safe teardown with .env.local backup and service-side eval cleanup, Harness coverage in harden, CodeZip version-creation caveat, Omni collector install, and accuracy nits.

- Teardown names the target on status/deploy so it can't hit the default stack. - Quality gate scores only the CI run's own sessions (--session-ids) and fails on errored evaluations (errorCode / no value) instead of dropping them. - Add teardown trigger phrases to agents-deploy; 'tear down' prompts were routing to agents-build's v0 teardown reference.

* feat(aws-agents): update skills to reflect AgentCore CLI v1 * fix(aws-agents): fix runtime and harness configuration guidance * fix(aws-agents): correct header allowlist example * fix(aws-agents): ensure agent deployment workflows persist selected target * fix(aws-agents): fix Privy guidance in payments skill * fix(aws-agents): persist selected target on gateway and policy operations * fix(aws-agents): fix payment admin setup instructions * fix(aws-agents): update agent auth and cross-session memory examples * fix: address review findings for agentcore CLI v1 skill updates * fix(aws-agents): require shim manifests and prefer v1 payment status

* fix(agents-build): require human approval for payment sessions * fix(agents-pay): validate deployment target configuration * docs(payments): correct v1 setup and spending guidance * docs(agents-build): correct browser setup guidance * docs(observability): clarify local traces and streaming errors * docs(agents-get-started): explain failed create recovery * docs(agents-connect): require MCP for semantic search * docs(amazon-bedrock): clarify Harness CLI inputs and cleanup * docs(amazon-bedrock): correct Harness migration and rollback guidance

)

… switching (#559)

Co-authored-by: jariy17 <tjariy+jariy17@users.noreply.github.com>

Added warning about deploying the Strands agent with the Browser tool as a Container Runtime due to packaging limitations.

nborges-aws reviewed Oct 27, 2026

View reviewed changes

nborges-aws previously approved these changes Oct 27, 2026

View reviewed changes

nborges-aws mentioned this pull request Oct 29, 2026

feat(aws-agents): update skills to reflect AgentCore CLI v1 #522

Merged

9 tasks

notgitika dismissed nborges-aws’s stale review via d369825 October 29, 2026 20:08

notgitika marked this pull request as ready for review October 29, 2026 20:17

notgitika requested review from a team as code owners October 29, 2026 20:17

jariy17 mentioned this pull request Nov 2, 2026

fix(add-harness): document authorizer configuration casing aws/agentcore-cli#2744

Closed

13 of 14 tasks

jariy17 force-pushed the gitika/agentcore-cli-v1 branch from ef7f810 to d148c09 Compare November 2, 2026 22:36

notgitika mentioned this pull request Nov 3, 2026

fix(aws-agents): bug bash skill fixes for AgentCore CLI v1 #536

Merged

nborges-aws previously approved these changes Nov 6, 2026

View reviewed changes

nborges-aws force-pushed the gitika/agentcore-cli-v1 branch 2 times, most recently from 38e65ef to 7a802fa Compare November 6, 2026 16:44

tejaskash previously approved these changes Nov 6, 2026

View reviewed changes

arnewouters requested changes Nov 6, 2026

View reviewed changes

nborges-aws dismissed stale reviews from tejaskash and themself via 97fa394 November 11, 2026 16:09

gitikavj and others added 11 commits November 11, 2026 11:39

docs(agents-build): warn against CodeZip for browser tool

1f9597a

fix(agents-deploy): clarify removals across deployment targets

ddb532a

fix(aws-agents): avoid blanket git staging in migration guidance

2ef141f

fix(agents-build): correct Runtime header forwarding guidance

30379d7

fix(amazon-bedrock): correct Harness authorizer flag casing

7a1f498

jariy17 and others added 17 commits November 11, 2026 11:39

fix(agents-deploy): clarify rollback deployment behavior

bd6bb85

test(payments): run skill script tests in CI

4cd74de

docs(get-started): add teardown guidance

b11d8a9

docs(dev): distinguish Harness deployment behavior

8508a8e

docs(policy): explain strict validation findings

fb04bdc

fix(agents-harden): clarify JWT invoke failures

3edddb1

fix(agents-pay): stop tracking staged skill copy

e3268e8

fix(amazon-bedrock): use v1 harness configuration names

418c791

fix(amazon-bedrock): update harness shim deployment guidance

9af1097

fix(agents-build): correct shared memory list numbering

ecbd57d

fix(aws-agents): bug bash skill fixes for AgentCore CLI v1 (#536)

63087cd

fix(amazon-bedrock): use bedrock:Retrieve for KB shim IAM guidance (#556

66eef70

)

fix(skill): replace incorrect default CLI model with example of model…

6e9a9a9

… switching (#559)

fix(agents-optimize): use target region and check response traces (#560)

5691690

Co-authored-by: jariy17 <tjariy+jariy17@users.noreply.github.com>

Update browser.md with Container Runtime warning

9696bd8

Added warning about deploying the Strands agent with the Browser tool as a Container Runtime due to packaging limitations.

ci(aws-agents): run payment tests only for plugin changes

7f39e0a

nborges-aws force-pushed the gitika/agentcore-cli-v1 branch from 97fa394 to 7f39e0a Compare November 11, 2026 16:39

Navigation Menu

Repository navigation

feat(aws-agents): update agent skills for AgentCore CLI v1 - #502

feat(aws-agents): update agent skills for AgentCore CLI v1#502
notgitika wants to merge 28 commits into
mainfrom
gitika/agentcore-cli-v1

notgitika commented Oct 26, 2026

Uh oh!

nborges-aws left a comment

Uh oh!

nborges-aws Oct 27, 2026

Uh oh!

notgitika Oct 27, 2026

Uh oh!

nborges-aws Oct 27, 2026

Uh oh!

notgitika Oct 27, 2026

Uh oh!

nborges-aws Oct 27, 2026

Uh oh!

notgitika Oct 27, 2026

Uh oh!

arnewouters Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

notgitika commented Oct 26, 2026

Description

Type of Change

Checklist

Uh oh!

nborges-aws left a comment

Choose a reason for hiding this comment

Uh oh!

nborges-aws Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nborges-aws Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nborges-aws Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

arnewouters Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants