Skip to content

fix: add batch eval and recommendation IAM permissions to docs - #1106

Closed
notgitika wants to merge 138 commits into
mainfrom
fix/add-batch-eval-rec-permissions
Closed

fix: add batch eval and recommendation IAM permissions to docs#1106
notgitika wants to merge 138 commits into
mainfrom
fix/add-batch-eval-rec-permissions

Conversation

@notgitika

Copy link
Copy Markdown
Contributor

Summary

  • Add BatchEvalAndRecommendations statement to iam-policy-user.json with 9 actions for batch evaluation and recommendation APIs
  • Add CloudWatch Logs write permissions (CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) to LogsStreamingAndSearch — needed because the batch eval API creates a log group for results output
  • Update PERMISSIONS.md with batch eval/recommendations reference section, updated logging entries, and scoping-down table entry
  • Fix e2e test: config-bundle-eval-rec.test.ts searched for resourceType === 'configBundle' (camelCase) but the status command outputs 'config-bundle' (kebab-case)

Also applied logs:CreateLogGroup/CreateLogStream/PutLogEvents to the e2e-github-actions IAM role in the test account so CI passes immediately.

Context

E2E tests in shard 5/6 were failing with:

  1. Batch eval 403: FAS credentials do not have permission to create CloudWatch log groups — the batch eval API writes results to a CloudWatch log group and needs logs:CreateLogGroup
  2. Config bundle status: Config bundle "E2eTestBundle" should appear in status: expected undefined — the test used camelCase configBundle but the status command outputs kebab-case config-bundle

Test plan

  • iam-policy-user.json is valid JSON
  • prettier --check passes on all changed files
  • Applied IAM changes to e2e-github-actions role in test account
  • Re-run E2E shard 5/6 to confirm batch eval and config bundle status tests pass

avi-alpert and others added 30 commits March 5, 2026 13:20
GITHUB_TOKEN lacks the 'workflows' permission, so pushing workflow
file changes from the public repo causes the sync to fail. Use
--no-commit --no-ff and restore .github/workflows/ from HEAD before
committing, in both the clean merge and conflict paths.
aidandaly24 and others added 22 commits April 29, 2026 16:08
# Conflicts:
#	CHANGELOG.md
#	package-lock.json
#	package.json
The release workflow CDK sync step was stripping the ^ prefix,
pinning the dependency to an exact version.
fix: restore ^ caret prefix on @aws/agentcore-cdk dependency
# Conflicts:
#	CHANGELOG.md
#	package-lock.json
#	package.json
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Brings in all evo preview features from main:
- Config bundles, batch evaluation, recommendations, AB testing
- Deploy teardown cleanup for evo resources
- Schema fixes and JSON Schema compat
- Version bump to 0.13.0

Conflicts resolved keeping both preview's harness support and main's evo additions.
The main→preview sync overwrote these with main's versions.
Restore preview's version (1.0.0-preview.5) and changelog.
…teway tools (#1083)

* feat: replace credentialProviderName with outboundAuth for harness gateway tools

The CLI's harness schema for agentcore_gateway tools had a
credentialProviderName field that didn't exist in the harness service
model. The service expects an outboundAuth union with three members:
awsIam (SigV4), none, and oauth (Bearer token via AgentCore Identity).

- Replace credentialProviderName with outboundAuth union in harness
  schema (awsIam | none | oauth with providerArn, scopes, grantType,
  customParameters)
- Add superRefine to reject legacy credentialProviderName with
  migration message
- Add --outbound-auth, --provider-arn, --scopes, --grant-type flags
  to `agentcore add tool` command
- Add outbound auth type selection to TUI harness wizard (3 new steps)
- Update HarnessPrimitive to build outboundAuth config from options
- Add schema tests for all outboundAuth variants and mapper round-trip
  tests

* fix: reject oauth-only flags when outbound auth is awsIam or none

When --outbound-auth is set to awsIam or none, the CLI now rejects
--provider-arn, --scopes, and --grant-type instead of silently
ignoring them.

* fix: improve outbound auth DX — default indication, scopes example

- CLI help: indicate awsIam is the default when --outbound-auth is
  omitted, add scope format examples to --scopes description
- TUI: mark AWS IAM option as "(default)" in gateway auth selector
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
feat: add allowedTools and tools overrides to harness invocation API
These options existed in AddHarnessOptions and were handled by the add()
method and TUI wizard, but were never registered as Commander .option()
declarations in registerCommands(). This caused the documented CLI flags
(per the DevGuide) to silently fail on `agentcore add harness`.

Adds the missing CLI option registrations for:
- --system-prompt, --tools, --mcp-name, --mcp-url
- --gateway-arn, --gateway-outbound-auth, --gateway-provider-arn, --gateway-scopes

Also adds type annotations in the action handler, pass-through mapping
to this.add(), and unit tests for the tools code path.
Without validation, missing companion flags (e.g. --tools remote_mcp
without --mcp-url) cause tools to be silently dropped from harness.json.

Validates: unknown tool names, required companion flags for remote_mcp
(--mcp-name, --mcp-url) and agentcore_gateway (--gateway-arn), invalid
--gateway-outbound-auth values, and required oauth fields
(--gateway-provider-arn, --gateway-scopes).
…-cli-flags-public

fix: add --system-prompt and --tools CLI flags to add harness
- Add BatchEvalAndRecommendations statement to iam-policy-user.json
  (StartBatchEvaluation, GetBatchEvaluation, ListBatchEvaluations,
  StopBatchEvaluation, DeleteBatchEvaluation, StartRecommendation,
  GetRecommendation, ListRecommendations, DeleteRecommendation)
- Add CloudWatch Logs write permissions to LogsStreamingAndSearch
  (CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups)
  needed for batch eval results output
- Update PERMISSIONS.md with batch eval/recommendations reference
  section, logging entries, and scoping-down table
- Fix e2e test: config bundle status check used 'configBundle'
  (camelCase) but status command outputs 'config-bundle' (kebab-case)

Also applied logs:CreateLogGroup/CreateLogStream/PutLogEvents to the
e2e-github-actions role in the test account so CI passes immediately.
@notgitika
notgitika requested a review from a team May 4, 2026 16:28
@github-actions github-actions Bot added size/xl PR size: XL agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026
if (!roleArn) {
throw new Error(`IAM CreateRole succeeded but returned no role ARN for "${roleName}"`);
}
_needsPropagationWait = true;
@agentcore-cli-automation

Copy link
Copy Markdown

The actual fix commit (6837467) is tight and correct — the 9 bedrock-agentcore:* actions line up with the API operations in agentcore-batch-evaluation.ts and agentcore-recommendation.ts, the logs additions are documented in the scoping-down table in PERMISSIONS.md, and the configBundleconfig-bundle test fix matches the actual output in src/cli/commands/status/action.ts:250. That part all looks good.

However, there's one serious issue with the PR as a whole: the branch is based on preview but targets main, so in addition to the intended 3-file change it's pulling in ~137 commits and ~180 unrelated files from the preview branch (harness code, recommendations, batch eval, ab-tests, config-bundles, new workflows, CHANGELOG/package.json version bumps, etc., totaling roughly +16k/-943 lines). The PR description only describes the 3-file IAM/test change, so merging as-is would silently land a huge amount of preview-only work into main.

A couple of ways to fix this:

  1. Cherry-pick onto main. Create a new branch off origin/main and cherry-pick 6837467 onto it, then repoint the PR (or open a new one). This is the cleanest option given the scope mismatch with the description.
  2. Retarget the PR to preview. If the intent is actually to land this on the preview branch first (and let it reach main via the normal sync-preview flow), change the base from main to preview. The diff will then be just the 3 files the commit touches.
  3. Intentionally promote preview → main via this PR. If you actually do want all 137 preview commits to land in main now, the PR title/description should reflect that (and it should probably go through whatever the standard preview→main promotion path is rather than riding on a "fix" PR).

Option 1 or 2 seems most likely given the description. Worth sanity-checking which one matches intent before merging.

@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026
@notgitika notgitika closed this May 4, 2026
@aidandaly24
aidandaly24 deleted the fix/add-batch-eval-rec-permissions branch May 7, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl PR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants