fix: add batch eval and recommendation IAM permissions to docs - #1106
fix: add batch eval and recommendation IAM permissions to docs#1106notgitika wants to merge 138 commits into
Conversation
GITHUB_TOKEN lacks the 'workflows' permission, so pushing workflow file changes from the public repo causes the sync to fail. Use --no-commit --no-ff and restore .github/workflows/ from HEAD before committing, in both the clean merge and conflict paths.
# Conflicts: # CHANGELOG.md # package-lock.json # package.json
sync-preview: merge main into preview
The release workflow CDK sync step was stripping the ^ prefix, pinning the dependency to an exact version.
fix: restore ^ caret prefix on @aws/agentcore-cdk dependency
sync-preview: merge main into preview
# Conflicts: # CHANGELOG.md # package-lock.json # package.json
sync-preview: merge main into preview
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…harness invocation API
Brings in all evo preview features from main: - Config bundles, batch evaluation, recommendations, AB testing - Deploy teardown cleanup for evo resources - Schema fixes and JSON Schema compat - Version bump to 0.13.0 Conflicts resolved keeping both preview's harness support and main's evo additions.
The main→preview sync overwrote these with main's versions. Restore preview's version (1.0.0-preview.5) and changelog.
…teway tools (#1083) * feat: replace credentialProviderName with outboundAuth for harness gateway tools The CLI's harness schema for agentcore_gateway tools had a credentialProviderName field that didn't exist in the harness service model. The service expects an outboundAuth union with three members: awsIam (SigV4), none, and oauth (Bearer token via AgentCore Identity). - Replace credentialProviderName with outboundAuth union in harness schema (awsIam | none | oauth with providerArn, scopes, grantType, customParameters) - Add superRefine to reject legacy credentialProviderName with migration message - Add --outbound-auth, --provider-arn, --scopes, --grant-type flags to `agentcore add tool` command - Add outbound auth type selection to TUI harness wizard (3 new steps) - Update HarnessPrimitive to build outboundAuth config from options - Add schema tests for all outboundAuth variants and mapper round-trip tests * fix: reject oauth-only flags when outbound auth is awsIam or none When --outbound-auth is set to awsIam or none, the CLI now rejects --provider-arn, --scopes, and --grant-type instead of silently ignoring them. * fix: improve outbound auth DX — default indication, scopes example - CLI help: indicate awsIam is the default when --outbound-auth is omitted, add scope format examples to --scopes description - TUI: mark AWS IAM option as "(default)" in gateway auth selector
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Release v1.0.0-preview.6
feat: add allowedTools and tools overrides to harness invocation API
These options existed in AddHarnessOptions and were handled by the add() method and TUI wizard, but were never registered as Commander .option() declarations in registerCommands(). This caused the documented CLI flags (per the DevGuide) to silently fail on `agentcore add harness`. Adds the missing CLI option registrations for: - --system-prompt, --tools, --mcp-name, --mcp-url - --gateway-arn, --gateway-outbound-auth, --gateway-provider-arn, --gateway-scopes Also adds type annotations in the action handler, pass-through mapping to this.add(), and unit tests for the tools code path.
Without validation, missing companion flags (e.g. --tools remote_mcp without --mcp-url) cause tools to be silently dropped from harness.json. Validates: unknown tool names, required companion flags for remote_mcp (--mcp-name, --mcp-url) and agentcore_gateway (--gateway-arn), invalid --gateway-outbound-auth values, and required oauth fields (--gateway-provider-arn, --gateway-scopes).
…-cli-flags-public fix: add --system-prompt and --tools CLI flags to add harness
- Add BatchEvalAndRecommendations statement to iam-policy-user.json (StartBatchEvaluation, GetBatchEvaluation, ListBatchEvaluations, StopBatchEvaluation, DeleteBatchEvaluation, StartRecommendation, GetRecommendation, ListRecommendations, DeleteRecommendation) - Add CloudWatch Logs write permissions to LogsStreamingAndSearch (CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) needed for batch eval results output - Update PERMISSIONS.md with batch eval/recommendations reference section, logging entries, and scoping-down table - Fix e2e test: config bundle status check used 'configBundle' (camelCase) but status command outputs 'config-bundle' (kebab-case) Also applied logs:CreateLogGroup/CreateLogStream/PutLogEvents to the e2e-github-actions role in the test account so CI passes immediately.
| if (!roleArn) { | ||
| throw new Error(`IAM CreateRole succeeded but returned no role ARN for "${roleName}"`); | ||
| } | ||
| _needsPropagationWait = true; |
|
The actual fix commit (6837467) is tight and correct — the 9 However, there's one serious issue with the PR as a whole: the branch is based on A couple of ways to fix this:
Option 1 or 2 seems most likely given the description. Worth sanity-checking which one matches intent before merging. |
Summary
BatchEvalAndRecommendationsstatement toiam-policy-user.jsonwith 9 actions for batch evaluation and recommendation APIsCreateLogGroup,CreateLogStream,PutLogEvents,DescribeLogGroups) toLogsStreamingAndSearch— needed because the batch eval API creates a log group for results outputPERMISSIONS.mdwith batch eval/recommendations reference section, updated logging entries, and scoping-down table entryconfig-bundle-eval-rec.test.tssearched forresourceType === 'configBundle'(camelCase) but the status command outputs'config-bundle'(kebab-case)Also applied
logs:CreateLogGroup/CreateLogStream/PutLogEventsto thee2e-github-actionsIAM role in the test account so CI passes immediately.Context
E2E tests in shard 5/6 were failing with:
FAS credentials do not have permission to create CloudWatch log groups— the batch eval API writes results to a CloudWatch log group and needslogs:CreateLogGroupConfig bundle "E2eTestBundle" should appear in status: expected undefined— the test used camelCaseconfigBundlebut the status command outputs kebab-caseconfig-bundleTest plan
iam-policy-user.jsonis valid JSONprettier --checkpasses on all changed filese2e-github-actionsrole in test account