Repository navigation
fix(aws-startup-advisor): agent-advisor defaults to bedrock-runtime, selects Mantle only by a verified selector, adds Runtime Responses/Chat/Messages paths - #588
Open
herosjourney wants to merge 54 commits into
Conversation
added 6 commits
November 6, 2026 23:17
…tadata from the public model cards The plugin recorded GPT-5.4 and GPT-5.5 as bedrock-mantle-only (no CRIS, no inference profile, no bedrock-runtime surface) with Chat Completions "unconfirmed", a 272K context window and no long-context price tier, and the canonical catalog cited "internal reference" probes as provenance. The public model cards no longer say any of that. This PR corrects the metadata only; it does not change endpoint selection, dispatch, or add new code paths. Sources (read 2026-11-06 via the AWS Documentation MCP tools): - https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-54.html - https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-55.html What the cards state, now recorded verbatim in the catalog and prose: - Endpoints: bedrock-runtime supported (Responses, Chat Completions, Converse; not Messages/Invoke) and bedrock-mantle supported (Responses, Chat Completions). Runtime has no in-region form; requests name us.openai.gpt-5.4 / global.openai.gpt-5.4 and us.openai.gpt-5.5 / global.openai.gpt-5.5. Mantle in-region: 5.4 = us-east-1, us-east-2, us-west-2, us-gov-west-1; 5.5 = us-east-1, us-east-2. US Geo CRIS from 6 regions, Global CRIS from 33. - Context 1,050,000 tokens shared, 128,000 max output. - Pricing (Standard, In-Region only; Priority/Flex unsupported): 5.4 short 2.75 / 0.275 cache read / 16.50, long 5.50 / 0.55 / 24.75, GovCloud short 3.375 / 0.3375 / 20.25; 5.5 short 5.50 / 0.55 / 33.00, long 11.00 / 1.10 / 49.50. No Geo/Global price rows, so prose keeps the ~10% figure for these two on every inference option and never claims parity. - Runtime features: invocation logs, streaming, Guardrails and application inference profiles (Converse only), implicit prompt caching and structured outputs (Responses only); no server-side tool use. Changes: - skills/shared/ai/models/openai-bedrock.json: GPT-5.4/5.5 rows re-sourced to the public cards (no "internal reference"), mantle_openai_chat available, runtime_converse path with CRIS, inference_profiles (us/global), region_availability, pricing tiers, context/output limits; path-level constraints corrected. - Canonical shared docs: ai-openai-to-bedrock.md (Tier 0 table, region gate, caching, Bedrock-native section, volume guidance), ai-migration-guardrails.md, schema-design-aws-ai.md (ban is on BARE ids paired with converse/runtime, not on the models), bedrock-pricing-cache.md (rows, long-context table, caching note, GovCloud rate). - gcp-to-aws and azure-to-aws: references/shared/openai-on-bedrock.md, shared/schema-design-aws-ai.md (azure), design-ai.md, design.md/generate.md _assert lines (azure), generate-artifacts-ai.md (azure), generate-ai.md, discover-preview.md and clarify-ai-only.md (gcp), clarify-ai-only.md (azure). - bedrock-migration-assessment clarify-ai-only.md; agent-advisor model-recommend.md (documents the engine's current GPT-5.x reshape as an engine limitation, not a model fact). - llm-to-bedrock: resolve-bedrock-model-id.md (Case A/B), behavior-delta openai-to-bedrock.md, SKILL.md (two id-form sentences); docstrings/comments in preflight_bedrock.py, iam_policy.py, bedrock_pricing.py and the preflight 404 remedy string. The is_mantle_model / is_mantle_gpt ID-form logic is unchanged. - agents: llm2bedrock-code-rewriter.md (Converse rule is id-form-dependent), llm2bedrock-report-generator.md (cost text), llm2bedrock-prompt-evaluator.md. - openai_model_recommendation.py: only the chat_completions_to_responses block message text (dropped the "GPT-5.x on Mantle rejects Chat Completions" claim). - Tests: tests/test_gpt6_review_round2.py, test_gpt6_review_round5.py, test_gpt6_round4_findings.py and agent-advisor scripts/test_openai_model_recommendation.py now assert the corrected public facts (CRIS ids present, "mantle-only"/"no CRIS"/"unconfirmed" absent, catalog provenance is the public card). New regression tests in test_preflight_bedrock.py, test_iam_policy.py, test_bedrock_pricing.py prove us./global.openai.gpt-5.4|5.5 classify as runtime and bare ids as mantle in all three helpers. - Vendored copies under agent-advisor, gcp-to-aws, azure-to-aws re-synced with tools/sync-vendored.py. Validation (all run from the worktree with mise Python 3.12): - mise -C plugins/aws-startup-advisor run ci: PASS - lint:md 0 errors; lint:model-id OK (19 retired IDs enforced); lint:model-staleness fresh (4d old, 45d window); lint:model-catalog-drift OK (3 pre-existing warnings: sonnet-5-5, haiku-5-5, gpt-6.1-terra not in catalog; unrelated); lint:startup-model-pricing OK; lint:artifacts PASS (99 checked, 0 errors); lint:vendored-parity PASS (62 files byte-identical); lint:skill-refs PASS (1482 checked, 0 missing); lint:consent-disclosure OK; test: 1395 passed, 222 subtests passed; test:telemetry: 311 passed; test:heroku-source-review: 44 pass, 0 fail - agent-advisor pytest (skills/agent-advisor/scripts): 564 passed, 0 failed (562 at base + 2 new) - llm-to-bedrock pytest (skills/llm-to-bedrock/scripts): 418 passed, 0 failed (415 at base + 3 new; needs --with httpx --with aws-bedrock-token-generator) - root: python3 tools/validate.py "All validations passed"; mise run lint:md 0 issues / 2763 files; lint:manifests, lint:skill-parity, lint:spec all pass; test:tools 6 OK. mise run build's security step fails on a pre-existing gitleaks false positive in upstream commit f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118, "claude-3.5-sonnet"); gitleaks protect over this change set: no leaks found. - grep over plugins/ for mantle-only / "no CRIS" / "no inference profile" / "no runtime path" / "unconfirmed" / "internal reference" / "no 1M tier" near GPT-5.4/5.5: no remaining claims (only negative test assertions and the bare-id rule "bare proprietary openai.gpt-* IDs use Mantle only"). Base: 0a218f8 (upstream/main)
…ift caused by the GPT-5.4/5.5 catalog facts Review round 1 (CHANGES_REQUESTED, 2 blockers) on c1944a5. Finding 1: recording the public GPT-5.5/GPT-5.4 facts in skills/shared/ai/models/openai-bedrock.json (mantle_openai_chat available, region_availability) changes agent-advisor recommendations through the unchanged catalog-order fallback in openai_model_recommendation.py (_mantle_candidate_order -> _catalog_model_for_path): GPT-5.5 and GPT-5.4 precede GPT-6 Astra in catalog key order, so a GPT-6 Sol / 6.1 Sol Chat Completions source outside us-east-1 now lands on - openai.gpt-5.5 via mantle_openai_chat in us-east-2 (base: decision_required, mantle_region_unavailable, options mantle chat / runtime_converse) - openai.gpt-5.4 via mantle_openai_chat in us-west-2 (base: openai.gpt-6-astra) and a GPT-5.6 Sol Responses source in us-west-2 lands on openai.gpt-5.6-terra (base: openai.gpt-5.5, whose cataloged footprint is us-east-1/us-east-2 only). The previous commit message called this PR "no behavior change"; that was wrong for these cells. The catalog facts are kept (they are what the public cards state); the engine ordering is NOT changed in this PR. Instead: - skills/agent-advisor/scripts/test_openai_model_recommendation.py: new regression tests pin all five cells so the shift is deliberate and visible (test_gpt6_chat_outside_own_footprint_falls_back_to_in_region_gpt5_chat x4, test_gpt56_sol_responses_outside_us_east_skips_gpt55_for_in_region_terra). - skills/agent-advisor/references/phases/model-recommend/model-recommend.md: the engine-limitation paragraph now states that Chat fallbacks walk catalog order, that GPT-5.5/5.4 precede GPT-6 Astra, and that a GPT-6 Chat source outside us-east-1 lands on GPT-5.5 (us-east-2) / GPT-5.4 (us-west-2); a newer-generation source falling back to an older-generation model is an engine ordering limitation to resolve in the follow-up selection PR (PR 2). Finding 2: skills/agent-advisor/scripts/openai_model_recommendation.py:118-119 still said "the (Mantle-only) GPT-5.x family". Reworded to the engine fact (GPT-5.x sources not recognized as same-model runtime identities map to Claude tiers; GPT-5.5/5.4 do carry a cataloged runtime_converse path via us./global. CRIS ids). Grep over plugins/ for mantle-only / "no CRIS" / "no inference profile" / unconfirmed / "internal reference" near GPT-5.4/5.5/5.x is now clean apart from negative test assertions, the GovCloud Opus 5.5 pricing row, and the bare-id rule ("bare proprietary openai.gpt-* IDs use Mantle only"). Carry-forward for PR 2 (endpoint selection): reorder / tier the mantle_openai_chat candidate walk so a GPT-6 source does not fall back to GPT-5.4/5.5 ahead of GPT-6 Astra; add GPT-5.5/5.4 to _SOURCE_MODEL_KEYS so they are recognized as same-model runtime identities. No new behavior added; no dispatch or endpoint-selection code changed. No vendored files affected (none of the three files is in the sync manifest); lint:vendored-parity re-run and still 62/62 byte-identical. Sources (public model cards, read 2026-11-06 via the AWS Documentation MCP tools): - https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-54.html - https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-55.html Validation (run from the worktree with mise Python 3.12): - mise -C plugins/aws-startup-advisor run ci: PASS - lint:md 0 errors; lint:model-id OK (19 retired IDs enforced); lint:model-staleness fresh (4d old, 45d window); lint:model-catalog-drift OK (3 pre-existing warnings, unrelated); lint:startup-model-pricing OK; lint:artifacts PASS (99 checked, 0 errors); lint:vendored-parity PASS (62 files byte-identical); lint:skill-refs PASS (1482 checked, 0 missing); lint:consent-disclosure OK (6 checked); test: 1395 passed, 222 subtests passed; test:telemetry 311 passed; test:heroku-source-review 44 pass, 0 fail - agent-advisor pytest (skills/agent-advisor/scripts): 569 passed, 0 failed (564 at c1944a5 + 5 new) - llm-to-bedrock pytest (skills/llm-to-bedrock/scripts): 418 passed, 0 failed - root: python3 tools/validate.py "All validations passed"; mise run lint:md 0 issues / 2763 files; lint:manifests, lint:spec, lint:skill-parity pass; test:tools OK. mise run build's security step still fails only on the pre-existing gitleaks finding in upstream commit f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118); gitleaks protect --staged over this change set: no leaks found. Base: 0a218f8 (upstream/main)
…re constraints per model for GPT-5.4/5.5
The GPT-5.4 and GPT-5.5 model cards publish only us. and global. CRIS ids;
the in. profile exists for GPT-5.6 only. The runtime_converse path strength
and both schema-design vocabulary/checklist lines attached us./in./global.
to all three models, contradicting the catalog's own next constraint and
openai-on-bedrock.md. Rewritten so in. stays on GPT-5.6 / GPT-6.
The GPT-5.6 runtime constraint ("does not support server-side tool use,
structured outputs, or application inference profiles") is now scoped to
GPT-5.6, and a separate GPT-5.4/5.5 constraint records what their cards
mark: server-side tools unsupported; structured outputs and implicit prompt
caching Responses-only; Guardrails and application inference profiles
Converse-only.
Vendored copies re-synced (62 checked, 3 synced). Regression test added.
… the rewriter and Design/Generate checklists bc51f0c scoped the in. prefix to GPT-5.6 in the catalog and schema, but three consumers of that rule still said us./in./global. for GPT-5.4/5.5: agents/llm2bedrock-code-rewriter.md, gcp-to-aws design-ai.md's checklist, and azure-to-aws generate-artifacts-ai.md's checklist. Their model cards publish only us. and global. ids. Each now reads: us./in./global. for GPT-5.6; us./global. only for GPT-6, GPT-5.5 and GPT-5.4. Adds tests/test_gpt54_55_cris_prefix_scope.py, which reads the catalog, the shared/Azure schemas, the guardrails guide, and the three consumer files and fails on any window that pairs GPT-5.4/5.5 with a grouped us./in./global. grant without scoping in. to GPT-5.6. Verified to fail against the previous wording of all three files.
…and Luna only, not Sol
The GPT-5.6 Sol card (read 2026-11-07) publishes us.openai.gpt-5.6-sol and
global.openai.gpt-5.6-sol only; Terra and Luna publish us./in./us-gov./global.
Prior wording granted in. to "GPT-5.6" family-wide in the catalog strength,
the rewriter, the GCP Design and Azure Generate checklists, both schema
copies, and ai-migration-guardrails.md ("GPT-5.6 Sol/Terra/Luna
(us./in./global.)"). Each now reads: us./in./global. for GPT-5.6 Terra and
Luna; us./global. only for GPT-5.6 Sol, GPT-6, GPT-5.5 and GPT-5.4.
tests/test_gpt54_55_cris_prefix_scope.py now also scans the catalog strength
text and rejects any grouped in. grant to GPT-5.6 that does not name
Terra/Luna and place Sol in the us./global.-only set. Verified to fail
against the dfd07bd wording of every file above.
…-project IAM grant, Converse for application profiles, In-Region-only pricing Three inline findings by leon1418 on 6b873f8, each verified against the public GPT-5.4 / GPT-5.5 model cards (read 2026-11-07): https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-54.html https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-55.html 1. IAM default-project grant (iam_policy.py:29). The cards require bedrock:InvokeModel on arn:aws:bedrock:{region}:{account-id}:project/default in addition to the inference profile. iam_policy.py gated that grant on is_gpt6_cris_model(). Added is_gpt54_55_cris_model() (exact us./global. openai.gpt-5.4 / -5.5 short ids and their inference-profile ARNs; bare mantle ids, GPT-5.6 CRIS ids and unrelated models do not match) and requires_default_project_grant(); generate_policy() now emits the third ARN for those ids. Updated the GPT-6-only sentence in gcp generate-ai.md, gcp/azure generate-artifacts-ai.md Step 3, and the report generator §6.4. 2. Application inference profiles are Converse-only (gcp design-ai.md:425). GCP and Azure Design now require required_client_api: "converse" when application inference profiles are the driving need on GPT-5.5 / GPT-5.4 (and any model whose card lists them Converse-only), forbid leaving the field absent, keep the responses default for a region-only move, and record a conflicting Responses-only need (structured outputs, implicit caching) as an explicit decision instead of a silent drop. Added checklist items to the GCP Validation Checklist and the Azure shared schema (required_client_api section and checklist). 3. Unpublished Geo/Global prices (ai-openai-to-bedrock.md:46). The cards publish commercial In-Region prices only. Scoped the verified ~10% premium to the In-Region option in the canonical guide (Tier 0 rows included), both openai-on-bedrock.md fact bases, the report generator and the pricing-cache rows; a CRIS plan needs a separately sourced Geo/Global price or reports the comparison unavailable, and an In-Region proxy is an unverified planning assumption with no verified premium or savings percentage. bedrock_pricing.lookup() already returns available:false for the CRIS ids; pinned with a test (short ids and profile ARNs, no PriceList call). Tests: test_iam_policy.py TestGpt54Gpt55CrisProjectArn (short ids, profile ARNs, bare/GPT-5.6/Anthropic controls, mixed run); test_bedrock_pricing.py test_gpt54_55_cris_ids_are_unpriced_not_inferred_from_in_region; new tests/test_gpt54_55_runtime_review_round3.py text contracts for all three findings. tools/sync-vendored.py re-run (5 copies synced). Validation (this commit, in the worktree): - mise -C plugins/aws-startup-advisor run ci: exit 0 (test 1407 passed + 222 subtests; test:telemetry 311 passed; heroku-source-review 44/44; lint:md 0 errors; vendored-parity PASS 62 files; skill-refs PASS 1482) - llm-to-bedrock: uv run --project scripts python -m pytest: 453 passed (bare mise python3 shows 10 ModuleNotFoundError: httpx in test_astra_execution_contract.py — environment, not the change) - agent-advisor: python3 -m pytest skills/agent-advisor: 570 passed - tests/test_gpt54_55_cris_prefix_scope.py: still green - python3 tools/validate.py: All validations passed (exit 0) - tools/sync-vendored.py --check: PASS - mise run security (gitleaks): 1 pre-existing upstream false positive at f8d09fd (tests/test_bedrock_migration_assessment_discover.py, generic-api-key), unrelated to this PR; nothing new flagged.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
leon1418
requested changes
Nov 8, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed the full 100-file range at 9a9c60124c7d7ff2b5f1024e4d67f3af7ed80243 against 0a218f8cc13220013f72147e2e0fdbe350b34ae2. Eleven P2 correction items remain; overlapping reports are consolidated below and detailed inline.
| Item | Required correction |
|---|---|
| F1 | Preserve invocation logging across Mantle selection. |
| F2 | Allow CloudWatch metrics on Mantle. |
| F3 | Check Guardrails support per model and API. |
| F4 | Validate hard capabilities against the final candidate. |
| F5 | Scope structured-output capabilities to their API. |
| F6 | Keep application profiles off the runtime OpenAI APIs. |
| F7 | Serialize the selector in the migration injection. |
| F8 | Require a real profile matrix for runtime fallbacks. |
| F9 | Apply Astra combination checks to the new runtime APIs. |
| F10 | Build IAM targets from resolved model overrides. |
| F11 | Refresh Mantle Chat availability before changing models. |
Apply catalog fixes to both skills/shared/ai/models/openai-bedrock.json and its agent-advisor/references/vendored copy. Shared provider/schema guide changes must remain synchronized with the GCP and Azure vendored copies; the corresponding cloud Design/API guidance must agree. F1–F6/F8–F9/F11 need focused recommendation cases; F7 needs a persisted handoff round trip; F10 needs recovery-to-IAM coverage that retains each row’s API. These are one consolidated correction list.
Local validation: advisor 719 passed; migration scripts 622 passed; plugin contracts 1,431 passed plus 222 subtests; both TypeScript suites passed under Node 24 (57 tests); Markdown lint passed for 822 files; fixture, model, pricing, reference and vendored checks passed. Telemetry had one timeout with 310 passed; the isolated case passed on retry. Probes were offline against the exact source and current public AWS contracts; no live Bedrock account access or deployed behavior is claimed. GitHub CI is separate: the captured CodeQL check is failing and its existing bot threads remain open.
Requesting changes for the cases above. Approval remains for the user to submit.
8 of 13 tasks
GPT-5.4 and GPT-5.5 publish an In-Region price only, so a Geo or Global plan stays unpublished. Co-authored-by: Cursor <cursoragent@cursor.com>
herosjourney
force-pushed
the
fix/agent-advisor-runtime-default
branch
from
November 9, 2026 05:34
9a9c601 to
91827f9
Compare
leon1418
requested changes
Nov 9, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Follow-up at 91827f910085dd8ede0a67cbe2ff7a413a49cb4d; reviewed all 133 files in the PR diff and the changes since 9a9c60124c7d7ff2b5f1024e4d67f3af7ed80243.
Verified fixed: F1, F2, F5, F6, F7, F8, F9, F10 and F11. The original conversations will receive the verification evidence.
The consolidated remaining correction list is:
| Item | Correction |
|---|---|
| F3 | Complete the model-specific Guardrails API rule in Azure Design/schema and remaining migration guidance. The engine and GCP cases are fixed. |
| F4 | Validate selectors against the final provider/model/API in both modules and cloud Design flows. Web Search/background cannot simply preserve an incompatible Chat/Messages API. |
| F12 | Keep runtime stored-response flags from rejecting valid Mantle Responses plans. |
| F13 | Do not turn Anthropic client computer/editor/bash tools into Mantle-only server-tool requirements. |
| F14 | Enforce hard caching requirements beyond Astra rather than dropping them. |
| F15 | Carry Web Search and Fetch authorization/configuration through setup and generated IAM. |
F12 and F13 were introduced by the changes. F14 and F15 existed at the earlier reviewed head, and I missed them in that review; they are not regressions caused by these fixes. F3/F4 retain their existing identities and will be followed up in their original threads. Canonical catalog, policy and provider/schema-guide changes must stay synchronized with their actual vendored consumers; the affected cloud and report paths are named inline.
Local validation: 789 advisor tests, 655 migration tests, 1,496 plugin tests plus 222 subtests, 311 telemetry tests and 57 TypeScript tests passed. Markdown lint passed for 822 files; fixture, model, pricing, reference, vendored and consent checks passed. Independent probes used the exact original source and current public AWS contracts. No live AWS account/deployment behavior is claimed. GitHub CI is separately green in the current snapshot.
Requesting changes for the six items above. Resolving verified-fixed conversations does not dismiss a blocking review or submit approval.
…policy and make Runtime the primary same-model OpenAI path in design guidance Base: 6b873f8 (PR 1 aws#583 tip). Public basis: Amazon Bedrock User Guide endpoints.html, models-endpoint-availability.html, inference-responses-api.html, inference-chat-completions.html, inference-messages-api.html, tool-use-server-side.html, web-search.html, projects.html, workspaces.html, cost-mgmt-application-inference-profiles.html, model-invocation-logging.html, monitoring-mantle-metrics.html, quotas-runtime.html, quotas-mantle.html, scaling-throughput-best-practices.html (all read 2026-11-07). - New canonical skills/shared/ai/bedrock-endpoint-policy.md (rule, mantle_selector enum, public matrix, lifecycle and quota wording), vendored into gcp-to-aws and azure-to-aws (README rows + sync). - ai-openai-to-bedrock.md / ai-openrouter-to-bedrock.md: runtime_openai_cris with the source app's required_client_api is the Tier 0 default; mantle_* only with a mantle_selector; Runtime steps shown first; "smallest change" phrasing removed. - Three schema-design-aws-ai.md copies: runtime_messages, messages, same_model_runtime, mantle_selector vocabulary; Runtime default stated. - gcp/azure design-ai.md, azure design.md/generate.md _assert, gcp/azure generate-artifacts-ai.md, gcp generate-ai.md, gcp estimate.md, azure estimate-assemble.md, both openai-on-bedrock.md Client setup: Runtime first and labelled default, Mantle selector-gated, same_model_runtime route added. - Fixtures: mantle_selector on the two mantle_openai_responses rows. - Tests: new tests/test_endpoint_policy_contract.py; extended test_generate_mantle_dispatch.py, test_ai_only_option_c_framework_route.py, test_gpt6_review_round2.py, test_gpt6_round4_findings.py. Verification (from plugins/aws-startup-advisor): - python3 tools/sync-vendored.py --check -> PASS, 64 files - uv run --no-project --with pytest python -m pytest tests -q -> 1406 passed (base 6b873f8: 1398) - uv run --project skills/llm-to-bedrock/scripts --with pytest python -m pytest skills/llm-to-bedrock/scripts -q -> 418 passed (unchanged) - uv run --project skills/agent-advisor/scripts --with pytest python -m pytest skills/agent-advisor/scripts -q -> 570 passed (unchanged) - mise run lint:md (0 errors / 823 files), lint:skill-refs (1508 refs, 0 missing), lint:artifacts (99/0), lint:vendored-parity (64), lint:model-id -> PASS - grep -rn -i -E 'primary path|smallest (possible )?change|remains the default| Mantle is the PRIMARY' skills | grep -v vendored -> every Mantle-related hit gone; remaining hits are pre-existing, unrelated to Bedrock endpoints (design-ref-harness.md:29, heroku-to-aws, gcp elastic-beanstalk.md, prompt-library bedrock-quota-manager.md, agent-advisor test_unit_grouping.py). agents/llm2bedrock-code-rewriter.md:260 is left for the execution-lane change.
…drock-runtime Messages endpoint in the shared guide and design/generate phases The canonical ai-anthropic-to-bedrock.md now keeps anthropic.Anthropic and repoints it at https://bedrock-runtime.\{region\}.amazonaws.com/anthropic with a string Bedrock token minted per client (provide_token), records migration_path "runtime_messages" + required_client_api "messages", moves the boto3 Converse rewrite under an explicit-alternative section, and gates Mantle Messages behind a mantle_selector. gcp/azure design-ai.md Anthropic lines, gcp/azure generate-artifacts-ai.md Step 0/Step 1, the estimate route lists, skills/shared/ai/README.md, gcp SKILL.md and the behavior-delta Anthropic reference carry the same wording; vendored copies re-synced. Public basis: inference-messages-api.html and inference-profiles-prereq.html (read 2026-11-07). Verification (plugins/aws-startup-advisor): - python3 tools/sync-vendored.py --check: PASS, 64 files byte-identical - uv run --no-project --with pytest python -m pytest tests -q: 1407 passed, 222 subtests - uv run --project skills/llm-to-bedrock/scripts --with pytest python -m pytest skills/llm-to-bedrock/scripts -q: 418 passed - mise run lint:md: 823 files, 0 errors; mise run lint:skill-refs: 1514 checked, 0 missing; mise run lint:vendored-parity: PASS 64 - grep -rn 'Converse client swap' skills agents | grep -v vendored: no matches
…thout an API and add the Runtime Anthropic Messages lane to llm-to-bedrock normalize_api_path now raises for a runtime_openai_cris plan that recorded no source API instead of silently resolving to Converse; SKILL.md resolves that case with the user (resolved_runtime_api) and only an explicit required_client_api "converse" reaches the Converse rewrite. Adds the runtime_messages execution lane: create_messages_client / probe_runtime_messages_model (Anthropic SDK on bedrock-runtime /anthropic with a string token minted per client), --runtime-api messages, image_input.messages_message, the BedrockGetInferenceProfile IAM statement for CRIS targets, the rewriter's Runtime Anthropic Messages lane (Mantle express lane is now selector-gated, Converse rewrite is for explicit converse plans), evaluator §6b/§9.5b and the §10 messages branch, report-generator §6.4 and resolver notes, plus the endpoint policy section in SKILL.md. Pins anthropic>=0.75,<1 in the scripts project. Also un-indents the Converse snippet in the shared Anthropic guide so the agent-advisor snippet test executes the right block. Public sources (read 2026-11-07): https://docs.aws.amazon.com/bedrock/latest/userguide/inference-messages-api.html https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-prereq.html https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html Verification (plugins/aws-startup-advisor): - uv run --project skills/llm-to-bedrock/scripts --with pytest python -m pytest skills/llm-to-bedrock/scripts -q -> 431 passed (418 + 13 new), 0 failed - uv run --project skills/agent-advisor/scripts --with pytest python -m pytest skills/agent-advisor/scripts -q -> 570 passed; git diff --stat 6b873f8..HEAD -- skills/agent-advisor is empty - python3 tools/sync-vendored.py --check -> PASS 64 files - mise run ci -> exit 0 (lint:md 0 errors; lint:skill-refs 1517/0 missing; lint:vendored-parity 64; lint:artifacts 99/0; test 1408 passed + 222 subtests; test:telemetry 311; test:heroku-source-review 44) - root: python3 tools/validate.py -> All validations passed; mise run validate -> exit 0 - root: mise run build -> lint + validate pass; security fails only on the pre-existing gitleaks finding in upstream f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118); not from this change - gitleaks protect --staged -> no leaks found - residual Mantle-default grep over plugins/*.md|*.json -> no hits
… runtime_messages plan and align the Mantle Messages snippet with the public page Review iteration 1 findings (review.json CHANGES_REQUESTED): - major: preflight_bedrock.main() routed every non-Mantle id to the Messages probe when --runtime-api messages was set, so a runtime_messages plan whose $TARGET_MODELS also carried an embedding id (B1 collects every bedrock_models[].aws_model_id) failed preflight spuriously. Embedding ids now fall through to probe_model (InvokeModel) before the messages branch. New test_main_messages_plan_probes_embedding_ids_via_invoke_model covers a Claude profile + amazon.titan-embed-text-v2:0 list. - minor: agents/llm2bedrock-code-rewriter.md Mantle express lane Anthropic row and snippet used base_url .../anthropic/v1 with auth_token=; the SDK appends /v1/messages, so that resolved to /anthropic/v1/v1/messages. Now base_url .../anthropic with api_key=, matching the public Messages API page (inference-messages-api.html, read 2026-11-07) and the shared guide. - minor: evaluator §6b and §9.5b heredocs are now executed by test_runtime_messages_connectivity_snippet_uses_the_messages_client and test_runtime_messages_vision_snippet_sends_a_base64_source_block. - minor: preflight_bedrock.py module docstring now scopes in. to GPT-5.6 Terra and Luna, with GPT-5.6 Sol and GPT-6 Astra us./global. only. - minor: the mantle_selector enum paraphrase is removed from skills/shared/ai/ai-openai-to-bedrock.md and gcp design-ai.md; both now point at references/vendored/ai/bedrock-endpoint-policy.md. The llm-to-bedrock SKILL.md section stays as the one permitted restatement. Vendored copies re-synced (tools/sync-vendored.py). Verification (plugins/aws-startup-advisor, 2026-11-07): - sync-vendored --check: PASS 64 (2 re-synced) - llm-to-bedrock scripts: 435 passed (431 + 4 new) - agent-advisor scripts: 570 passed; no non-vendored agent-advisor change - mise run ci: exit 0 (lint:md 0 errors, lint:skill-refs 1520/0 missing, lint:vendored-parity PASS 64, lint:artifacts 99/0, test 1408 passed + 222 subtests, test:telemetry 311, test:heroku-source-review OK) - root python3 tools/validate.py: All validations passed; mise run validate: exit 0 - residual Mantle-default grep over plugins/aws-startup-advisor *.md/*.json: nothing - gitleaks protect --staged: no leaks found
…ject grant by id form/API, per-row preflight APIs, mantle_messages lane gate, resolved_runtime_api in resume context Re-checked each leon1418 finding against 6ca403b (post-rebase onto 8c37ef0): 1. Default-project grant (ai-openai-to-bedrock.md:339). Partly satisfied by the rebase: 8c37ef0 grants project/default for GPT-6 and the GPT-5.4/5.5 CRIS ids, but GPT-5.6 Sol/Terra/Luna were still excluded and the grant was model-family-based. The public Responses API page (https://docs.aws.amazon.com/bedrock/latest/userguide/inference-responses-api.html, "Permissions", read 2026-11-07) authorizes bedrock:InvokeModel on the default project for every Responses request on bedrock-runtime, and the GPT-5.6 Sol/Terra/Luna cards (read 2026-11-07) repeat it for their runtime CRIS ids (Chat tab included). iam_policy.py now classifies by id form (is_runtime_openai_cris_model: us./in./us-gov./global. openai.gpt-* ids and their inference-profile ARNs; _GEO_PREFIX gains in. and us-gov.) and decides the grant by the selected API (--runtime-api responses|chat_completions emit it, converse/messages do not, id-only keeps it). GCP generate-ai.md / generate-artifacts-ai.md, Azure generate-artifacts-ai.md, both openai-on-bedrock.md fact bases, the shared guide (+vendored copies) and the report-generator §6.4 recipe now state the rule and cite the page. 2. Mixed-model preflight (preflight_bedrock.py:417). Confirmed open: one --runtime-api was applied to every id. preflight gains --target ID=API pairs (per-row required_client_api; invoke for embeddings) and fails closed with reason unknown_runtime_api before any probe when a row's API is missing or unknown; --models/--runtime-api stays as the legacy single-API form. B1/B4 pass each row as its own pair. 3. mantle_messages stop (SKILL.md:936). Confirmed open. New lane gate `preflight_bedrock.py --check-lanes aws-design-ai.json` (unsupported_lane_stop) halts before B2/B4/Phase C with a structured unsupported_lane result naming the deferred Mantle Messages lane and the workspaces selector; iam_policy --migration-path mantle_messages refuses to emit runtime permissions for the bare Claude id; the context rule no longer labels it converse. GCP generate-artifacts-ai.md Step 0: the broad Mantle branch is scoped to the three OpenAI-SDK spellings so mantle_messages matches exactly one branch and runtime_messages keeps Messages. The lane is not implemented. 4. resolved_runtime_api (SKILL.md:913). Confirmed open. The field is now in the C0 current-context field list (always present), resolved in B4 before preflight (read back from run-context.json on resume), and has an invalidation row (EVAL, REWRITE, REPORT archived; ANALYSIS, INGESTION kept). Rebase-review non-blocking items: reworded the "keeps the `responses` default" checklist sentences (gcp design-ai.md, azure shared schema-design-aws-ai.md) to the source-API default rule; removed the EOF blank line in tests/test_ai_only_option_c_framework_route.py. Tests: test_iam_policy.py (id-form x API matrix incl. in./us-gov., Converse-only and Anthropic/oss controls, CLI flags, mantle_messages refusal), test_preflight_bedrock.py (offline fake SDK clients: Claude/Messages + GPT/Chat, Claude/Messages + Nova/Converse + embeddings, fail-closed unknown API, --check-lanes), test_validate_result.py (actual comparator: unchanged resume matches, dropped field reproduces the reviewer's MISMATCH, changed API named), tests/test_runtime_default_review_round1.py (text contracts + comparator). Existing GPT-5.6 "no grant" assertions in test_iam_policy.py were flipped to match the 2026-11-07 card read. Validation (worktree fix-llm-to-bedrock-runtime-default): - mise -C plugins/aws-startup-advisor run ci: EXIT 0 (lint:md 0 errors; vendored-parity 64 PASS; skill-refs 1521 PASS; artifacts 99 PASS; test 1429 passed + 222 subtests; test:telemetry 311 passed; heroku 44 pass) - llm-to-bedrock scripts pytest: 599 passed - agent-advisor scripts pytest: 570 passed - python3 tools/validate.py: All validations passed - tools/sync-vendored.py --check: PASS 64 byte-identical - git diff --check: clean - mise run build (gitleaks): fails ONLY on pre-existing upstream f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118 generic-api-key, leaks found: 1) — not introduced by this branch
…s[] row in iam_policy (--target ID=API) so a GPT secondary under a Claude Messages primary keeps it Review-gate finding B1 (pr585-review-round-1): the §6.4 report-generator recipe passed the primary's single `Target API surface` as --runtime-api for every id, and generate_policy applied it to all ids, so a runtime_messages plan with a GPT Chat/Responses secondary emitted no arn:...:project/default for that secondary (a regression against 6ca403b's id-only call). - iam_policy.py: new repeatable `--target <aws_model_id>=<required_client_api>` form (parse_targets, fails closed with reason unknown_runtime_api before any policy is built) and a `target_apis` kwarg on generate_policy; each row's own API decides requires_default_project_grant for that id, overriding the plan-wide runtime_api. `--models --runtime-api` stays as the legacy single-API form; --models/--target are mutually exclusive; the mantle_messages lane stop still wins over every other check. - llm2bedrock-report-generator.md §6.4: recipe passes one --target pair per bedrock_models[] row read from aws-design-ai.json (primary row = Target API surface; `invoke` for embedding rows), explicitly forbids applying the primary's surface to other rows, reports unknown_runtime_api verbatim, and documents the legacy form as single-row only. - Shared guide IAM step, GCP generate-ai.md / generate-artifacts-ai.md and Azure generate-artifacts-ai.md recipes name the per-row form; both vendored copies of the shared guide re-synced (byte-identical). - Tests: test_iam_policy.py TestPerRowTargetApis (reviewer's spot-check as the regression; Claude/messages + GPT/chat -> project/default exactly once; Claude/messages + Nova/converse + embeddings/invoke -> absent; GPT chat row beside GPT converse row -> once; row API overrides plan-wide; fail-closed parse; CLI --target grant / unknown API exit 1 / mutual exclusion). tests/test_runtime_default_review_round1.py: text contract pinning the §6.4 recipe (--target per row, no --runtime-api in the command, per-row warning, unknown_runtime_api) and the four setup recipes; mixed plan driven through parse_targets + generate_policy. Validation (run in this worktree, mise python 3.12): - mise -C plugins/aws-startup-advisor run ci: exit 0 -- test 1431 passed + 222 subtests; test:telemetry 311 passed; vendored parity 64 PASS - uv run --project skills/llm-to-bedrock/scripts pytest skills/llm-to-bedrock: 614 passed (bare mise python without the uv project: 603 passed, 11 failed on ModuleNotFoundError httpx in test_astra_execution_contract -- environment, not code) - pytest skills/agent-advisor: 570 passed (no agent-advisor files touched) - pytest tests/test_gpt54_55_cris_prefix_scope.py: 3 passed - python3 tools/validate.py: All validations passed - tools/sync-vendored.py --check: PASS, 64 byte-identical - git diff --check: clean - mise run build: gitleaks fails only on pre-existing upstream commit f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118 generic-api-key); unrelated to this branch
… one Bedrock id sits in several --target rows PR 585 review round 1, pass 2, blocker B2: `target_apis = dict(targets)` kept only the LAST row's API per id, so `--target us.openai.gpt-5.6-sol=chat_completions --target us.openai.gpt-5.6-sol=converse` emitted 0 project/default while the reverse order emitted 1. The §6.4 recipe emits one pair per bedrock_models[] row without deduplicating, and the schema lets a GPT CRIS id appear on a converse row (Guardrails / application-inference-profile call site) and a chat_completions/responses row for a second source model. - iam_policy.py: new pure `target_apis_by_id(pairs) -> dict[str, set[str]]` groups every API an id is called on; `generate_policy(target_apis=)` accepts a str or a set per id, validates each API in the set (fail closed on an unknown one), and grants `project/default` once when ANY of the id's rows is responses/chat_completions. CLI `--target` now builds the set instead of `dict(targets)`. - test_iam_policy.py::TestPerRowTargetApis: [gpt=chat,gpt=converse] and the reverse order and [gpt=converse,gpt=responses] all grant exactly once; [gpt=converse, gpt=converse] grants none; target_apis_by_id keeps every row's API; an unknown API inside a row set fails closed; CLI duplicate-id rows grant once in both orders (the reviewer's spot-check). - llm2bedrock-report-generator.md §6.4: tell the agent not to deduplicate rows and that a duplicated id is granted once if any row is responses/chat_completions. Validation (run in this worktree, 2026-11-07): - mise -C plugins/aws-startup-advisor run ci: exit 0 — [test] 1431 passed, 222 subtests passed; [test:telemetry] 311 passed; lint:md 0 errors; vendored-parity PASS 64; artifacts PASS 99; skill-refs PASS 1521. (A stale gitignored skills/llm-to-bedrock/scripts/.venv from the previous step broke lint:md on the first run; it was removed, not committed.) - pytest skills/llm-to-bedrock (uv project env): 622 passed (was 614; +8 new). - pytest skills/agent-advisor: 570 passed (no agent-advisor files touched). - pytest tests/test_gpt54_55_cris_prefix_scope.py: 3 passed. - python3 tools/validate.py: All validations passed. - tools/sync-vendored.py --check: PASS, 64 byte-identical. - git diff --check: clean. - mise run build: exit 1 ONLY on gitleaks' pre-existing upstream finding at f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118, generic-api-key); not introduced by this branch. - CLI spot-check: sol=chat+converse -> 1 project/default; converse+chat -> 1; converse+converse -> 0.
…ds in the §6.4 IAM recipe, project-scoped GetInvoke for retained stored-response ops, identity-checked saved-API reuse in B4 Three P2 findings from leon1418 on f3c0155 (threads 4505069045, 4505069047, 4505069050). 1. Report §6.4 IAM recipe reloaded aws-design-ai.json's ORIGINAL ids, so a C4 target change (Sol → Terra, recorded in resolved_model_overrides) emitted Sol permissions and omitted the Terra profile. The recipe now joins each bedrock_models[] row's required_client_api to the validated analysis target_models (overrides applied) and never falls back to the row's aws_model_id; iam_policy.validated_target_pairs() does the join and fails closed on a row without a validated entry or an API. Regression: a C4 Sol → Terra change yields a C6 policy naming Terra (+ project/default) and no Sol resource; the pre-fix design-row path is kept as the failing control. 2. Retained client.responses.retrieve(id) / cancel / delete on bedrock-runtime authorize bedrock:GetInvoke / CancelInvoke / DeleteInvoke on project/default, separately from the InvokeModel creation needs (inference-responses-api.html "Permissions", read 2026-11-08). analysis.json gains optional stored_response_ops (enum retrieve|cancel|delete); the analyzer §8.3 scans for the call sites; iam_policy.py gains --stored-response-ops and emits a BedrockStoredResponseOps statement with ONLY the retained actions on project/default, and only when a --target row is a runtime GPT CRIS id on `responses` — Converse, Anthropic Messages, Chat Completions, embeddings, bare mantle ids and id-only legacy calls are refused (reason stored_response_ops_no_responses_row). The report recipe passes the field through, the How to Apply checklist gains a create-plus-retrieve permission check, and the GCP / Azure setup_bedrock contracts, GCP generate-ai and the shared OpenAI guide (vendored copies re-synced) name the grant, its controls and the create-plus-retrieve verification. 3. SKILL.md B4 read the saved resolved_runtime_api before C0's assess_design_sha256 check, so a cached `messages` answer could send a new GPT row to the Anthropic endpoint. B4 now runs preflight_bedrock.py --resume-runtime-api <run-context> --design <aws-design-ai.json>, which reuses the saved answer only when it exists AND the saved assess_design_sha256 and target_models (source, aws_model_id) identity match the current design; any mismatch, missing context or missing field forces re-resolution before any probe. The context-block recovery sentence points at the same gate. Validation (worktree fix-llm-to-bedrock-runtime-default): - mise -C plugins/aws-startup-advisor run ci: exit 0 — test 1441 passed (+222 subtests), test:telemetry 311 passed, lint:md 0 errors, vendored-parity PASS (64), skill-refs PASS, model-id OK, artifacts PASS, consent-disclosure OK, model-catalog-drift OK (3 pre-existing warns) - llm-to-bedrock scripts pytest (uv run --project .): 655 passed (622 baseline + 33 new) - agent-advisor scripts pytest: 570 passed (unchanged; no agent-advisor edits) - tests/test_gpt54_55_cris_prefix_scope.py and test_runtime_default_review_round1.py green - python3 tools/validate.py: All validations passed - tools/sync-vendored.py --check: PASS - git diff --check: clean - mise run build: exit 1 ONLY on the pre-existing gitleaks finding at upstream f8d09fd (tests/test_bedrock_migration_assessment_discover.py:118, generic-api-key on the string "claude-3.5-sonnet"); lint:manifests passed. Not introduced or touched by this commit.
…nd record them as mantle_signals Discover now records the five code-detectable bedrock-mantle selectors (background_inference, web_search, server_side_tools, projects, workspaces) with file:line evidence so Clarify can confirm them instead of guessing the endpoint from source code alone. mantle_only_model_or_api stays Design-derived and in_region_processing is never detected (always asked). - skills/shared/ai/bedrock-endpoint-policy.md: new "## How a selector is learned" section (positive/negative constructs per selector, cites inference-responses-api.html read 2026-11-07); vendored copies re-synced (gcp-to-aws, azure-to-aws references/vendored/ai). - gcp-to-aws discover-app-code.md: Step 5C "Mantle-Only Selector Signals" (table + workloads[].mantle_signals[] output), Step 5B example, checklist. - azure-to-aws discover-app-code.md: Step 5C (bullet style), checklist. - gcp/azure schema-discover-ai.md: workloads[].mantle_signals[] shape, CRITICAL field bullet / enum, validation rule. - discover-openai-api.md / discover-anthropic-api.md / discover-openrouter-api.md (gcp + azure): Step 4 merge adds a projects / workspaces / web_search (:online) signal with the <usage_api>/line 0 sentinel; OpenRouter workspace_id is explicitly NOT a workspaces signal. - bedrock-migration-assessment discover-app-code.md: Step 4.5 with top-level mantle_signals[] output; Step 5 example and closing rule allow the key. - scripts/contracts/ai-workload-profile.schema.json (+ byte-identical BMA copy): optional root mantle_signals[] and mantleSignal definition. - validate-ai-profile.py: _check_mantle_signals (unmatched_mantle_signal) and mantle_signals_detected in the summary. - fixtures/bma-scan-openai-mantle-signals: positives on app/llm.py:5,9,14,21; negatives (function tool named web_search, store=True) on lines 30 and 37. - tests/test_mantle_selector_discovery.py (15 tests); test_bedrock_migration_assessment_core.py summary key set updated. Verification (plugins/aws-startup-advisor): - python3 tools/sync-vendored.py && --check: PASS (64 files, 2 synced) - python3 scripts/validate-artifacts.py --fixtures: PASS, 100 artifacts, 0 errors, no new baseline entries - python3 tools/check-skill-refs.py --check: PASS (1525 refs) - markdownlint-cli2 (823 tracked files): 0 errors - uv run --no-project --with pytest python -m pytest tests -q: 1456 passed, 222 subtests passed, 1 failed — the one failure is test_model_id_lint::test_real_plugin_passes flagging a retired model id inside the gitignored, untracked skills/llm-to-bedrock/scripts/.venv/ (anthropic SDK constants); pre-existing local environment, not a tracked file. Acceptance subset (6 files): 253 passed. - python3 tools/validate.py (repo root): All validations passed - git diff --check: clean
…22b / Q10.5) and persist ai_constraints.mantle_requirements PR aws#585 defined the mantle_selector enum but nothing populated it from the user. Add one question block with four sub-questions (asynchronous/background inference; provider-run server-side or pre-configured tools incl. web search; customer-created Projects/Workspaces; strict single-Region processing + Region): - gcp-to-aws clarify-ai.md: Q22b (ESSENTIAL, Gate-2 AI batch, prefilled from workloads[].mantle_signals), Mantle-only signals line in the AI Context Summary, Preferences Output block, question budget 4 -> 5 globals - gcp-to-aws clarify-assemble.md: Q22b in the AI/agentic batch, its "use defaults" bullet and defaults-table row - gcp-to-aws / bedrock-migration-assessment clarify-ai-only.md: Q10.5 between Q10 and Q11 with identical table rows and Interpret line (BMA parity); BMA prefills from the validator summary's mantle_signals_detected; gcp fast path now presents Q10.5 (never dropped); Step 3 field row mantle_requirements - azure-to-aws clarify-ai.md: Q22b Category F bullet, Step 3 rows entry, Who-consumes row; azure clarify-ai-only.md: Q10.5 paragraph, fast path, Step 3 bullet, AI-only example block entry (shows web_search + region) - schema-preferences.md (gcp + BMA): mantle_requirements catalog rows (Q22b (AI-only: Q10.5) / ai_constraints Q10.5) - fixtures/bma-ai-only-clarify/preferences.json: mantle_requirements answer and Q10.5 in questions_asked - tests/test_mantle_requirements_clarify_contract.py: 13 new contract tests Wrapper: ai_constraints.mantle_requirements.value = [{ workload_id, selector, source, evidence?, region? }], selector in background_inference | server_side_tools | web_search | projects | workspaces | in_region_processing | none, source in detected | user | none; chosen_by user or default (source default:<Qid> + report caveat), never extracted. Verification (plugins/aws-startup-advisor): - pytest tests/test_mantle_requirements_clarify_contract.py tests/test_bedrock_migration_assessment_core.py tests/test_validate_artifacts.py tests/test_azure_clarify_fast_path_contracts.py tests/test_estimate_clarify_volume_resolution.py tests/test_live_cli_ai_metrics.py: 243 passed - pytest tests (full plugin suite): 1475 passed, 222 subtests, 1 failed = tests/test_model_id_lint.py::test_real_plugin_passes flagging the untracked skills/llm-to-bedrock/scripts/.venv (pre-existing local env, not a tracked file) - validate-artifacts.py --fixtures: PASS 100 artifacts, 0 errors, baseline unchanged - validate-artifacts.py --run-dir fixtures/bma-ai-only-clarify --skill bedrock-migration-assessment --no-baseline: PASS - markdownlint-cli2: 0 errors on 823 files; sync-vendored --check PASS (64); check-skill-refs --check PASS (1525); tools/validate.py passed; git diff --check clean
….mantle_requirements in Design and disclose the bedrock-runtime assumption
Design no longer infers the Bedrock endpoint from the source API, the model or a
guess: gcp-to-aws and azure-to-aws design-ai.md gain Step 0.8, which reads
preferences.json -> ai_constraints.mantle_requirements (Clarify Q22b / Q10.5),
derives bedrock_models[].mantle_selector only from a detected|user entry (plus
the catalog for mantle_only_model_or_api), records the cited entry as
bedrock_models[].mantle_requirement on every mantle_* row, and writes the
verbatim endpoint_assumption sentence ("No Mantle-only requirement was recorded
for <workload_id>; bedrock-runtime selected. ... re-run Clarify.") on every
runtime row. Generate echoes whichever field is present in the script/adapter
header; the llm-to-bedrock report renders an Endpoint line from a new
"Endpoint basis:" context line and cites the public Responses API page for the
Runtime background=true 400.
Design: skills/gcp-to-aws/references/phases/design/design-ai.md (Step 0.8,
Part 5, Part 6 table, Validation Checklist, Present Summary item 6)
skills/azure-to-aws/references/phases/design/design-ai.md (Step 0.8,
Parts 3/5, Completion gate, Present summary)
Schema: skills/shared/ai/schema-design-aws-ai.md (bedrock_models[] bullet,
new "mantle_requirement and endpoint_assumption" section, 2 checklist
items) + vendored copies in gcp-to-aws/azure-to-aws references/vendored/ai
skills/gcp-to-aws/references/shared/schema-design-aws-ai.md
skills/azure-to-aws/references/shared/schema-design-aws-ai.md
Generate: skills/gcp-to-aws/references/phases/generate/generate-artifacts-ai.md
skills/gcp-to-aws/references/phases/generate/generate-ai.md
skills/azure-to-aws/references/phases/generate/generate-artifacts-ai.md
Report: skills/llm-to-bedrock/SKILL.md (Endpoint policy paragraph, B1 tuple,
Endpoint basis: context line)
agents/llm2bedrock-report-generator.md (Executive Summary Endpoint
line, Model Mapping notes, Limitations)
Fixtures: fixtures/aws-design-ai/multi-provider-direct (detected web_search
Mantle row + runtime endpoint_assumption), openrouter-gateway-multi-
upstream (in_region_processing mantle_requirement + 2 assumptions),
new runtime-default-no-requirement (runtime-only design)
Tests: tests/test_mantle_requirements_design_contract.py (17 tests)
Verification (all run on this commit's tree):
- python3 tools/sync-vendored.py && --check: PASS, 64 files (2 re-synced)
- python3 scripts/validate-artifacts.py --fixtures: PASS, 101 artifacts, 0 errors,
baseline unchanged (MISSING_REQUIRED=6, UNKNOWN_KEY=44)
- mise -C plugins/aws-startup-advisor run ci: exit 0 — lint:md 0 errors/823
files, lint:artifacts PASS, lint:vendored-parity PASS, lint:skill-refs PASS
(1532 refs), model-id/staleness/catalog-drift/pricing/consent PASS,
test 1496 passed + 222 subtests, test:telemetry 311 passed,
test:heroku-source-review PASS
- skills/llm-to-bedrock/scripts: 655 passed (unchanged)
- skills/agent-advisor/scripts: 570 passed (no file under agent-advisor changed)
- python3 tools/validate.py: all validations passed; git diff --check clean
…cover, Clarify and Design Cross-FEAT integration pass over a6f9ef1 (mantle_signals), 73835b1 (ai_constraints.mantle_requirements) and 922687e (mantle_requirement / endpoint_assumption): - endpoint_assumption sentence: the shared schema-design-aws-ai.md and the gcp design-ai.md Step 0.8 rendered the placeholder as `<workload_id>` in backticks while the azure docs, the aws-design-ai fixtures and tests/test_mantle_requirements_design_contract.py expect a plain id. The placeholder is now unquoted in both and the "where <workload_id> is" note says the id is written plain (no backticks or quotes). - gcp clarify-ai.md Q22b Preferences Output example: the `evidence` of the detected `projects` entry now copies the Step 5B/5C construct verbatim (src/ai/openai_client.py:9 default_headers={"OpenAI-Project": ...}) so the example matches the "copy the signal's file:line construct" rule. - gcp and azure discover-app-code.md output checklists: the "real line that contains the construct" rule now names the `"<usage_api>"` / `line: 0` sentinel that the usage-API Step 4 merges append, matching schema-discover-ai.md. - bedrock-endpoint-policy.md "How a selector is learned": names the Clarify question ids (Q22b / Q10.5) and the Design outputs (mantle_requirement / endpoint_assumption) so the policy describes the whole chain. - Vendored copies re-synced (gcp-to-aws and azure-to-aws references/vendored/ai/bedrock-endpoint-policy.md and schema-design-aws-ai.md). Verification (worktree fix-llm-to-bedrock-runtime-default): - mise -C plugins/aws-startup-advisor run ci: exit 0 (lint:md 0 errors, lint:artifacts PASS 101 artifacts 0 errors, lint:vendored-parity PASS 64, lint:skill-refs PASS 1532, model-id / staleness / catalog-drift / startup-model-pricing / consent-disclosure PASS, test 1496 passed + 222 subtests, test:telemetry 311 passed, test:heroku-source-review PASS). The gitignored local skills/*/scripts/.venv dirs were parked outside the tree for the run (markdownlint and model-id-lint walk into them) and restored. - python3 tools/sync-vendored.py --check: PASS 64 - python3 scripts/validate-artifacts.py --fixtures: PASS 101 artifacts, 0 errors - skills/llm-to-bedrock/scripts: 655 passed - skills/agent-advisor/scripts: 570 passed (no file under agent-advisor changed) - python3 tools/validate.py: All validations passed - git diff --check: clean - mise run build (gitleaks) not run: it fails on pre-existing upstream commit f8d09fd, unrelated to this branch.
… a source with no Mantle path; widen the Anthropic and report-fallback wording Review round (verdict.json CHANGES_REQUESTED, 1 major + 2 minor): - major — Step 0.8 (gcp and azure design-ai.md) and the derivation section of skills/shared/ai/schema-design-aws-ai.md now define the outcome for a matched ai_constraints.mantle_requirements entry on a workload whose source API has no Mantle path (Gemini / Vertex AI via google-genai, or a workload whose target is a cross-family direct / converse / gpt-oss row): the row keeps bedrock-runtime and the verbatim endpoint_assumption sentence, and discloses the unhonored selector with its source and evidence in rationale and capability_gaps[] (a regional_warnings[] entry for in_region_processing); a Mantle path is never invented and the answer is never dropped. The gcp and azure shadow schema-design-aws-ai.md docs carry the same clause; the Present Summary endpoint line (gcp item 6, azure summary) names the gap. The Clarify copy (gcp Q22b, gcp/BMA Q10.5 identical sentence, azure Q22b bullet and Q10.5 paragraph) tells the user up front that a "yes" for such a workload records a capability gap rather than an endpoint change. tests/test_mantle_requirements_design_contract.py now asserts the Step 0.8 slice names the no-Mantle-path case and capability_gaps. - minor — gcp design-ai.md Step 0 Anthropic bullet and Part 5 Anthropic row, and azure design-ai.md Part 5: "(a `workspaces` entry)" -> "(any matched entry, typically `workspaces`)", matching Step 0.8 and the scanners (Anthropic web_search_20250305 -> web_search, code_execution_* -> server_side_tools). - minor — llm-to-bedrock SKILL.md emits the C6 "Endpoint basis:" line whenever any bedrock_models[] row carries migration_path mantle_*, mantle_selector, mantle_requirement or endpoint_assumption, rendering a Mantle row without mantle_requirement as "bedrock-mantle (<selector or 'selector unrecorded'>; source unknown — plan predates mantle_requirements)"; B1 keeps migration_path / mantle_selector for that purpose. The report generator's Endpoint line renders that form and falls back to the runtime wording only when the line is absent (no row carries any marker). - Vendored copies re-synced (gcp-to-aws and azure-to-aws references/vendored/ai/schema-design-aws-ai.md). Verification (worktree fix-llm-to-bedrock-runtime-default): - mise -C plugins/aws-startup-advisor run ci: exit 0 (lint:md 0 errors, lint:artifacts PASS 101 artifacts 0 errors, lint:vendored-parity PASS 64, lint:skill-refs PASS 1532, model-id / staleness / catalog-drift / startup-model-pricing / consent-disclosure PASS, test 1496 passed + 222 subtests, test:telemetry 311 passed, test:heroku-source-review PASS). The gitignored local skills/*/scripts/.venv dirs were parked outside the tree for the run and restored. - python3 tools/sync-vendored.py --check: PASS 64 - python3 scripts/validate-artifacts.py --fixtures: PASS 101 artifacts, 0 errors - skills/llm-to-bedrock/scripts: 655 passed - skills/agent-advisor/scripts: 570 passed (no file under agent-advisor changed) - python3 tools/validate.py: All validations passed - git diff --check: clean - mise run build (gitleaks) not run: it fails on pre-existing upstream commit f8d09fd, unrelated to this branch.
…edrock endpoints The GPT-5.4 and GPT-5.5 cards publish an In-Region rate only, so a missing Geo or Global row stays unpublished. Co-authored-by: Cursor <cursoragent@cursor.com>
…s by endpoint and emits Web Search grants for a web_search row (PR 588 F12, F15)
F12 — `generate_policy` now splits `responses` rows by endpoint: a runtime GPT CRIS row
keeps `bedrock:GetInvoke` / `CancelInvoke` / `DeleteInvoke` on `project/default`; a bare
Mantle id on `responses` is covered by the `BedrockMantleInference` statement (Mantle
inference, stored-response retrieval at `GET /v1/responses/{id}` included, authorizes
`bedrock-mantle:CreateInference`) and no longer raises
`stored_response_ops_no_responses_row`. Converse / Messages / Chat / embedding / id-only
plans still fail closed. Public basis: inference-responses-api.html "Permissions",
read 2026-11-09.
F15 — new `--mantle-selector` (repeatable; `parse_mantle_selectors` fails closed with
reason `unknown_mantle_selector`) and `--external-web-access`. A `web_search` selector with
a Mantle row appends `BedrockWebSearch`: `bedrock-websearch:InvokeSearch` + `InvokeFetch`
on `*` with an `aws:RequestedRegion` condition (the service's only condition key; all three
actions take `*`), plus `bedrock-websearch:ExternalWebAccess` only when the user confirmed
external access. `web_search` without a Mantle row → `web_search_without_mantle_row`;
`--external-web-access` without `web_search` → `external_web_access_without_web_search`.
Other selectors add nothing. Public basis: web-search.html, security-web-search.html,
read 2026-11-09 (`external_web_access` defaults to true; without the action every Fetch
fails authorization while the request still returns 200).
Recipes: report §6.4 command block gains both flags; the stored-response paragraph keeps
its pinned sentences and adds the Mantle dispatch; new "Web Search needs its own grants"
paragraph; How-to-Apply gains the Mantle create-plus-retrieve variant and a conditional
"Tool-level permission check" bullet. GCP / Azure generate-artifacts-ai.md and the canonical
OpenAI guide (+ vendored copies via sync-vendored) carry the same facts and a Step-5
tool-level check.
Tests: test_iam_policy_round2.py — TestStoredResponseOps replaces the bare-Mantle raise test
with all-Mantle retained-retrieve (Mantle policy, no GetInvoke, legacy form too), runtime
control, mixed runtime+Mantle (GetInvoke once + Mantle sids), Mantle Chat-only still fails
closed, CLI all-Mantle exits 0; new TestWebSearchGrants (grant shape, external action,
parametrized absence, runtime-only raise, unknown selector, CLI emit / refuse / reasons).
New tests/test_runtime_default_review_round3.py pins the recipe strings.
Validation: skills/llm-to-bedrock/scripts `uv run --group dev pytest -q` 672 passed
(was 655); plugin `pytest tests -q` 1532 passed, 1 failed (known
test_model_id_lint::test_real_plugin_passes local .venv noise, pre-existing);
tools/sync-vendored.py --check PASS (64); markdownlint-cli2 0 issues.
…nt tools are not server_side_tools signals (PR 588 F13) The Sonnet 5 and Opus 5.5 model cards list `computer_20251124`, `computer_toolset_20260801`, `browser_toolset_20260801`, `bash_20250124`, `text_editor_20250728` and `memory_20250818` under "Computer use" and "Client tools and toolsets using `bedrock-runtime` and `bedrock-mantle` endpoints" (model-card-anthropic-claude-sonnet-5.html, model-card-anthropic-claude-opus-5-5.html, read 2026-11-09): they are client-executed and available on both endpoints, so they must not select Mantle. bedrock-endpoint-policy.md (canonical + 2 vendored mirrors via sync-vendored) and the gcp / azure / bedrock-migration-assessment discover-app-code.md scanners: the `server_side_tools` positive list keeps `"type": "mcp"`, `code_interpreter`, `file_search`, `computer_use_preview`, `image_generation` and Anthropic `code_execution_*` (provider-executed); `computer_*` / `text_editor_*` / `bash_*` (+ toolsets, `memory_*`) move to "Not a signal" with the card citation; both cards added to § Sources. New fixture fixtures/bma-scan-anthropic-client-tools (Anthropic SDK `claude-sonnet-5` with computer / text_editor / bash tools, profile with no `mantle_signals`). Tests (tests/test_mantle_selector_discovery.py): test_policy_and_scanners_keep_anthropic_client_tools_out_of_server_side_tools and test_client_tool_only_anthropic_fixture_has_no_mantle_signal_and_reaches_runtime_messages (validator ok, `mantle_signals_detected == []`, and iam_policy on `runtime_messages` emits `BedrockInvokeModelScoped` — the lane reaches IAM/preflight, not unsupported_lane). Validation: plugin `pytest tests -q` 1532 passed, 1 failed (known test_model_id_lint::test_real_plugin_passes .venv noise); scripts/validate-artifacts.py --fixtures PASS (103); tools/sync-vendored.py --check PASS (64).
herosjourney
force-pushed
the
fix/agent-advisor-runtime-default
branch
from
November 9, 2026 19:14
91827f9 to
2697099
Compare
leon1418
requested changes
Nov 9, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Follow-up at 26970991a3e8f41892d91bad872bf3234a6e3abd: OCR plus the parent repository review and one isolated independent reviewer accounted for all 139 changed files and relevant unchanged consumers.
The original F3, F12, F13 and F14 reproduction cases are corrected. F4 and F15 remain partially fixed. The consolidated correction list is:
| Item | Required correction |
|---|---|
| F4 | Validate required/detected Web Search even when projects or server_side_tools is the primary selector; incompatible Chat must require a decision. |
| F15 | Carry secondary Web Search requirements through Design into setup/report permissions and tool checks. Background + Web Search currently loses the search grants. |
| F16 | Add the documented caching evidence for native Claude fallback candidates. The new filter rejects a supported GPT-4o → Claude Converse migration requiring caching. |
| F17 | Require the documented Responses path for OpenAI server-side Lambda/MCP execution, including cloud Design/schema guidance and tests. |
F16 was introduced by the caching fix. F17 already existed at the previous reviewed head, and I missed it in that review. It is not a regression caused by these fixes. F4/F15 retain their existing conversations. The F14 conversation will remain open for its linked F16 regression; only fully addressed conversations will be resolved.
Local validation: 800 advisor, 672 migration and 1,533 plugin tests plus 222 subtests passed; 57 TypeScript tests and Markdown lint (822 files) passed. Other fixture, model, pricing, reference, vendored and consent checks passed. Telemetry initially had one five-second CLI timeout with 310 passes; all ten related host-marker cases passed unchanged in the targeted rerun. No live AWS/model invocation is claimed. CI is separate: the Startup Advisor check passed; the latest branch-state snapshot was BEHIND.
Requesting changes for these four items. No approval or review dismissal is submitted.
The scan on main now writes schema 1.1 workloads, integration, and agents. Step 4.8 records mantle_signals on that profile instead of replacing the scan. Co-authored-by: Cursor <cursoragent@cursor.com>
… grants when another selector is primary A Chat path no longer hides web_search, background inference, or server-side tools behind projects. Claude Converse rows carry prompt-caching evidence, and IAM receives every confirmed selector. Co-authored-by: Cursor <cursoragent@cursor.com>
…1 profiles The 1.1 scan contract requires workloads and integration. These two fixtures still said 1.0, so the fixture sweep failed after the main merge. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Fixed the open threads on
301 recommendation, catalog, and recipe tests passed locally. |
This was referenced Nov 10, 2026
5 of 9 tasks
leon1418
previously approved these changes
Nov 10, 2026
… scan. main added workload, integration, and agent steps at 4.5-4.7. The Mantle selector scan moves to Step 4.8 so both behaviors stay. Co-authored-by: Cursor <cursoragent@cursor.com>
…r-runtime-default Conflict resolution keeps both sides' behavior: - anthropic_model_recommendation.py: upstream's _explicit_profile_matches helper (aws#600) plus this branch's selector-aware _decision_options signature. - test_model_recommendation.py / test_openai_model_recommendation.py: union of both branches' tests. The cost-priority Mantle test keeps this branch's workspaces selector with upstream's "Haiku 5.5 is not a Mantle option in us-east-1" assertion; upstream's GovCloud Mantle test gains the selector it needs now that Mantle is only reached through a verified selector. - Shared model catalogs: upstream's Haiku 5.5 source/disclaimer sentences merged into this branch's longer evidence strings; this branch's path rows (runtime_openai_*, guardrails/prompt-caching flags) kept where upstream only reformatted. Haiku 5.5 gets the rows this branch requires on every model: runtime_messages (anthropic catalog) and guardrails_supported / prompt_caching_supported on runtime_converse (openai catalog), both from the Haiku 5.5 model card's bedrock-runtime feature table (read 2026-11-10). - Two Haiku-filter tests now assert the runtime default selects Haiku 5.5 and the Mantle (selector) branch still lands on Sonnet 5. - test_astra_recommendation_probes_selected_api_in_supported_regions uses the surface-neutral mantle_only_model_or_api selector; server_side_tools is Responses-only and turns a Chat source into a decision (PR 588 F4). Also fixes the MD032 markdownlint error in bedrock-migration-assessment discover-app-code.md that failed both CI checks.
Keep Haiku 5.5 as the fast Converse tier from main and the GPT-5.4/5.5 runtime catalog re-read from this branch. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Problem, Fix, and Scope are filled in. A change type is selected and all four Startup Advisor author confirmations are complete. The author checklist check now passes. CI and reviewers still need to verify the change. |
Conflict resolution (plugins/aws-startup-advisor/skills/shared/ai/models/openai-bedrock.json, top-level metadata only; every model entry auto-merged): - source: union. Keeps main's Haiku 5.5 model-card evidence (read 2026-11-09) and the Opus 5.5 line, plus this branch's GPT-5.5/GPT-5.4 model-card re-read (2026-11-06, runtime CRIS ids, Chat Completions on mantle). Trailing sentence merged to "Other OpenAI and unchanged entries retain their evidence ...". - disclaimer: union. Main's Haiku 5.5 fast/low-cost Converse tier sentence first, then this branch's GPT-5.5/GPT-5.4 re-read note (runtime CRIS paths, Chat Completions on mantle, 1.05M context, long-context pricing), then the shared Opus 5.5 / Fable / Mythos / eu-west-2 text that both sides carried unchanged. - verified_at: main's 2026-11-09 (newer than the branch's 2026-09-24). - openai_gpt_5_5 / openai_gpt_5_4 entries: branch values (refreshed runtime metadata, pricing, inference_profiles, region_availability, capabilities); main did not touch these entries. - anthropic_claude_haiku_5_5, Haiku 4.5 display_name/capability_source, and the pretty-printed GPT-6 Sol/Luna/6.1 Sol arrays: main values; the branch did not touch these. - Vendored copy regenerated with tools/sync-vendored.py (--check passes), not hand-edited. Verification on this merge: - cd plugins/aws-startup-advisor && mise run ci: all tasks finished (lint:md 0 errors, lint:model-id, lint:model-staleness, lint:model-catalog-drift, lint:startup-model-pricing, lint:artifacts PASS 104 artifacts, lint:vendored-parity PASS 62 files, lint:skill-refs PASS 1498 refs, pytest 1489 passed + 225 subtests, telemetry 311 passed). - skills/agent-advisor/scripts pytest: 597 passed. - skills/llm-to-bedrock/scripts pytest: 453 passed (run via uv with httpx/openai/ aws-bedrock-token-generator/jsonschema/boto3 available).
leon1418
previously approved these changes
Nov 11, 2026
5 of 9 tasks
Catalog changes re-homed in bedrock-model-registry.json and re-rendered (build-model-catalogs.py, sync-vendored.py); selectors, schema and tests merged to keep aws#567/aws#600/aws#607 and this PR's runtime Responses/Chat/Messages behavior. Verified: mise run ci (plugins/aws-startup-advisor) green, 1857 passed.
added 4 commits
November 11, 2026 18:04
Port GPT-5.4/5.5 catalog edits into bedrock-model-registry.json and re-render openai-bedrock.json via tools/build-model-catalogs.py, re-sync vendored copies. Add optional pricing object to registry/catalog schemas and renderer. Verified: mise run ci (plugins/aws-startup-advisor) passed, 1731 tests.
…un ci green (1857 passed); build-model-catalogs/sync-vendored --check pass
Registry/schemas hand-merged, catalogs re-rendered, vendored copies re-synced. Verified: build-model-catalogs --check and sync-vendored --check pass; mise run ci green (1857 passed).
Keep aws#588's _decision_options (runtime paths, mantle_selector) and apply aws#620's yield-then-pin selection to each option's candidates. Adapt aws#620's Mantle/default-path tests to the bedrock-runtime default (Mantle via mantle_selector; Messages source lands on runtime_messages) and add a runtime_messages option test. Verified: mise run ci in plugins/aws-startup-advisor passed (1857 tests).
Contributor
Author
|
This branch now contains the updated #583 and #585 and current main, including #620 (head |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
agent-advisorrecommends a Bedrock endpoint and model per AI workload. Its two recommendation engines (anthropic_model_recommendation.py,openai_model_recommendation.py) were written when the only choices were bedrock-mantle (OpenAI-SDK Responses / Chat Completions, Anthropic Messages) or bedrock-runtime Converse / Invoke. That binary turned API continuity into a Mantle selector: an OpenAI Responses or Chat Completions source, or an Anthropic Messages source, landed on Mantle just to keep its request shape, and any governance requirement (Guardrails, invocation logging) was then shown to the user as a governance-vs-continuity conflict. The public model cards list Responses, Chat Completions and Messages on bedrock-runtime for these models, so that default is wrong, and it contradicts the Runtime-default rule #585 added inskills/shared/ai/bedrock-endpoint-policy.md.Specific defects this PR fixes:
_HOSTED_TOOL_IMPACTS["web_search"]said the built-in Web Search tool does not execute on Mantle and is passed through. The docs say the opposite: Web Search is a pre-configured server-side tool served only on bedrock-mantle. It is a reason to pick Mantle, not a Mantle limitation.runtime_conversestrength listed "invocation logging, and CloudWatch" as bedrock-runtime-only. Mantle publishes CloudWatch metrics underAWS/BedrockMantle; model invocation logging is the runtime-only part.model-recommend.mdas a deliberate consequence to be fixed "in the follow-up PR" and pinned it in five tests.gcp-to-awsdesign-ai.md had no translation from advisorapi_pathvalues to themigration_path/required_client_api/mantle_selectorvocabulary fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585 gavellm-to-bedrock.invocation_loggingandcloudwatchalone forced Converse, although every bedrock-runtime path satisfies them.Fix
Plain English: bedrock-runtime is now the default for every source API. The engine keeps the source API (Responses, Chat Completions or Messages) on the same model through the runtime
/openai/v1and/anthropicendpoints when the model card lists it. Mantle is chosen only when the user has a verified reason from the shared endpoint policy, recorded as amantle_selector.runtime_openai_responsesandruntime_openai_chatjoin the OpenAI catalog;runtime_messagesjoins the Anthropic catalog. Path definitions carry SDK, endpoint (https://bedrock-runtime.{region}.amazonaws.com/openai/v1,.../anthropic), IAM action andcris_inference_profileid strategy. Rows are added on everyopenai_gpt_*model in the catalog (including GPT-6.1 Terra) and on every Claude model in the Anthropic catalog (Opus 5.5, Sonnet 5.5, Sonnet 5, Haiku 5.5 and Haiku 4.5), each with the card URL and read date as evidence. Model ids on runtime paths are CRIS-prefixed per card;in.stays limited to GPT-5.6 Terra / Luna (tests/test_gpt54_55_cris_prefix_scope.pyis green). No existing row'savailableflag or evidence changed. The source of truth is nowskills/shared/ai/models/bedrock-model-registry.json(added by refactor(aws-startup-advisor): render model catalogs from one shared registry #581). The runtime path rows live there,skills/shared/ai/models/*-bedrock.jsonare rendered from it withtools/build-model-catalogs.py, and the vendored copies are re-synced.bedrock-endpoint-policy.md(vendored into agent-advisor and referenced, not restated). API continuity never selects Mantle on its own.requirements.mantle_selectortakes the policy's seven-value enum; the engines also deriveweb_search(detected or critical feature) andin_region_processing(data_residency: in_region_requiredon a runtime candidate that is CRIS-only in the region, with no governance pull). Every result emitsmantle_selector(a string onmantle_*paths,nullotherwise); the output schema requires it and rejects amantle_*path with a null selector. Guardrails and multi-model Converse still route toruntime_converse;invocation_logging/cloudwatchalone stay on the runtime OpenAI API or runtime Messages. Apreferred_api_pathofmantle_*without a selector fails closed with amantle_selector_requiredblock. Conflict presentation is unchanged in shape: selector plus Converse-only governance gives two options with trade-offs; continuity plus Converse-only governance now gives two runtime options and no Mantle option._HOSTED_TOOL_IMPACTS["web_search"]now states it is a Mantle-only pre-configured server-side tool (portable onmantle_*paths, rearchitecture on runtime paths), citestool-use-server-side.htmlandweb-search.html, and the test that pinned the old text is replaced._openai_candidate_orderreplaces_mantle_candidate_order: same model first, then the same GPT major family newest-first, then other OpenAI models newest-first. The five "deliberate consequence" tests from fix(aws-startup-advisor): refresh GPT-5.4/GPT-5.5 Bedrock endpoint metadata from the public model cards #583 are deleted and replaced with tests asserting the corrected walk, and the disclosure is removed frommodel-recommend.md.runtime_conversestrength now says Guardrails and model invocation logging are bedrock-runtime-only and CloudWatch metrics exist on both endpoints (AWS/Bedrock,AWS/BedrockMantle).verify_model_path.pybinds the existing Responses / Chat probes to a runtime client atbedrock-runtime.{region}.amazonaws.com/openai/v1and adds a runtime Messages probe against.../anthropicwith a Bedrock bearer token. New client-factory kwargs keep the probes injectable; default builders name the missing package whenopenai/anthropicare not installed. Application-profile pre-check and the fail-closed not-implemented fallback are unchanged.model-selection.md,model-recommend.md,confirm.md,design.md,poc.md,migration-plan.mdStep 3.5 andhandoff-migration.mdcarry the new paths andmantle_selector; the handoff gains a translation table (advisorapi_path→migration_path+required_client_api+mantle_selector) andmatches_advisor_modelgains branches for the runtime OpenAI paths,runtime_messagesandmantle_*with an equal selector. TheTODO(PR 3)line ingcp-to-awsdesign-ai.md becomes that translation paragraph; nothing else in that file changes.SKILL.mdgains one Files row; its frontmatter description is unchanged.Stacked on #583 and #585
This is PR 3 of 3 and still builds on #583 and #585, which are open. Until they merge, the Files tab also shows their commits. Main has been merged into this branch (head
57344faf), including #567, #581, #607, #619 and #620, with the conflicts resolved by re-rendering the catalogs from the registry. Review this PR's own change withgit diff <#585 tip>...57344faf. The earlier per-commit and per-line counts in this description were measured against an older base and have been removed.What this PR does NOT do
selection_statusfield. feat(aws-startup-advisor): add model-selection catalog contract and selection_status #567 added it; the new path rows follow its schema.llm-to-bedrockscripts,bedrock-endpoint-policy.mdor anyazure-to-awsfile. fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585 owns them; agent-advisor only gains a byte-identical vendored copy of the policy.availableflag. Seven places where the live cards have drifted from the catalog are listed under Limitations and left as they were.Scope
Skills:
agent-advisor(engines, verifier, schemas, phase docs, handoff, vendored copies, tests),shared/ai/models(the registry and the two catalogs rendered from it),gcp-to-aws(one paragraph indesign-ai.md), plugintests/test_endpoint_policy_contract.py.Files: the counts from the first revision no longer apply after merging main; use the Files tab.
skills/shared/ai/models/bedrock-model-registry.jsonplus the renderedopenai-bedrock.json,anthropic-bedrock.json: new path definitions and rows; A6 sentence;sourcestrings name the rows and read date.skills/agent-advisor/references/vendored/ai/models/*.json,vendored/ai/bedrock-endpoint-policy.md(new),vendored/README.md.skills/agent-advisor/scripts/anthropic_model_recommendation.py,openai_model_recommendation.py,verify_model_path.py.skills/agent-advisor/scripts/schemas/model-recommendation-input.json(preferred_api_pathenum,data_residency: in_region_required,requirements.mantle_selector),model-recommendation.json(apiPathenum, requiredmantle_selector, conditional onmantle_*),model-verification.json(apiPathenum).test_catalog_paths.py(new),test_runtime_default_docs.py(new),test_model_recommendation.pyandtest_openai_model_recommendation.py(rewritten for the runtime default),test_opus55_integration.py,test_verify_model_path.py,tests/test_endpoint_policy_contract.py.SKILL.md,decision-refs/model-selection.md,phases/model-recommend/model-recommend.md,phases/confirm/confirm.md,phases/design/design.md,phases/poc/poc.md,phases/migration-plan/migration-plan.md,handoff/handoff-migration.md,gcp-to-aws/references/phases/design/design-ai.md.User-visible artifact deltas:
confirm.jsonmodel_decisionanddesign.jsonmodel_recommendationgainmantle_selector(null on runtime paths);handoff-summary.mdgains per-unitapi_path,invocation_model_id,mantle_selectorand the translation table; Step 7 names--with anthropic --with aws-bedrock-token-generatorforruntime_messagesand--with openai --with aws-bedrock-token-generatorfor the four OpenAI-SDK paths.Not touched:
skills/llm-to-bedrock/**,skills/azure-to-aws/**,skills/bedrock-migration-assessment/**,skills/shared/ai/bedrock-endpoint-policy.md,plugin.json, marketplaces,.mcp.json,tests/test_gpt54_55_cris_prefix_scope.py.Validation
Current head
57344faf: after merging main, the fullcd plugins/aws-startup-advisor && mise run ciwas run and passed (1,857 tests). The per-gate figures below are from the first revision (2026-11-08, head9a9c6012, basef3c01556) and are kept as history; they do not match the current head (P=plugins/aws-startup-advisor):cd P/skills/agent-advisor/scripts && uv run --group dev pytest -q: 719 passed (570 at base; +32 catalog/schema/verifier, +27 Anthropic engine, +65 OpenAI engine and verifier, +25 doc locks). Re-run at the pushed tip immediately before opening this PR: 719 passed.cd P/skills/llm-to-bedrock/scripts && uv run --group dev pytest -q: 622 passed, unchanged from base.P/tests/test_gpt54_55_cris_prefix_scope.py+P/tests/test_endpoint_policy_contract.py: 11 passed.mise run test: 1,430 passed, 222 subtests passed, 1 failed (tests/test_model_id_lint.py::test_real_plugin_passes, local-venv noise, see below; 11 passed on a venv-free copy, so the tracked tree is 1,431 passed, matching base).lint:artifactsPASS (99 artifacts, 0 errors);lint:vendored-parityPASS;lint:skill-refsPASS (1,531 refs, 0 missing, 7 baselined);lint:model-stalenessOK;lint:model-catalog-driftOK with 3 pre-existing warnings (sonnet-5-5, haiku-5-5, gpt-6.1-terra absent by design, Shared Bedrock model catalog: follow-up sequence #566 stage 2);lint:startup-model-pricingOK;lint:consent-disclosureOK;lint:model-idOK (19 retired IDs enforced) on a venv-free copy.python3 tools/sync-vendored.py --check: PASS, 65 files byte-identical (64 at base).test:telemetry311 passed;test:heroku-source-reviewpass.markdownlint-cli2with#**/.venv/**: 824 files, 0 errors.python3 tools/validate.py: all validations passed.git diff --check: clean..venvdirectories thatuv runcreates underP/skills/agent-advisor/scriptsandP/skills/llm-to-bedrock/scriptstriplint:md(75 errors, all under the venv),lint:model-id(a retired Opus 4.1 id inside theanthropicpackage),test_model_id_lint.py::test_real_plugin_passesand rootlint:spec(6.venv/bin/python*symlinks). Every one passes on agit archivecopy without those directories. Becauselint:mdabortsmise run cias a single command, thecitasks were run one at a time. CI has no such venvs.mise run buildrunsgitleaks detect --source .. It reports exactly one finding, on the pre-existing upstream commitf8d09fd1:plugins/aws-startup-advisor/tests/test_bedrock_migration_assessment_discover.py:118, where the stringclaude-3.5-sonnetmatches thegeneric-api-keyrule. That is a false positive on a model name, it is onmain, it is not introduced by this branch, and it is not fixed here.Review round 1
Rebased onto #585's current tip
e6795f32(no conflicts; backup refbackup/pr588-pre-rebase-9a9c6012). Four fix commits address all eleven findings and both CodeQL alerts (per-thread replies cite the commit, change and tests):43d9fc8acatalog (Mantle Chat on GPT-5.6 Sol/Terra/Luna, GPT-5.6inference_profilesmaps, per-path capabilities for GPT-5.4/5.5,guardrails_supportedandserver_side_tools_supportedflags,_cites_aws_docsURL check),727b7e8eengines and verifier (runtime-only invocation logging vs CloudWatch, Guardrails resolved per model/API, selector-vs-API and preference conflicts as explicit decisions, application profiles kept off the OpenAI APIs, Astra non-streaming JSON Schema guard on all runtime APIs, fail-closedinference_profile_unverified),ea0798b3docs (injection template carriesmantle_selectorandmantle_requirement; Guardrails scope per card),91827f91CRIS prefix prose. F10 (§6.4 IAM recipe) was already fixed by #585's00cf7b6dand arrived with the rebase; nollm-to-bedrockfile changes on this branch.Behavior changes disclosed: GPT-6 Sol/Luna + Guardrails without continuity now stays on
runtime_openai_responses(wasruntime_converse); GPT-6 Sol + Guardrails from an EU-Geo caller fails closed withinference_profile_unverifiedinstead of falling to Claude on Converse; a GPT-6.1 Sol source needingstructured_output_jsonplus streaming falls to Astra via the evidence walk because GPT-6.1 Sol has no capability rows (unchanged from before, now documented).Gates run 2026-11-08 at
91827f91:P/skills/agent-advisor/scriptspytest: 789 passed (719 before this round).P/skills/llm-to-bedrock/scriptspytest: 655 passed (no files changed).python3 tools/sync-vendored.py --check: PASS, 65 files byte-identical. Rootpython3 tools/validate.py: all validations passed.git diff --check: clean.citasks run individually:lint:artifactsPASS (101),lint:vendored-parityPASS,lint:skill-refsPASS (1,544),lint:model-stalenessOK,lint:model-catalog-driftOK (same 3 pre-existing Shared Bedrock model catalog: follow-up sequence #566 warnings),lint:startup-model-pricingOK,lint:consent-disclosureOK,test1,495 passed + 222 subtests,test:telemetry311 passed,test:heroku-source-reviewpass; markdownlint-cli2 on the 822 tracked.mdfiles 0 errors. The only failures are the same untracked.venvnoise described above (lint:md,lint:model-id,test_model_id_lint.py::test_real_plugin_passes), absent in CI.mise run build), reported distinctly: still exactly one finding, on pre-existing upstreamf8d09fd1(claude-3.5-sonnetintests/test_bedrock_migration_assessment_discover.py:118); not introduced or fixed here.Review round 2
Rebased from
e6795f32onto #585's new tip4f861123(backup refbackup/pr588-pre-rebase-91827f91); the ten PR 3 commits replayed with no conflicts. Three commits address the remaining open items (per-thread replies cite the commit, change and tests):0f4c56e2.tools/sync-vendored.pyre-syncedskills/agent-advisor/references/vendored/ai/bedrock-endpoint-policy.mdwith fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585's F13 change (Anthropiccomputer_*,browser_toolset_*,text_editor_*,bash_*,memory_*moved out of theserver_side_toolspositive list as client-executed tools).sync-vendored --check: 65 files OK.af7ef41a.openai_model_recommendation.pyadds_RESPONSES_ONLY_SELECTORS = {background_inference, web_search}; a Chat Completions source matched by either (explicit, detected or criticalweb_search) is aselector_api_incompatibledecision betweenmantle_openai_responses(carrying the selector) andruntime_openai_chat(web-search.html / inference-responses-api.html, read 2026-11-09); the free-continuity reshape to Mantle Responses is unchanged, and_decision_optionsnever builds amantle_openai_chatoption for a Responses-only selector.workspaceson an OpenAI SDK workload is aselector_provider_mismatchdecision whose two options both stay on bedrock-runtime.anthropic_model_recommendation.py_path_constraintsaddsmantle_selector_vs_anthropic_api(background_inference: Messages has no background mode) andmantle_selector_provider_mismatch(projects) withruntime_messagesplus the runtime Converse/InvokeModel path as options (nevermantle_messages);_decision_requiredemits theselector_api_incompatible/selector_provider_mismatchfindings.model-recommend.mddocuments the per-selector validation. Tests:test_openai_model_recommendation.py::{test_gpt55_chat_source_with_web_search_is_a_decision_between_mantle_responses_and_runtime_chat, test_gpt55_chat_with_web_search_and_free_continuity_reshapes_to_mantle_responses, test_chat_source_with_web_search_plus_logging_never_offers_a_mantle_chat_option, test_workspaces_selector_on_an_openai_source_is_a_provider_mismatch_decision};test_model_recommendation.py::{test_anthropic_messages_with_background_inference_is_a_decision_not_mantle_messages, test_projects_selector_on_an_anthropic_source_is_a_provider_mismatch_decision, test_projects_with_a_preferred_mantle_path_still_never_offers_mantle_messages, test_background_inference_with_native_payload_offers_runtime_messages_or_invoke}; parametrized per-selector tests exclude the now-deciding selectors; the GPT-6.1 Sol Chat fallback case usesserver_side_toolsplus a free-continuityweb_searchrow. agent-advisor 795 passed. The design-doc half of F4 (GCP/Azuredesign-ai.mdrule 1b) is on fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585 (a8a0ec30).26970991.skills/shared/ai/models/openai-bedrock.json(+ agent-advisor vendored copy) gains a booleanprompt_caching_supportedon every GPT path row with the card read inevidence(read 2026-11-09): GPT-5.5 / GPT-5.4 true onruntime_openai_responsesonly (implicit caching, Responses API only; Mantle feature list names no caching); GPT-5.6 Sol/Terra/Luna true on every runtime row and Mantle Responses, false on Mantle Chat (the 2026-08-21 "Responses-only" Converse note marked superseded); GPT-6 Sol / Luna / 6.1 Sol true everywhere; GPT-6 Astra mirrors its existing per-path capabilities, so no PR 1/2/3 fact changes. Engine:_evidenced_capabilitiesaddsprompt_cachingiff the row's flag is true;_unsupported_required_capabilitiesand_compatibilitytakepath_configand check caching generically (Astra display-name special case removed); the candidate walk skips a candidate whose selected-API row lacks caching, so GPT-5.5 Chat +api_continuity: required+ criticalprompt_cachinglands on the same-family GPT-5.6 Chat row (prompt_cachinginnative, rationale names the Responses-only scope instead of "unavailable"); with no caching candidate theunverified_capabilityfinding names the Chat-versus-Responses trade-off (_other_runtime_api_with_prompt_caching); never a cleanruntime_openai_chatwith empty buckets.model-recommend.mdandai-openai-to-bedrock.md(+2 vendored) carry the flag. Tests:test_catalog_paths.py::test_every_gpt_path_row_carries_a_boolean_prompt_caching_flag;test_openai_model_recommendation.py::{test_gpt55_chat_with_required_continuity_and_critical_prompt_caching_never_recommends_a_cache_less_row, test_gpt55_responses_with_critical_prompt_caching_is_recommended_natively, test_prompt_caching_decision_names_the_responses_tradeoff_when_no_chat_candidate_has_it, test_gpt55_chat_without_a_caching_requirement_is_still_a_clean_recommendation};test_astra_required_caching_is_checked_on_the_selected_apiunchanged.Items F12, F15, F13 and F3-azure raised on this PR were fixed on fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585 (
c23d9072,c7641574,afcd9834) and arrive here through the rebase; see fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585's "Review round 3" section.Gates run 2026-11-09 at
26970991:P/skills/agent-advisor/scriptspytest: 800 passed (789 at91827f91).P/skills/llm-to-bedrock/scriptspytest: 672 passed (no files changed on this branch).tests/test_model_id_lint.py::test_real_plugin_passeson the untracked.venv, absent in CI).validate-artifacts.py --fixturesPASS (103);python3 tools/sync-vendored.py --checkPASS, 65 files byte-identical;check-skill-refs --checkPASS (1,544);lint:model-catalog-driftexit 0 (5 pre-existing warnings);git diff --checkclean; markdownlint 0 errors on touched docs.mise run cifails only atlint:mdwith 75 errors, all under the untracked.venv(0 outside).mise run build) not re-run this round: the only known finding is the pre-existing upstreamf8d09fd1described above.Negative tests added for the required behavior:
runtime_openai_responseson the same model; Chat Completions continuity →runtime_openai_chaton the same model; Anthropic Messages continuity →runtime_messages.mantle_selectorvalues → the Mantle path with the selector echoed (parametrized in both engines); the enum is pinned to the input schema; an unknown value raises.mantle_selector: web_search; Web Search marked absent → no Mantle; Web Search on a runtime path stays a rearchitecture finding.in_region_requiredon a CRIS-only runtime model → Mantle within_region_processingrecorded; the same with Guardrails → stays on runtime and blocks.preferred_api_path: mantle_*without a selector →mantle_selector_requiredblock; the output schema rejects amantle_*path with a null selector.runtime_openai_chat.apiPathenum value is probed; default client builders point at the right base URL.Model cards read 2026-11-07 for every new row (all under
https://docs.aws.amazon.com/bedrock/latest/userguide/):model-card-openai-gpt-56-sol.html,model-card-openai-gpt-56-terra.html,model-card-openai-gpt-56-luna.html,model-card-openai-gpt-55.html,model-card-openai-gpt-54.html,model-card-openai-gpt-6-astra.html,model-card-openai-gpt-6-sol.html,model-card-openai-gpt-6-luna.html,model-card-openai-gpt-6-1-sol.html,model-card-anthropic-claude-opus-5-5.html,model-card-anthropic-claude-sonnet-5.html,model-card-anthropic-claude-haiku-4-5.html.Policy and API pages read 2026-11-07 (same base):
endpoints.html,inference-responses-api.html,inference-chat-completions.html,inference-messages-api.html,workspaces.html,monitoring-mantle-metrics.html,model-invocation-logging.html,tool-use-server-side.html,web-search.html.Limitations
the GPT-5.6 Sol card lists Chat Completions on bedrock-mantle; the catalog keeps(resolved in review round 1, F11: Sol/Terra/Luna Mantle Chat rows now available). (2) GPT-5.6 Sol / Terra / Luna cards list structured outputs on runtime and implicit caching on Responses / Chat / InvokeModel / Converse; themantle_openai_chat.available: falseruntime_converseconstraint still says GPT-5.6 runtime lacks structured outputs and caching is Responses-only. (3) GPT-5.6 cards list Invoke as unsupported on runtime; theruntime_converseevidence still says "Responses/Chat Completions/Invoke/Converse". (4) The GPT-6 Astra card lists Invoke as supported on runtime and structured outputs plus explicit caching on Mantle Chat; the catalog says "Invoke and Messages unsupported" and "prompt caching is Responses-only". (5)GPT-6 Sol / Luna cards list Guardrails on Responses / Chat / InvokeModel / Converse and GPT-6.1 Sol on Chat / InvokeModel / Converse only; older GPT-6(resolved in review round 1, F3: every GPT path row carriesruntime_converserows do not mention Guardrailsguardrails_supported). (6) Sonnet 5 publishesin.anthropic.claude-sonnet-5and Haiku 4.5 publishesin.andjp.profiles; the Anthropic catalog has noinference_profilesblock for them. (7) The Opus 5.5 runtime card lists server-side tool search; the catalog makes no claim either way.matches_advisor_modelinmigration-plan.md/handoff-migration.mdtreatsruntime_converseon a proprietary GPT model (openai.gpt-6*,gpt-5.6-*,gpt-5.5,gpt-5.4) as a CRIS match and keeps the exact comparison for open-weightgpt-oss; a legacy advisor contract that has nomantle_selectorkey also keeps the exact comparison. Both are required byllm-to-bedrock'stest_astra_execution_contract.pyfrom fix(aws-startup-advisor): default to bedrock-runtime in llm-to-bedrock and gcp/azure AI guidance, add a Runtime Anthropic Messages lane #585, which this PR does not edit. Every artifact the current engines emit carries the key, so the behavior for new artifacts is the strict one.openai/anthropicpackages plusaws-bedrock-token-generatorat run time; the default builders raise a clear message when they are missing. No live probe runs in CI.Landing order
#567 (
selection_statusand the catalog schema) and #581 (catalogs rendered from the registry) have merged, and this branch is updated onto them, so the earlier "#567 first, then this PR, then #581" plan is done. What remains is #583, then #585, then this PR, because this branch still carries their commits. Thepricingobject onopenai_gpt_5_4andopenai_gpt_5_5and the runtime path rows passvalidate-artifacts.pyunder #567's schema in the full CI run. See #566.Type of Change
Checklist
python3 tools/validate.pyStartup Advisor review checklist
By submitting this PR, I confirm that:
mise -C plugins/aws-startup-advisor run ciand all checks pass.Note on the
ciconfirmation: eachcitask was run individually; the only local failures were the untracked.venvdirectories described under Validation, which do not exist in CI, and the GitHubstartup-advisorcheck (which runsmise run ci) passes on this head.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.