Repository navigation
fix(aws-startup-advisor): re-verify Bedrock lifecycle statuses and remove Legacy/gated models from recommended targets - #598
Conversation
…move Legacy/gated models from recommended targets Re-verified 2026-11-09 against the AWS legacy lifecycle page, model cards, OpenAI deprecations, and Google's Gemini deprecations / Vertex model-versions pages. - Pricing cache: Gemma 3 4B/12B/27B and MiniMax M2/M2.1 are Legacy (EOL 2027-03-30), not active; DeepSeek-V3.1 is not "Sydney only"; Nova 2 Pro is a gated preview with no Bedrock model card and is now `restricted`. - Lifecycle registry: add the five Legacy rows; stop naming Nova 2 Pro as an Active replacement; add the Gemini 3.5/3.6/3.7 Flash Developer-API routings. - OpenAI guide: Pro and o-series cross-family rows target Claude Opus 5.5 (Global) instead of Nova 2 Pro; o3 no longer maps to Legacy DeepSeek-R1; rows for models with announced shutdowns (gpt-5*, o3, o3-pro on 2026-12-11; gpt-4.1*, gpt-4o-mini on 2027-01-14; gpt-5.1 on 2027-04-01) carry the date and OpenAI's named replacement. - Gemini guide: Gemini 3.8 Flash is the current Flash and the successor for 3.5/3.6/3.7 Flash; add Gemini 4 Argon rows with its Dec 1, 2026 price step; drop the shut-down Gemini 3 Pro rows and the Legacy Llama 4 / Nova 2 Pro targets. - gcp-to-aws / bedrock-migration-assessment / agent-advisor phase files: same target swaps; sample Gemini ids move from gemini-3.5-flash to gemini-3.8-flash. - New text-contract test pins the statuses and keeps Legacy/restricted models out of the live-tier target columns.
herosjourney
left a comment
herosjourney
left a comment
There was a problem hiding this comment.
Reviewed: head 41abb09 (base 6d6948d), 25 files (+462/−314) — skills/shared/ai (lifecycle registry, pricing cache, OpenAI/Gemini/OpenRouter guides) plus gcp-to-aws, bedrock-migration-assessment, and agent-advisor phase files and vendored copies; one new test.
Ran: mise -C plugins/aws-startup-advisor run ci (exit 0: lint:md 0 errors; model-id lint OK, 19 retired ids; model-staleness, vendored-parity, skill-refs, artifacts, consent-disclosure, model-pricing PASS; test 1415 passed / 222 subtests; test:telemetry 311 passed). git merge-tree against upstream/main and against open #583, #585, #588, #592, which touch the same files: all clean. Recomputed days-to-EOL from 2026-11-09 and the OpenAI cross-family deltas at 150M in / 75M out. Primary sources read 2026-11-09: OpenAI deprecations, Gemini API deprecations, Vertex AI model versions.
Did not run: no live Bedrock calls or ListFoundationModels; no end-to-end skill run. The Nova 2 Pro gated-preview status and the Gemma 3 / MiniMax Legacy dates were not re-fetched; the body's citations were taken as given.
Vendored parity: PASS — sync-vendored.py --check reports 62 vendored files byte-identical to skills/shared/. The gcp-to-aws adapter-local design-refs/ai-gemini-to-bedrock.md carries the same table edits as the shared guide; only its existing header differs.
Change narrative: Lifecycle statuses were stale, and Legacy or gated models were still recommended. The PR re-verifies statuses in the pricing cache and lifecycle registry, swaps Nova 2 Pro, Llama 4, and DeepSeek-R1 out of live targets, and annotates OpenAI and Gemini source shutdowns. It touches the shared guides, their vendored copies, and the Clarify, Discover, and Design phase files that restate them. Body vs diff: matches, but two claims are not fully met — "every row whose source has an announced shutdown carries the date" (see Blocking 1), and Nova 2 Pro removed as a target from the gcp-to-aws Clarify files (see Blocking 3).
| Axis | Stance | One line |
|---|---|---|
| Parsimony | clear | Most of the line count is regenerated vendored copies; the source edits are row-level. |
| Completeness | finding | One in-diff Clarify file still recommends Nova 2 Pro; Q19's menu still teaches 3.5 Flash as the flagship. |
| Security | clear | Markdown and one text-contract test; CodeQL and dependency review green; no IaC, scripts, or credentials touched. |
| Alternatives | advisory | Two below. |
| Contradictions | finding | The 3.7 Flash Status contradicts Rule 2 in the same file; Q19's menu contradicts the Gemini guide's Competitive Reality. |
| Overlap | advisory | OpenAI shutdown dates are restated per row in the guide and again in Clarify; see Alternative 2. |
| Vendored parity | clear | 62/62 byte-identical. |
| Propagation & data | finding | SSOT: ai-model-lifecycle.md / bedrock-pricing-cache.md. Consumers swept: shared guides, Clarify (gcp, bedrock-migration-assessment), Discover, Design, agent-advisor. Stale forms found: 5 surfaces, listed below. |
Blocking
-
[Propagation & data] OpenAI sources that already shut down are priced as live comparisons.
skills/shared/ai/ai-openai-to-bedrock.md:214,216(rewritten in this PR), plus unchanged rows at:225,:228,:229. Vendored copies: gcp-to-aws and azure-to-aws.| o1-pro | 150.00 / 600.00 | Claude Opus 5.5 (Global) | 4.00 / 20.00 | Bedrock 97% cheaper |
| o1 | 15.00 / 60.00 | Claude Opus 5.5 (Global) | 4.00 / 20.00 | Bedrock 69% cheaper |The OpenAI deprecations page, 2026-04-22 notice: "Access to these models was shut down on October 23, 2026" for
o1(replacementgpt-5.6-sol),o1-pro(gpt-5.6-sol,reasoning.mode: pro),gpt-4andgpt-4-turbo(gpt-5.6-sol), andgpt-4.1-nano(gpt-5.6-luna).
Consequence: a customer still naming one of these ids gets a 69–97% "Bedrock cheaper" headline against a price nobody has been billed since October 23. The PR annotates the later shutdowns (Dec 11, Jan 14, Apr 1), so this table now looks complete while missing the ones already past.clarify-ai.md:365repeatso1 at $15/$60 → Opus 5.5 (Global) 69% cheaper, and:359repeats the GPT-4 Turbo 72% figure.
Fix: mark these five rows**Shut down 2026-10-23** → <OpenAI replacement>. Price them against a measured bill, or against the replacement's rate, the same way the lifecycle file's "Source baseline by serving API" rule handles retired Gemini 2.0 sources. Drop the live "Bedrock N% cheaper" cell. Apply the same edit toclarify-ai.md:359,365. -
[Contradictions] The Gemini 3.7 Flash Status breaks the 90-day rule in the same file.
skills/shared/ai/ai-model-lifecycle.md:195(vendored copies: gcp-to-aws, azure-to-aws).| Gemini 3.7 Flash (gemini-3.7-flash) | 2027-01-28 | … |legacy (EOL 2027-01-28)|Rule 2 in the same file (
:76): "0 < days_to_eol ≤ 90→ Exclusion zone. … Markexcluded." From the file's own as-of date, 2027-01-28 − 2026-11-09 is 80 days. The Vertex AI model-versions page confirms "gemini-3.7-flash| August 13, 2026 | January 28, 2027 |gemini-3.8-flash". The 3.6 Flash row directly below gets this right (10 days,excluded).
Consequence: an agent that applies the enum as written keeps a model with 80 days left in comparison tables, which is exactly what the exclusion zone exists to stop.
Fix:`excluded (EOL 2027-01-28)` on Vertex AI; routed on the Developer API. That mirrors the 3.5 Flash row, since the Developer API routesgemini-3.7-flashtogemini-3.8-flash. -
[Completeness] A file this PR edits still recommends Nova 2 Pro.
skills/gcp-to-aws/references/phases/clarify/clarify-assemble.md:385| GPT-5.5 Pro migration | Q19 = GPT-5.5 Pro | Nova 2 Pro — 95% cheaper on Bedrock |This PR sets Nova 2 Pro to
restricted (gated preview …).ai-model-lifecycle.mdsays restricted means "never a default in a mapping guide." The PR also changes the matching rows inclarify-ai.md,clarify-ai-only.md, anddiscover-preview.mdto Opus 5.5, and edits this same file at:66and:554, but not this row.
Consequence: the Clarify confirmation sheet tells a GPT-5.5 Pro user that their baseline is a model they cannot get access to, which is the exact failure the PR body describes.
Fix:| GPT-5.5 Pro migration | Q19 = GPT-5.5 Pro | GPT-5.6 Sol (same vendor), or Claude Opus 5.5 (Global) — 88% cheaper at 150M/75M |. The 88% figure matches the rewritten cross-family row.
Should-fix
-
[Contradictions] Q19's cold-path menu still teaches 3.5 Flash as the current flagship and leaves out Gemini 4 Argon.
skills/gcp-to-aws/references/phases/clarify/clarify-ai.md:320,351> 1. Gemini 3.5 Flash (GA — current flagship Flash model)
… recommend upgrading source model to 3.5 Flash or 3.5 Flash-Lite before or alongside migrationThis PR's Competitive Reality section in the Gemini guide says "Gemini 3.8 Flash (GA Sep 2, 2026) is the current Flash model", names 4 Argon as the flagship, and adds a
Gemini 4 Argon / 3.1 Promapping row at:350. The menu the user picks from has no Argon option, and option 8 still listsGemini 3 Pro, which was shut down 2026-03-09.
Fix: option 1 →Gemini 3.8 Flash (GA — current Flash; Google's upgrade for 3.5 / 3.6 / 3.7 Flash); demote 3.5 Flash to a "still GA on Vertex AI" entry; addGemini 4 Argon; change option 8 toGemini 3.1 Pro Preview. At:351, change "upgrading source model to 3.5 Flash" to "3.8 Flash". The gcp-to-aws source cache repeats the same claim atskills/gcp-to-aws/references/shared/pricing-cache.md:427("Gemini 3.5 Flash is now GA and the current flagship Flash model"). It is outside this diff, but this PR vouches for the same fact, so update it here. -
[Propagation & data] A past-EOL row stays in a comparison table.
skills/shared/ai/ai-gemini-to-bedrock.md:69(anddesign-refs/…:65, vendored copies).| Nova Premier | **Past EOL** (Sep 14, 2026) — prefer Nova 2 Lite or Claude Opus 5.5 | … |Rule 1 (
ai-model-lifecycle.md:75): "days_to_eol ≤ 0→ EOL. Remove from all tables." The model is already in that file's Removed list. Fix: delete the row; the Removed entry carries the warning. -
[Completeness §5] The new test does not cover the surfaces that went stale.
tests/test_legacy_and_restricted_targets.pyreads only the shared pricing cache, lifecycle file, and the OpenAI and Gemini guides. It never checks Clarify or Discover, and it does not check that Status agrees withdays_to_eol. All three Blocking items pass it. Suggested regressions:
(a) everylegacy (EOL YYYY-MM-DD)value in the lifecycle tables is more than 90 days from the file'sLast updated;
(b)Nova 2 Pro/nova-2-pronever appears as a recommendation outside arestricted/gated previewannotation underskills/;
(c) no OpenAI id on the deprecations page's past-shutdown list is in the cross-family table without aShut downmarker.
Nits
ai-model-lifecycle.md:143— "New Legacy rows this refresh … Llama 4 Maverick / Scout, Llama 3.3 70B, DeepSeek-R1 / V3.1, Claude Sonnet 4.5" lists the 2026-10-26 additions. The rows actually added in this refresh are Gemma 3 4B/12B/27B and MiniMax M2/M2.1; swap them in, or rename the label "Legacy rows still >90 days".
Heads-up (out of diff, pre-existing — not blocking)
tools/model-catalog-drift.pystill warns thatopenai-bedrock.jsonis 46 days old and Sonnet 5.5, Haiku 5.5, and GPT-6.1 Terra are missing from the catalogs. The body says this is out of scope; it needs a follow-up issue.
Alternatives (max 2)
- Compute Status instead of hand-writing it — extend
tools/model-staleness.py --strictto recomputedays_to_eolfrom each row's EOL date and fail when the Status bucket disagrees. Wins: Blocking 2 becomes impossible, and the next refresh cannot drift. Costs: the source-model table mixes per-API statuses, so the parser has to handleX on Vertex AI; Y on the Developer API. Recommendation: follow-up. - One OpenAI source-retirement table in
ai-model-lifecycle.md, matching the Gemini 2.0/2.5 section, with the guides linking to it instead of carrying per-row shutdown prose. Wins: one place to refresh, and it would have caught the October 23 batch. Costs: adds a level of indirection to the cross-family table. Recommendation: follow-up.
Verdict
Request changes — three blocking items, each checked against the primary source or the file's own rule. Commenting rather than requesting changes: this review is posted from the PR author's account (herosjourney), and GitHub doesn't let an author approve or request changes on their own PR. Code owners per CODEOWNERS (Agent Toolkit Admins) decide the merge.
…I sources, 3.7 Flash exclusion, Nova 2 Pro in Clarify Blocking - OpenAI guide: o1, o1-pro, o3-mini/o4-mini, GPT-4, GPT-4 Turbo, GPT-4.1 Nano, and GPT-3.5 Turbo were shut down 2026-10-23 (deprecations page, 2026-04-22 notice). Their rows now carry the marker and OpenAI's substitute and quote no live "Bedrock N% cheaper" figure; the intro says how to price them. gcp-to-aws Q19 rows for GPT-4/4 Turbo and the o-series say the same. - Lifecycle registry: Gemini 3.7 Flash is 80 days from its Vertex AI retirement, so its Status is `excluded`, matching Rule 2. - gcp-to-aws clarify-assemble.md: the GPT-5.5 Pro confirmation row offers GPT-5.6 Sol or Claude Opus 5.5 (88% cheaper at 150M/75M) instead of Nova 2 Pro. Should-fix - Q19 cold-path menu: 3.8 Flash is option 1 (current Flash); 3.5 Flash is "still GA on Vertex AI, routed to 3.6 on the Developer API"; Gemini 4 Argon added; 3.6/3.7 Flash listed with their Vertex retirement dates; Gemini 3 Pro Preview flagged as shut down 2026-03-09. The retired-Flash row and the gcp-to-aws source pricing cache now name 3.8 Flash as the upgrade. - Gemini guide: the past-EOL Nova Premier row is removed (the Removed list in the lifecycle file carries the warning). - Tests: Status bucket must agree with days-to-EOL from Last updated; Nova 2 Pro may appear only with a restricted/gated annotation anywhere under skills/; every cross-family row for a 2026-10-23 shutdown carries the marker and no live delta. The Nova 2 Pro check also caught the two multimodal/batch pricing rows, which now carry the annotation. Nit - Lifecycle notes label the Gemma 3 / MiniMax rows as this refresh's additions and the Llama 4 / Llama 3.3 / DeepSeek / Sonnet 4.5 rows as carried over.
herosjourney
left a comment
herosjourney
left a comment
There was a problem hiding this comment.
Re-review at 36ea47e (previous: 41abb09). Delta: 1 commit, 18 files. GitHub startup-advisor CI passed on this head. Re-ran tests/test_legacy_and_restricted_targets.py: 9 passed. Re-read the OpenAI deprecations page (2026-04-22 and 2025-04-28 notices). Did not re-run the full mise suite locally and made no live AWS calls.
| Prior finding | Status at head |
|---|---|
| Blocking 1 — shut-down OpenAI sources priced as live | Fixed at 36ea47e. o1, o1-pro, gpt-4, gpt-4-turbo, gpt-4.1-nano, and gpt-3.5-turbo now say Shut down 2026-10-23 and no longer carry Bedrock N% cheaper. clarify-ai.md Q19 matches for GPT-4 / GPT-4 Turbo and for o1. Replacements match the page (gpt-5.6-sol, gpt-5.6-luna for nano, gpt-5.6-terra for gpt-3.5-turbo). |
Blocking 2 — Gemini 3.7 Flash marked legacy at 80 days |
Fixed at 36ea47e. Status is `excluded (EOL 2027-01-28)` on Vertex AI (80 days); routed on the Developer API. The new bucket test measures that from Last updated and passes. |
Blocking 3 — clarify-assemble.md still recommended Nova 2 Pro |
Fixed at 36ea47e. The GPT-5.5 Pro row is now GPT-5.6 Sol, or Claude Opus 5.5 (Global) at 88% (150M/75M). A repo sweep finds no Nova 2 Pro line outside vendored/ that lacks a restricted / not-a-target annotation. |
| Should-fix — Q19 menu still called 3.5 Flash the flagship | Fixed at 36ea47e. Option 1 is 3.8 Flash; 3.5 Flash is "still GA on Vertex AI"; Gemini 4 Argon is option 3; Gemini 3 Pro is marked shut down 2026-03-09; the 1.5/2.0 row upgrades to 3.8 Flash. pricing-cache.md no longer calls 3.5 Flash the flagship. |
| Should-fix — Nova Premier left in the quick-reference table | Fixed at 36ea47e. The row is gone from the shared guide, the gcp-to-aws design ref, and both vendored copies. |
| Should-fix — the test missed those surfaces | Fixed at 36ea47e. Three regressions: status bucket vs days_to_eol, Nova 2 Pro annotation sweep, and a shut-down marker with no live delta. All three fail on the previous head's text and pass here. |
| Nit — "New Legacy rows this refresh" named the October set | Fixed at 36ea47e. The sentence now names Gemma 3 and MiniMax M2/M2.1, and labels Llama 4 / DeepSeek / Sonnet 4.5 as carried over from 2026-10-26. |
New in the delta: three leftovers, below. Sweep for Bedrock N% cheaper on the shut-down ids inside the cross-family table: none remain.
Should-fix
-
[Propagation & data] The
o1-minihedge is the opposite of the page it cites.skills/shared/ai/ai-openai-to-bedrock.md:218(vendored copies too).**Shut down 2026-10-23** (o4-mini→gpt-5.6-terra,o3-mini→gpt-5.6-sol;o1-miniis not on the deprecations page — verify)The 2025-04-28 notice says
o1-minishut down 2025-10-27, replacemento4-mini.o4-minithen shut down 2026-10-23, replacementgpt-5.6-terra. An agent told to "verify" can treato1-minias possibly still billed. The same row's target column is stillGPT-5.6 Luna, while the delta says to price againstgpt-5.6-terra.
Fix:**Shut down** —o1-mini2025-10-27 →o4-mini;o4-miniando3-mini2026-10-23 →gpt-5.6-terra/gpt-5.6-sol. Price against a measured bill or those substitutes, not $1.10/$4.40.Point the target column atgpt-5.6-terra. -
[Completeness] Two summary rows this commit's files still quote a live saving on a shut-down price. The guide now says both sources shut down on 2026-10-23.
skills/gcp-to-aws/references/phases/clarify/clarify-assemble.md:388(three rows under the line this commit edited):GPT-4 Turbo migration … Claude Sonnet 5 — 80% cheaper on inputskills/gcp-to-aws/references/phases/clarify/clarify-ai.md:358: the GPT-3.5 Turbo row still gives Haiku 56.25% and Luna 54.29% against that source, with no shut-down marker.
The new test only reads the cross-family table, so both passed CI.
Fix: same wording as the GPT-4 row atclarify-ai.md:359— shut down, price a measured bill or the substitute (gpt-5.6-solfor GPT-4 Turbo,gpt-5.6-terrafor GPT-3.5 Turbo), no live percentage.
Verdict
Comment — supersedes the request-changes on 41abb09. All three blocking findings and the earlier should-fix items are fixed at 36ea47e and re-probed. Two should-fix items are new in the delta; neither recommends a model the customer cannot access. Commenting rather than approving: this review is posted from the PR author's account (herosjourney). Code owners (Agent Toolkit Admins) decide the merge.
…, remaining live savings on shut-down sources - OpenAI guide: the o4-mini / o3-mini / o1-mini row now states o1-mini shut down 2025-10-27 → o4-mini and o4-mini / o3-mini shut down 2026-10-23 → gpt-5.6-terra / gpt-5.6-sol (deprecations page, 2025-04-28 and 2026-04-22 notices); the target column points at GPT-5.6 Terra, matching the substitute. - gcp-to-aws Clarify: the GPT-3.5 Turbo Q19 row and the GPT-4 Turbo and o-series confirmation-sheet rows carry the shut-down marker and quote no live saving. - Test (d): gcp-to-aws Clarify rows for shut-down OpenAI sources must carry a shut-down marker and no live "N% cheaper" figure (o3's still-live "source 14% cheaper" is allowed until 2026-12-11).
herosjourney
left a comment
herosjourney
left a comment
There was a problem hiding this comment.
Re-review at e5ffe44 (previous: 36ea47e). Delta: 1 commit, 6 files. GitHub startup-advisor passed on this head. Re-ran tests/test_legacy_and_restricted_targets.py: 10 passed. Re-read the o1-mini row against the 2025-04-28 and 2026-04-22 notices.
| Prior finding | Status at head |
|---|---|
Should-fix — o1-mini "not on the deprecations page" |
Fixed at e5ffe44. The row now says o1-mini shut down 2025-10-27 → o4-mini, then o4-mini / o3-mini on 2026-10-23 → gpt-5.6-terra / gpt-5.6-sol. The target column is GPT-5.6 Terra. Vendored copies match. |
| Should-fix — GPT-4 Turbo confirmation sheet still said 80% cheaper | Fixed at e5ffe44. clarify-assemble.md now says GPT-5.6 Sol, shut down 2026-10-23, price a measured bill, no live saving. The o-series sheet row got the same treatment. |
| Should-fix — GPT-3.5 Turbo Q19 still quoted live Haiku/Luna savings | Fixed at e5ffe44. The note says shut down 2026-10-23 → gpt-5.6-terra, and the 56.25% / 54.29% figures are gone. |
New in the delta: one should-fix, below. Sweep for Bedrock N% cheaper on the shut-down rows: none. The new Clarify test requires a shut-down marker and rejects a live % cheaper on those four rows.
Should-fix
-
[Propagation & data] The renamed target kept Luna's price.
skills/shared/ai/ai-openai-to-bedrock.md:218(vendored gcp and azure copies too).| o4-mini / o3-mini / o1-mini | 1.10 / 4.40 | GPT-5.6 Terra | 0.22 / 1.32 | **Shut down** — …The same file prices GPT-5.6 Terra at
2.20 / 13.20(the cost table,GPT-5.6 Terra | 2.20 / 13.20).0.22 / 1.32is Luna. An agent that copies the Bedrock-price cell bills Terra at a tenth of its rate.o3-mini's substitute on the deprecations page isgpt-5.6-sol, not Terra, so one price cell cannot cover the row. The new test only checks that the cell contains "Shut down" and "2026-10-23", so this passed CI.
Fix: split the row.o4-mini/o1-mini→ targetGPT-5.6 Terra, Bedrock price2.20 / 13.20.o3-mini→ targetGPT-5.6 Solat Sol's rate from the cost table. Keep the shut-down dates and "do not bill $1.10/$4.40".
Verdict
Comment — supersedes the comment on 36ea47e. The three should-fix items from that review are fixed and re-probed. One new should-fix is open: the Terra target is still priced at Luna's rate. Commenting rather than approving: this review is posted from the PR author's account (herosjourney). Code owners (Agent Toolkit Admins) decide the merge.
… its own Bedrock rate o4-mini / o1-mini -> GPT-5.6 Terra at 2.20 / 13.20; o3-mini -> GPT-5.6 Sol at 4.40 / 22.00 (the merged row had kept Luna's 0.22 / 1.32 under a Terra target). The test now checks the Bedrock-price cell against the named target's rate from the same file.
herosjourney
left a comment
herosjourney
left a comment
There was a problem hiding this comment.
Re-review of #598 at 4e1ba0b (delta since e5ffe44: one commit, 4 files)
Ran: diff e5ffe44..4e1ba0b; tests/test_legacy_and_restricted_targets.py (10 passed); tools/sync-vendored.py --check (62 vendored files byte-identical); cmp of the shared guide against the gcp-to-aws and azure-to-aws vendored copies (identical); CI on this head (all 10 checks pass).
Did not run: a live AWS Price List query (the pricing MCP servers were unavailable). Gemini/Argon and the Nova 2 Pro / Gemma / MiniMax facts are still taken from the PR body, as in the first review.
Prior findings
| Finding | Status |
|---|---|
Should-fix: ai-openai-to-bedrock.md:218 listed target GPT-5.6 Terra with Luna's 0.22 / 1.32 price, and one cell could not cover both Terra and Sol |
Fixed at 4e1ba0b. The row is split. o4-mini / o1-mini → Terra at 2.20 / 13.20, which matches the cost table at line 181. o3-mini → Sol at 4.40 / 22.00, which matches line 180. The o3-mini row also carries the 2026-10-23 shutdown date. |
New in delta
None. The three guide copies change identically, and the test change adds a check that each row's price cell equals the named target's own rate (prices[target] == price). That check would have caught the original bug.
Sweep
- Searched
skills/andtests/, vendored copies included, for the old labelo4-mini / o3-mini / o1-miniand foro3-mini / o1-mini. No stale references remain. OPENAI_SHUT_DOWNnow mapso3-miniando4-minito their own rows, so the shut-down marker test still covers every source named in the 2026-10-23 batch.
Still open from earlier (non-blocking, unchanged)
- Model-catalog-drift warnings (Sonnet 5.5, GPT-6.1 Terra).
- Teaching
model-staleness.py --strictto recomputedays_to_eol.
Verdict (supersedes the 05:00Z review)
No blocking or should-fix items are open at 4e1ba0b, and CI is green. This is posted as a COMMENT review from the author's account (herosjourney), so it cannot be an approval. The merge decision is for Agent Toolkit Admins.
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed 4e1ba0ba2a79180ce54799706a1875b785bb0b3a with OCR-assisted repository review and one fresh independent general reviewer. The Terra/Sol row split is verified fixed in all three guide copies. Three P2 source-pricing handoffs remain:
- Gemini serving API: Extend full-flow and AI-only Clarify plus Estimate/cache handling so the new 3.5 Flash billing split uses a confirmed API or labeled alternatives.
- Argon source rate: Add the dated Gemini 4 Argon rates to the GCP source cache used by token-only estimates.
- Retired OpenAI baselines: Apply the new shutdown/substitute rule in both GCP and Azure Estimate/source-cache lookups for direct OpenAI sources. Keep measured spend first and unverified substitute prices unavailable.
Validation on original source: 26 focused contract tests and all 153 model-selector tests passed; 62 vendored files passed parity; strict lifecycle freshness and git diff --check passed. The lookup findings were checked against original source and captured primary vendor documents. All 10 GitHub checks passed at this head. No live inference or end-to-end skill run was performed.
…arify Q19b and Estimate Part 1 Addresses the three P2 handoffs from the automated review on aws#598. - Gemini serving API: Clarify Q19b (full flow and AI-only) now fires for Gemini 3.5 / 3.6 / 3.7 Flash as well as 2.5, because Google bills 3.5 Flash at list on Vertex AI but routes it to 3.6 Flash on the Developer API. gcp Estimate Part 1 picks the row by serving API and, when unknown, presents both figures as labeled alternatives (cached_fallback) instead of silently taking the list row. - Argon source rate: the gcp source cache gains Gemini 4 Argon (2.00/10.00 through 2026-11-30, 4.00/20.00 from 2026-12-01), 3.6 Flash, and 3.7 Flash rows; the 3.5 Flash row says it is the Vertex AI list rate. Estimate applies the Argon date step. - Retired OpenAI baselines: the source cache's OpenAI rows carry their shutdown dates and substitutes, with a note on how to price a direct-OpenAI source after shutdown (substitute's row; unavailable when that row is _unverified_, e.g. o4-mini -> Terra). gcp and azure Estimate Part 1 apply that rule before the cached fallback, keep measured spend first and the raw source id, and scope the dates to OpenAI-hosted sources (Azure OpenAI follows Microsoft's schedule).
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Re-reviewed 7adde9582034ddeaac18d131184ee00843cbccf2 with actual OCR delegation, the repository review, and one fresh independent general reviewer. Argon source pricing is verified fixed. Two existing propagation concerns remain, with detailed replies in their original threads:
- Gemini serving API: The new question dispatch and Estimate split are present, but Q19b’s answer effects and
schema-preferences.mdstill apply the retired-2.5 explanation/catalog to live 3.x inputs. Correct both so the saveddesign_consequenceand report appendix describe the actual model/API/date. - Azure OpenAI-direct fallback: Add the shutdown metadata consumed by Azure’s new conditional lookup to its local source cache or a vendored shared contract. The GCP correction alone leaves Azure’s date predicate unsatisfied.
Validation on original source: 26 focused contract tests and 153 selector tests pass; 62 vendored files match; strict lifecycle freshness, model-ID lint (19 IDs), and git diff --check pass. All 10 GitHub checks pass at this head. A tentative Vertex 3.7 numeric-pricing concern was discarded after fresh primary pricing confirmed equal Standard Global 3.7/3.8 rates. No live inference, end-to-end conversational run, or report rendering was performed.
…source cache carries OpenAI shutdown metadata - Q19b's answer table and the schema-preferences design_consequence row no longer tell every Vertex AI answer that the source is past EOL. They branch by model: a 2.5 model is retired on Vertex AI and billed at list on the Developer API until Dec 1, 2026; a live 3.5 Flash is priced at the Vertex AI list rate and at the routed 3.6 Flash rate on the Developer API; 3.6 / 3.7 Flash price at 3.8 Flash where retired or routed. The saved consequence names the model, API, and date that applied. - azure-to-aws/references/shared/pricing-cache.md: OpenAI rows carry their shutdown dates and substitutes, with the same provider-scoped shutdown note as the gcp cache, so Azure Estimate Part 1's date predicate has local data. The Estimate text now points at this skill's own cache rather than gcp-to-aws's.
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖] Reviewed 0beb1461ae80c4f6e4e86468f6dd4125f1bd0aa7 with OCR delegation, the parent repository review, and one fresh isolated general reviewer.
Both previously open concerns are verified fixed: Q19b's persisted explanation/catalog and Azure's local shutdown metadata. Their original conversations have been resolved. Argon's cache entry and December date split also remain intact.
One P2 remains: Argon's new source comparison assumes Global/Developer prices for non-global Vertex usage. I missed this case at 7adde958; the inline finding records the primary rates and reproduction.
The required correction is confined to the Argon endpoint distinction:
- Update GCP
references/shared/pricing-cache.mdandreferences/phases/estimate/estimate-ai.mdto select the source endpoint and date. - Keep full and AI-only Clarify plus
references/shared/schema-preferences.mdconsistent when recording or labeling an unknown source endpoint. - Qualify Argon parity/cost statements in
skills/shared/ai/ai-gemini-to-bedrock.md, GCP's design reference, and the GCP/Azure vendored copies.
At 150M input/75M output, non-global Vertex should yield $1,155 on November 30 and $2,310 on December 1; Global/Developer remains $1,050/$2,100. Measured spend must still win, and an unknown endpoint must not produce an unqualified Global baseline.
Validation: 26 focused tests and 153 selector tests passed; all 62 vendored files match; strict model freshness and the full model-ID lint passed. These are local source-contract and arithmetic checks, not a live inference or rendered-report run. Approval remains for the user.
…as well as date Vertex AI bills Argon 10% more on a non-global (regional) endpoint than on the Global endpoint or the Developer API: $2.20/$11 vs $2/$10 through 2026-11-30, $4.40/$22 vs $4/$20 from 2026-12-01 (Vertex AI pricing, read 2026-11-10). - gcp source cache: the Argon row is split into Global/Developer and Vertex AI regional rows with the date step on each. - gcp Estimate Part 1: Argon is priced by endpoint and date (at 150M/75M: $1,050 vs $1,155 before the step, $2,100 vs $2,310 after); an unknown endpoint presents both as labeled alternatives, never an unqualified Global baseline. - Clarify: new Q19c (Vertex AI Global vs regional endpoint) for Argon baselines on Vertex AI or an unknown API, writing ai_constraints.ai_source_endpoint_scope; the Q19 menu and rationale, AI-only Clarify, and schema-preferences carry the field. - Gemini guide (shared and gcp-local): Argon parity/cost statements and the Pro-tier and 150M/75M tables carry both endpoint rates (Sonnet 5 is parity vs Global, 9% cheaper vs regional, before the step).
Brings in #598 and #600 (Claude Haiku 5.5 as the fast/low-cost tier, re-verified lifecycle statuses). Catalog conflicts resolved by taking main's catalogs and adding selection_status (all 17 entries selectable, which keeps main's ranking: Haiku 4.5 stays a lower-ranked explicit alternative). The catalog schema gains the new optional model fields mantle_regions and service_tiers. Verified: mise run ci passes (1,542 plugin tests, 600 agent-advisor tests); recommendation output is byte-identical to main across 56,160 cases; drift check now reports 2 warnings (Haiku 5.5 resolved).
Brings in #567's merge of main (#598, #600: Claude Haiku 5.5 as the fast/low-cost tier). The registry is regenerated from #567's catalogs: 13 records, with Haiku 5.5 stored once and rendered into both views. The generator and model-registry.schema.json gain the new optional fields mantle_regions and service_tiers. Rendered catalogs are JSON-equal to #567's apart from generated_from; recommendation output is byte-identical to #567's head across 56,160 cases.
…r-runtime-default Conflict resolution keeps both sides' behavior: - anthropic_model_recommendation.py: upstream's _explicit_profile_matches helper (aws#600) plus this branch's selector-aware _decision_options signature. - test_model_recommendation.py / test_openai_model_recommendation.py: union of both branches' tests. The cost-priority Mantle test keeps this branch's workspaces selector with upstream's "Haiku 5.5 is not a Mantle option in us-east-1" assertion; upstream's GovCloud Mantle test gains the selector it needs now that Mantle is only reached through a verified selector. - Shared model catalogs: upstream's Haiku 5.5 source/disclaimer sentences merged into this branch's longer evidence strings; this branch's path rows (runtime_openai_*, guardrails/prompt-caching flags) kept where upstream only reformatted. Haiku 5.5 gets the rows this branch requires on every model: runtime_messages (anthropic catalog) and guardrails_supported / prompt_caching_supported on runtime_converse (openai catalog), both from the Haiku 5.5 model card's bedrock-runtime feature table (read 2026-11-10). - Two Haiku-filter tests now assert the runtime default selects Haiku 5.5 and the Mantle (selector) branch still lands on Sonnet 5. - test_astra_recommendation_probes_selected_api_in_supported_regions uses the surface-neutral mantle_only_model_or_api selector; server_side_tools is Responses-only and turns a Chat source into a decision (PR 588 F4). Also fixes the MD032 markdownlint error in bedrock-migration-assessment discover-app-code.md that failed both CI checks.
Problem
The shared AI mapping files still present models as current targets that AWS, OpenAI, or Google have since retired, gated, or scheduled for shutdown. A customer following the plugin today is told a Legacy model is
active, is offered a model they cannot get access to as the "Best Bedrock Match", or is priced against a source model that stops being billable next month.Read 2026-11-09:
google.gemma-3-4b-it,-12b-it,-27b-it,minimax.minimax-m2, andminimax.minimax-m2.1are Legacy since 2026-09-30 with EOL 2027-03-30.bedrock-pricing-cache.mdmarks all fiveactive, andai-model-lifecycle.md's Legacy table omits them. The DeepSeek-V3.1 card lists eight Regions; the cache says "Sydney only".active;ai-openai-to-bedrock.mdnames it the cross-family target for GPT-5.5 Pro, 5.4 Pro, 5.2 Pro, 5 Pro, o1, o1-pro, o3-pro;ai-gemini-to-bedrock.mdmaps Gemini 3.1 Pro to it; the lifecycle table names it as DeepSeek-R1's and Nova Premier's replacement; gcp-to-aws Clarify Q19 and the Discover preview offer it. The pricing cache's ownStatusvocabulary hasrestricted (gated preview)for exactly this case.ai-openai-to-bedrock.mdrowo3 → DeepSeek-R1andai-gemini-to-bedrock.mdrowsGemini 3 Pro → Llama 4 Maverick / Scoutuse Legacy models as primary targets, whichai-model-lifecycle.mdRule 3 forbids. Gemini deprecations also showsgemini-3-pro-previewshut down 2026-03-09.gpt-5,gpt-5-mini,gpt-5-nano,gpt-5-pro,o3,o3-proshut down 2026-12-11 (32 days out, inside the file's own 90-day window);gpt-4.1,gpt-4.1-mini,gpt-4o-minion 2027-01-14;gpt-5.1on 2027-04-01. Only the GPT-4o row carried a shutdown note.gemini-3.5-flashhas been removed from AI Studio and Developer API requests auto-route togemini-3.6-flash(which Vertex AI retires 2026-11-19);gemini-3.7-flashroutes togemini-3.8-flash. 3.5 Flash is still GA on Vertex AI (no sooner than 2027-05-19). Google names 3.8 Flash as the upgrade for all of them.gemini-4-argon(released 2026-10-13, $2/$10 through 2026-11-30 and $4/$20 from 2026-12-01) is Google's current flagship and appears nowhere in the plugin. The Gemini guide calls 3.5 Flash "the current flagship Flash model" and tells 1.5 Flash users to migrate to it.Fix
Bring the shared files, their vendored copies, and the gcp-to-aws / bedrock-migration-assessment / agent-advisor phase files that repeat these facts into line with the sources above. Prices are unchanged.
bedrock-pricing-cache.md: Gemma 3 ×3 and MiniMax M2/M2.1 →legacy (EOL Mar 30, 2027); DeepSeek-V3.1 drops "Sydney only"; Nova 2 Pro →restricted (gated preview — Nova Forge early access only; no model card); the Nova 2 multimodal table says the same.ai-model-lifecycle.md: the five Legacy rows are added with Active replacements (Gemma 4 on Mantle / Nova, MiniMax M2.5); days-to-EOL recomputed for 2026-11-09; Nova 2 Pro is no longer a named replacement for DeepSeek-R1 or Nova Premier; the Gemini source-model section gains 3.5 / 3.6 / 3.7 Flash routing rows; the notes list the AWS Legacy rows this registry does not price so a reader checks the card.ai-openai-to-bedrock.md: the Pro and o-series cross-family rows target Claude Opus 5.5 (Global, $4/$20) with deltas recomputed at the table's 150M/75M volume (GPT-5.5 Pro 88%, GPT-5 Pro 81%, o1-pro 97%, o3-pro 77%, o1 69%);o3 → Claude Sonnet 5(source 14% cheaper); every row whose source has an announced shutdown carries the date and OpenAI's named replacement; the intro says why Opus replaces Nova 2 Pro and that Global pricing needs worldwide routing.ai-gemini-to-bedrock.md(shared and the gcp-to-aws adapter-local copy, kept in parity): Competitive Reality rewritten for November 2026 (Gemini 4 Argon, 3.8 Flash, 3.5 Flash routing); Gemini 4 Argon Pro-tier rows against Sonnet 5 and Opus 5.5 at both the 2026 and the Dec 1 rates;Gemini 3.1 Pro → Nova 2 Litereplaces the Nova 2 Pro row; the three Gemini 3 Pro rows are removed; 3.5 Flash rows keep the first cellGemini 3.5 Flash (GA)(the preview/Clarify contract tests key on it) and note the Vertex-vs-Developer-API billing; Flash 1.5 → "Migrate to Gemini 3.8 Flash"; Llama 4 and DeepSeek-R1 rows in the quick table are marked Legacy, Nova 2 Pro restricted, Nova Premier past EOL; volume and cost-comparison tables use Nova 2 Lite / Nova Pro instead of Llama 4;text-bison → Nova Micro.clarify-ai.mdQ19:Gemini 4 Argon / 3.1 Prorow with Sonnet 5 / Nova 2 Lite; GPT-5.x Pro row offers Opus 5.5; o-series row drops DeepSeek-R1 and notes the o3 shutdown; ultra-long-context row drops Llama 4 and Nova 2 Pro.design-ai.mdtiered strategy uses Nova Lite / Nova 2 Lite.discover-preview.mdPro row → Opus 5.5; the 3.5 Flash pattern row also matchesgemini-3.6-flash.clarify-ai-only.md(gcp and bedrock-migration-assessment) Pro row → Opus 5.5.gemini-3.5-flash→gemini-3.8-flashin gcp-to-aws Clarify files, bedrock-migration-assessment Clarify, agent-advisor Discover, and the OpenRouter guide.tests/test_legacy_and_restricted_targets.pypins the five statuses, the Nova 2 Prorestrictedstatus, their presence in the Legacy table, and that no Legacy or restricted model sits in the target column of the OpenAI cross-family table or the Gemini live tiers.Left in place on purpose:
openai_model_recommendation.pyandopenai-bedrock.json. Haiku 4.5 is Active (card: "Model EOL date: N/A"); moving to Haiku 5.5 is a separate selector change.Scope
skills/shared/ai/bedrock-pricing-cache.mdskills/shared/ai/ai-model-lifecycle.mdskills/shared/ai/ai-openai-to-bedrock.mdskills/shared/ai/ai-gemini-to-bedrock.mdskills/shared/ai/ai-openrouter-to-bedrock.mdskills/gcp-to-aws/references/design-refs/ai-gemini-to-bedrock.mdskills/gcp-to-aws/references/phases/clarify/{clarify-ai,clarify-ai-only,clarify-assemble}.mdskills/gcp-to-aws/references/phases/design/design-ai.mdskills/gcp-to-aws/references/phases/discover/discover-preview.mdskills/bedrock-migration-assessment/references/phases/clarify/clarify-ai-only.mdskills/agent-advisor/references/phases/discover/discover.mdskills/{gcp-to-aws,azure-to-aws,agent-advisor}/references/vendored/ai/*(regenerated bytools/sync-vendored.py)tests/test_legacy_and_restricted_targets.py(new)No catalog JSON, script, or Terraform change. azure-to-aws changes only through its vendored copies. Does not overlap the GPT-5.4/5.5 endpoint refresh in #583 except for shared touch points in
clarify-ai-only.md,discover-preview.md, and the vendoredbedrock-pricing-cache.md.Validation
mise -C plugins/aws-startup-advisor run ci: all tasks pass (lint:md 0 errors; lint:model-id, lint:model-staleness --strict, lint:vendored-parity, lint:artifacts, lint:skill-refs, lint:consent-disclosure, lint:startup-model-pricing PASS; test 1415 passed; test:telemetry 311 passed; test:heroku-source-review pass).tools/model-staleness.py --strict: registry fresh and in sync.tools/sync-vendored.py --check: 62 vendored files byte-identical.uv run --no-project python scripts/model-id-lint.py: OK, 19 retired ids enforced.tools/model-catalog-drift.py: unchanged pre-existing warnings (openai-bedrock.json 46 days old; Sonnet 5.5 / Haiku 5.5 / GPT-6.1 Terra missing from catalogs). Not addressed here.pytest tests -q: 1415 passed, 222 subtests passed.markdownlint-cli2over the plugin: 0 errors.python3 tools/validate.py(repo root): all validations passed.Type of Change
Select at least one.
Startup Advisor review checklist
By submitting this PR, I confirm that:
mise -C plugins/aws-startup-advisor run ciand all checks pass.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Made with Kiro