Repository navigation
ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist - #387
Merged
Merged
Conversation
added 2 commits
September 30, 2026 21:13
…artifacts against their published schemas
The migration skills pass ~17 JSON artifacts between phases, each with a
contract published as a schema-*.md "shape" block (JSON/JSONC example) or,
for two of them, a real JSON Schema under skills/shared/. Nothing checked
that a producer writes what the next phase reads: a key could be renamed,
emitted without being documented, or an enum widened, and every DSL
_assert still passed because the model that writes the artifact evaluates
the assertion. This is the precondition the backlog names for any engine
consolidation or AI-engine extraction — moves across adapters have no
drift detection without it.
scripts/validate-artifacts.py (stdlib, Python 3.9+)
- Parses each contract document's example blocks (JSONC comments, trailing
commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments,
`// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open
object, `"<key>": {...}` = template for every key) into a key-path
contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`)
refine paths; manifest can pin a block by `@L<line>` anchor, merge other
documents' additions (graviton, workshop), or declare a row-annotation
vocabulary (azure preferences' source/reason/context/...).
- Validates real JSON Schemas with a draft-07 subset (type, required,
properties, additionalProperties, enum, const, items, pattern, min/max,
minItems, allOf/anyOf/oneOf, $ref).
- Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION,
SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken),
NO_CONTRACT (info). `_`-prefixed keys are annotations.
- Modes: --self-check (contracts parse), --fixtures (CI; goldens against
the baseline), --run-dir <.migration/run> --skill <name> (users).
- scripts/artifact-contracts.json maps artifact → contract per skill.
- scripts/artifact-contracts-baseline.json: 21 explained entries covering
44 known gaps between what phases emit and what their contract documents
say, each naming the document that must change. Stale entries are
reported so the list only shrinks.
Contract documents fixed where the document contradicted itself (its own
prose named a key its example block omitted):
- gcp schema-preferences.md: the shape block was not JSON (`{ "value",
"chosen_by" }` set notation); rewritten as a JSONC row template.
- gcp schema-estimate-infra.md: root block lacked complexity_tier, which
estimation-infra.schema.json marks required.
- gcp schema-discover-iac.md § Live discovery extensions: block now shows
the metadata/resources[] additions the prose above it documents.
- heroku schema-discover-heroku.md: resources[] example carries
source/unmanaged_by_terraform/not_found_live (table at lines 93-95);
addon config example carries the per-add-on and live-enrichment fields
the bullets below it list.
- azure schema-design-aws.md: services[] block carries
routing_provenance/sizing_provenance/sizing_source/is_compute_unit/
hosted_app_azure_ids (the same document marks them REQUIRED); deferred[]
carries routing_provenance.
- azure schema-discover-azure.md: clusters[] carries pattern_status
(discover-assemble.md § Status).
- azure schema-preferences.md: top-level clarify_status (REQUIRED per
clarify-assemble.md § clarify_status).
Fixture: gcp-decision-gate's estimation-infra.json stub gains the four
keys estimation-infra.schema.json requires; check_expected_decide.py
still passes.
Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it).
README scripts/ entry updated.
Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping,
template inference (enum/required/placeholder/any-key), each finding
code, JSON Schema subset, baseline matching + stale detection, CLI
self-check and fixtures pass on the real repo, and two negatives: an
injected off-contract key and a schema enum violation both exit 1.
…(and reject unknown schema keywords); stale baseline fails Review on aws#385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE.
6 of 40 tasks
herosjourney
commented
Oct 1, 2026
herosjourney
left a comment
herosjourney
left a comment
Contributor
Author
There was a problem hiding this comment.
Reviewed: head 5cd405e (base ec0fa3d). This commit is 4 files (+521/−1): tools/check-skill-refs.py, tools/skill-refs-baseline.json, tools/test_check_skill_refs.py, mise.toml. The PR also contains #385 (befd32f) and #386 (c6f0c23, plus 3e02e8e).
Ran: python3 tools/check-skill-refs.py --check --json → 1118 references, 0 missing, 7 baselined, 0 stale. --no-baseline → 39 missing, of which 32 are the generated scripts/*.sh the ignore rule names and 2 are fixtures/data.json. uv run --with pytest → 8 passed. Stale-entry probe (one never-matching baseline row) → exit 1. Agent-resolution probe below. git merge-tree --write-tree upstream/main HEAD → CLEAN. Pairwise: #349 #381 #382 #383 #385 #386 CLEAN; #384 CONFLICTS.
Did not run: mise run build locally (GitHub build passed on this head). No live AWS calls.
Cross-plugin parity: N/A. This repo is one tree under plugins/aws-startup-advisor; there is no migrate/ twin. The new files live in tools/, outside skills/.
Change narrative: A phase file can name a path the agent will load, and nothing checked the file exists → check-skill-refs.py resolves path tokens in skills/**/*.md and agents/*.md and fails on unbaselined misses, with a shrinking baseline → surfaces are the lint, mise validate, and the seven baselined dead refs. Body vs diff: the 1118 / 7 / 8-tests figures match this head. The body says agents/ resolve against llm-to-bedrock; the resolver searches every skill and keeps the first hit. See Should-fix.
| Axis | Stance | One line |
|---|---|---|
| Parsimony | clear | One stdlib checker, one baseline, one mise task, in the same shape as the artifact and vendored gates. |
| Completeness | finding | Agent paths accept any skill's file, so a missing llm-to-bedrock target can pass. |
| Security | clear | No credentials or IaC. This gate fails closed on missing and on stale (probed, exit 1). |
| Alternatives | advisory | Baselining the seven dead refs, rather than fixing them here, matches the body's scope. |
| Contradictions | finding | The comment above the agent loop says llm-to-bedrock; the loop does not. |
| Overlap | clear | Path existence is not artifact-schema or vendored-byte identity. |
| Cross-plugin parity | clear | Single tree. |
| Propagation & data | clear | The seven baseline entries are the seven live misses, at the lines the tool reports. The ignore rule swallows only the generated shell scripts it names. |
Blocking
None.
Should-fix
- [Completeness] Agent references resolve to whichever skill sorts first, not to llm-to-bedrock —
tools/check-skill-refs.py:165# agents are llm-to-bedrock's subagents; their paths are relative to that skill's
for other in sorted((plugin / "skills").iterdir()):
On this head,agents/llm2bedrock-prompt-evaluator.md:233`scripts/pyproject.toml`is reported resolved atskills/agent-advisor/scripts/pyproject.tomlbecauseagent-advisorsorts beforellm-to-bedrock. The llm-to-bedrock copy exists too, so today's corpus is not a hidden dead path. A tree where that file exists only underagent-advisorstill resolves (probed:RESOLVED, notMISSING). Deleting the llm-to-bedrock copy would not fail the gate.
Fix: when the source is underagents/, candidate list is onlyskills/llm-to-bedrock(the<BDD_DIR>rglob can stay, scoped to that skill). Add a test:scripts/pyproject.tomlpresent only underagent-advisorisMISSING; present underllm-to-bedrockresolves there even ifagent-advisoralso has it.
Heads-up (out of this commit — parent commits this PR still contains)
- #385 is still open inside this head.
plugins/aws-startup-advisor/scripts/validate-artifacts.py:884isreturn 1 if errors else 0, and a stale baseline is printed aswarnand does not fail.minPropertiesis still not implemented (no match in that file). Same notes as the review on #385. - #386's README overclaim is fixed at
3e02e8e: the vendored READMEs now say a newskills/shared/file that nobody vendors is what CI cannot see. git merge-treeagainst #384 still conflicts (the #385schema-preferences.mdedit). #349 and #381–#383 merge clean with this head.
Alternatives (max 2)
- Fix the seven dead refs in this PR — wins: the baseline starts empty; costs: Bicep/ARM and the pattern catalog are design decisions the body correctly defers. Recommendation: keep the baseline.
Verdict
Comment — one should-fix on agent resolution; nothing blocking in this commit. Commenting rather than approving: author == reviewer identity. Code owners decide. The #385 should-fixes above are still in the stack.
herosjourney
pushed a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 1, 2026
…inst llm-to-bedrock Review on aws#387 (Completeness / Contradictions, should-fix): the agents/ branch of the resolver searched every skill and kept the first hit, so agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml` resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a tree with the file only under agent-advisor still passed — deleting the llm-to-bedrock copy would not have failed the gate. The comment above the loop said llm-to-bedrock; the loop did not. - Agent references now resolve ONLY against skills/llm-to-bedrock (_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to that skill. - Tests: scripts/pyproject.toml present only under agent-advisor is MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy even when agent-advisor also has it; the fixture tree's agents/worker.md reference to another skill's file is now correctly MISSING. - Real repo unchanged: 1118 references, 0 missing, 7 baselined.
added 4 commits
October 1, 2026 09:09
…ndored/ must match skills/shared/ Each migration skill is installable on its own (`npx skills add … --skill gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate schemas, …) is vendored into skills/<skill>/references/vendored/ from the canonical skills/shared/. The vendored READMEs said: "This repository has no automated sync task for these copies — keeping them byte-identical is part of the change that touches the canonical file." All 44 copies are identical today; nothing stopped the next edit to skills/shared/ from leaving a skill running stale logic. - tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py. Default mode copies skills/shared/<path> over every vendored copy that differs; --check exits 1 listing any copy that differs or has no canonical source (a vendored file nobody owns). README.md in a vendored dir is explanatory and skipped. The vendored tree is a subset of shared/, so files in shared/ that no skill vendors are fine. - mise `lint:vendored-parity`, added to `validate` (the CI build runs it). - The four vendored READMEs replace the manual cp + md5sum instructions with the tool and the CI rule. - tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and orphan detection (README excluded), sync repairs drift and leaves orphans as errors, CLI exit codes — plus the real repository is in sync. No skill content changes. Stacked on aws#385 only because both add a task to the same `validate` depends line.
…t; deleted copies fail; READMEs say what the gate sees Review on aws#386 (Contradictions / Completeness, should-fix): the three migration READMEs claimed "a change to skills/shared/ that forgets the copies cannot merge", but the check only walked files that still existed — a deleted vendored copy, or a new shared file never vendored, passed. Rather than only weakening the sentence, the gate now uses the record of intent each vendored README already carries: its `| Vendored path | Canonical source |` table. - check: every row must exist on disk (deleted copy → FAIL), every file on disk must have a row (README stays honest), and each row's canonical path must be skills/shared/<same path>. A README with no table is an error. - sync: restores a listed-but-missing copy from canonical before comparing. - What the gate still cannot see — a NEW skills/shared/ file no skill vendors — is now stated in the tool docstring and in all four READMEs, with the instruction "add the copy AND a table row". The three READMEs were already complete tables (gcp 14/14, heroku 8/8, azure 21/21, agent-advisor 1/1); the real repo passes unchanged. Tests (7): deleted copy caught via the table and restored by sync; unlisted file and wrong canonical path caught; a new shared file nobody vendors is NOT an error (pins the documented limit); CLI exit codes; real repo in sync. Reviewer probes re-run on a throwaway copy of the plugin: deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode restores it; adding skills/shared/brand-new.md → no error, as documented.
…l mentions must exist
The skills are prose state machines: a phase file says "Load
`references/phases/design/design-ai.md`" or "run `python3
\"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing
checked the target exists. A renamed or never-written file is a dead end
the user hits mid-migration, and any file move during consolidation would
break references silently. This is the third gate in the family
(validate-artifacts.py: artifact contracts; sync-vendored.py: vendored
parity) and the precondition for moving phase files between skills.
tools/check-skill-refs.py (stdlib)
- Scans skills/**/*.md and agents/*.md for path tokens into the skill
tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the
agent-advisor short forms phases/, shared/, design-refs/) with optional
$PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> /
$GCP_BASE / ./ / ../ prefixes.
- Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form> → plugin/repo-relative. Library-origin
files (skills/shared/*, */references/vendored/*, */references/shared/*)
speak from the consuming skill's point of view and resolve against
skills/shared and every skill; agent-advisor and llm-to-bedrock may name
gcp-to-aws's tree (documented inline execution); agents/ resolve against
llm-to-bedrock.
- Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…),
templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no
skill-tree root in them (generated outputs like ./deploy.sh).
- tools/skill-refs-baseline.json: `ignore` rules (permanent policy —
scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in
the user's app) and `entries` (known dead references with the fix;
STALE when they stop matching, failing the gate so the list only shrinks).
1118 references checked on the real repo; 7 dead ones baselined, each a
real user-facing defect: azure discover-iac.md points Bicep/ARM input at
extract-bicep.md / extract-arm.md that do not exist (so Bicep halts);
design-refs/patterns.md referenced from two azure files before it exists;
contextual-offers SKILL.md names references/offers.md unqualified when the
file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at
shared/ai-model-lifecycle.md, which in gcp is vendored at
references/vendored/ai/.
mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution
rule, each skip rule, library-origin resolution, gcp-dependent and agents
fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS.
…inst llm-to-bedrock Review on aws#387 (Completeness / Contradictions, should-fix): the agents/ branch of the resolver searched every skill and kept the first hit, so agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml` resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a tree with the file only under agent-advisor still passed — deleting the llm-to-bedrock copy would not have failed the gate. The comment above the loop said llm-to-bedrock; the loop did not. - Agent references now resolve ONLY against skills/llm-to-bedrock (_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to that skill. - Tests: scripts/pyproject.toml present only under agent-advisor is MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy even when agent-advisor also has it; the fixture tree's agents/worker.md reference to another skill's file is now correctly MISSING. - Real repo unchanged: 1118 references, 0 missing, 7 baselined.
herosjourney
force-pushed
the
feat/skill-refs-lint
branch
from
October 1, 2026 16:09
7f35023 to
990fd39
Compare
Contributor
Author
|
Review finding addressed — ready for re-review.
Heads-up items: the stack is re-based. #386 and #387 now sit on #385's current head |
leon1418
requested changes
Oct 2, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed head 06ecee29062af1b7778a0b11dc79595084e28ab3 against acc890da1028c9c9e7c4d496238789d4d77f201d, covering all 23 changed files, including the stacked validation changes. Seven P2 issues were reproduced on the original head and are detailed inline.
Consolidated correction list:
- Scope the default fixture baseline so valid real runs succeed.
- Map GCP scenario deltas and named cost entries to the existing object contract.
- Accept Azure's documented composite discovery-source values.
- Preserve declared enums through example merging and named preference refinements.
- Keep separate references after run-artifact tokens in the scan.
- Respect explicit path bases before applying cross-skill fallbacks.
- Enforce Heroku's three explicitly required top-level inventory sections.
The affected implementation paths are scripts/validate-artifacts.py, scripts/artifact-contracts.json, skills/gcp-to-aws/references/shared/schema-preferences.md (and estimate section selection if needed) under plugins/aws-startup-advisor/, plus tools/check-skill-refs.py. Add focused coverage in the two corresponding test files. Each inline comment includes the failure case and valid behavior to preserve.
Validation: all 35 authored tests and all six registered validation tasks pass; git diff --check passes. The additional original-head probes reproduce the seven defects above. Existing agent-owner resolution and stale-baseline regression tests also pass. GitHub checks were successful at the reviewed head. No cloud or deployed execution was performed.
…for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on aws#386/aws#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale.
… token; explicit bases are authoritative Review round 2 on aws#387 (the two findings unique to this tool): - The run-artifact skip looked 40 characters back, so an earlier, separate token (`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line (azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…` forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve). - An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md` traversal still resolves. Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing, 7 baselined, 0 stale.
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # plugins/aws-startup-advisor/skills/azure-to-aws/references/shared/schema-preferences.md
…ferences example The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys. Co-authored-by: Cursor <cursoragent@cursor.com>
leon1418
requested changes
Oct 3, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Completed the hybrid follow-up at f7a3ee35b4c20aad9b61d8c6e1ff7382ec777c66 against 1de6d7a92f4bfc114bc74eae3587619716328402, covering all 23 changed files and their affected contracts. Six original concerns are verified fixed. The explicit-relative-path case remains in its original conversation.
Consolidated correction list:
- In
scripts/validate-artifacts.pyand its tests, refine the unspecified item type from an empty example array when a later section declares object entries. Reject scalar/array entries in the refined Azure services/clusters/deferred/pending_rubric arrays while retaining empty arrays, valid objects and truly unspecified item shapes. - In
tools/check-skill-refs.pyand its tests, preserve the complete./or../prefix without the unprefixed skill-root fallback. The valid Azure knowledge link must pass; removing one parent traversal must report the missing literal target.
The array-item defect existed at the previous reviewed head and I missed it in the earlier review; it is detailed inline. The relative-path issue is a remaining case of the original explicit-prefix finding, so no duplicate thread is added.
Validation: 52 authored tests, all six registered validation tasks, contract self-check and the decision-gate fixture pass. Both remaining failures were independently reproduced against the current raw source and compared with the previous reviewed source. GitHub checks were successful at the captured head. No cloud or deployed execution was performed. Approval remains for the user to submit.
…relative prefixes An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root. Co-authored-by: Cursor <cursoragent@cursor.com>
leon1418
requested changes
Oct 3, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖] Re-reviewed e086f5108fa1355b686a09370bfa9f1a8c0c6f85 with OCR delegation, full coverage of all 23 changed files, an isolated general reviewer, repository review and the complete convergence gate. The relative-path and array-item concerns are verified fixed.
Four additional P2 cases remain. They already existed at f7a3ee35; I missed them in the earlier review. The consolidated corrections are:
- Azure preferences: represent AI-only and mixed AI layouts in the shape/mapping, retaining route-specific requiredness and invalid-key/type rejection.
- GCP drift records: accept the original JSON types for terraform_value/live_value, while keeping the surrounding record checks.
- Shared vendored-parity CI: guard an absent skills directory before iteration in both check/sync modes; preserve existing parity and restoration checks.
- Azure workshop index: accept required fingerprint/active-ID/cap metadata for baseline and active indexes, preserving the existing scenario store and rejecting unrelated keys. No new snapshot-contract coverage is required.
All 55 authored pytest tests and all six registered validation tasks pass; the independent producer-derived probes expose these uncovered cases. The general reviewer also executed all 55 plain test functions through a recorded harness because pytest was unavailable in its environment. No source changes, approval or review dismissal were performed.
…d drift values An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate.
…t-wt # Conflicts: # plugins/aws-startup-advisor/README.md
Co-authored-by: Cursor <cursoragent@cursor.com>
herosjourney
disabled auto-merge
October 9, 2026 04:55
# Conflicts: # plugins/aws-startup-advisor/README.md # plugins/aws-startup-advisor/skills/azure-to-aws/references/shared/schema-preferences.md # plugins/aws-startup-advisor/skills/heroku-to-aws/references/shared/schema-discover-heroku.md
…skill-refs-lint # Conflicts: # plugins/aws-startup-advisor/README.md # plugins/aws-startup-advisor/mise.toml
# Conflicts: # plugins/aws-startup-advisor/mise.toml
leon1418
previously approved these changes
Oct 12, 2026
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 12, 2026
…phase artifacts (aws#385) * feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas The migration skills pass ~17 JSON artifacts between phases, each with a contract published as a schema-*.md "shape" block (JSON/JSONC example) or, for two of them, a real JSON Schema under skills/shared/. Nothing checked that a producer writes what the next phase reads: a key could be renamed, emitted without being documented, or an enum widened, and every DSL _assert still passed because the model that writes the artifact evaluates the assertion. This is the precondition the backlog names for any engine consolidation or AI-engine extraction — moves across adapters have no drift detection without it. scripts/validate-artifacts.py (stdlib, Python 3.9+) - Parses each contract document's example blocks (JSONC comments, trailing commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments, `// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open object, `"<key>": {...}` = template for every key) into a key-path contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`) refine paths; manifest can pin a block by `@L<line>` anchor, merge other documents' additions (graviton, workshop), or declare a row-annotation vocabulary (azure preferences' source/reason/context/...). - Validates real JSON Schemas with a draft-07 subset (type, required, properties, additionalProperties, enum, const, items, pattern, min/max, minItems, allOf/anyOf/oneOf, $ref). - Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION, SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken), NO_CONTRACT (info). `_`-prefixed keys are annotations. - Modes: --self-check (contracts parse), --fixtures (CI; goldens against the baseline), --run-dir <.migration/run> --skill <name> (users). - scripts/artifact-contracts.json maps artifact → contract per skill. - scripts/artifact-contracts-baseline.json: 21 explained entries covering 44 known gaps between what phases emit and what their contract documents say, each naming the document that must change. Stale entries are reported so the list only shrinks. Contract documents fixed where the document contradicted itself (its own prose named a key its example block omitted): - gcp schema-preferences.md: the shape block was not JSON (`{ "value", "chosen_by" }` set notation); rewritten as a JSONC row template. - gcp schema-estimate-infra.md: root block lacked complexity_tier, which estimation-infra.schema.json marks required. - gcp schema-discover-iac.md § Live discovery extensions: block now shows the metadata/resources[] additions the prose above it documents. - heroku schema-discover-heroku.md: resources[] example carries source/unmanaged_by_terraform/not_found_live (table at lines 93-95); addon config example carries the per-add-on and live-enrichment fields the bullets below it list. - azure schema-design-aws.md: services[] block carries routing_provenance/sizing_provenance/sizing_source/is_compute_unit/ hosted_app_azure_ids (the same document marks them REQUIRED); deferred[] carries routing_provenance. - azure schema-discover-azure.md: clusters[] carries pattern_status (discover-assemble.md § Status). - azure schema-preferences.md: top-level clarify_status (REQUIRED per clarify-assemble.md § clarify_status). Fixture: gcp-decision-gate's estimation-infra.json stub gains the four keys estimation-infra.schema.json requires; check_expected_decide.py still passes. Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it). README scripts/ entry updated. Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping, template inference (enum/required/placeholder/any-key), each finding code, JSON Schema subset, baseline matching + stale detection, CLI self-check and fixtures pass on the real repo, and two negatives: an injected off-contract key and a schema enum violation both exit 1. * fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails Review on aws#385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE. * fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on aws#386/aws#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale. * fix(aws-startup-advisor): close the remaining artifact-contract review gaps Empty-array seeds, quoted enums, and a workloads object were slipping past the shape check, and several producer fields the contracts already describe were rejected as unknown. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): check workshop snapshots against the active contracts Scenario copies were skipping the active shape or had no contract, and the documented cache category, scenario outcome, and Azure discovery verdict were rejected or optional. * fix(aws-startup-advisor): define lint:artifacts in a plugin-local mise-tasks.toml Root mise.toml now sets [task_config].includes once, pointing at plugins/aws-startup-advisor/mise-tasks.toml (alongside mise's default file-task directories, kept explicit so they are not silently dropped). This is the one repo-root touch a plugin needs to register its tasks; the lint:artifacts task's own description and run command now live in that plugin-local file instead of root mise.toml's [tasks] block. Future plugin-owned tasks only need an entry in that plugin-local file — no further root mise.toml edits required. depends chains elsewhere (validate -> lint:artifacts) are unaffected since task_config.includes merges included tasks into the same task namespace. Verified: mise tasks info lint:artifacts resolves to the plugin-local file; mise run lint:artifacts, mise run validate, and mise run build all pass; pytest on test_validate_artifacts.py (46/46) passes. * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs the script directly Root mise.toml now matches main exactly — no plugin-owned task registration, include path, or dependency lives there. The lint:artifacts task is defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience (run from inside the plugin directory). CI enforcement no longer goes through mise at all: a new dedicated .github/workflows/artifact-contracts.yml (same shape as the existing model-staleness.yml precedent from PR aws#348) invokes plugins/aws-startup-advisor/scripts/validate-artifacts.py directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. This mirrors how model-staleness.py's CI gate already bypasses mise.toml entirely. Net effect: root mise.toml has zero diff against main, and nothing outside the plugin reads, runs, or depends on anything this PR adds. The one remaining file outside the plugin is the GitHub Actions workflow file itself, which GitHub requires to live under .github/workflows/ — it only invokes the plugin's own script and carries no shared logic. * refactor(aws-startup-advisor): move artifact-contracts check into plugin dispatcher Drop the top-level .github/workflows/artifact-contracts.yml workflow and wire lint:artifacts into plugins/aws-startup-advisor/mise.toml's ci task instead, so the check runs via the already-merged startup-advisor dispatcher (upstream#423) without touching anything outside the plugin folder. --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # plugins/aws-startup-advisor/README.md # plugins/aws-startup-advisor/mise.toml # plugins/aws-startup-advisor/scripts/artifact-contracts.json # plugins/aws-startup-advisor/scripts/validate-artifacts.py # plugins/aws-startup-advisor/skills/gcp-to-aws/references/shared/schema-discover-iac.md # plugins/aws-startup-advisor/tests/test_validate_artifacts.py
leon1418
previously approved these changes
Oct 12, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 12, 2026
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 12, 2026
…must match skills/shared/ (aws#386) * feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas The migration skills pass ~17 JSON artifacts between phases, each with a contract published as a schema-*.md "shape" block (JSON/JSONC example) or, for two of them, a real JSON Schema under skills/shared/. Nothing checked that a producer writes what the next phase reads: a key could be renamed, emitted without being documented, or an enum widened, and every DSL _assert still passed because the model that writes the artifact evaluates the assertion. This is the precondition the backlog names for any engine consolidation or AI-engine extraction — moves across adapters have no drift detection without it. scripts/validate-artifacts.py (stdlib, Python 3.9+) - Parses each contract document's example blocks (JSONC comments, trailing commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments, `// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open object, `"<key>": {...}` = template for every key) into a key-path contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`) refine paths; manifest can pin a block by `@L<line>` anchor, merge other documents' additions (graviton, workshop), or declare a row-annotation vocabulary (azure preferences' source/reason/context/...). - Validates real JSON Schemas with a draft-07 subset (type, required, properties, additionalProperties, enum, const, items, pattern, min/max, minItems, allOf/anyOf/oneOf, $ref). - Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION, SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken), NO_CONTRACT (info). `_`-prefixed keys are annotations. - Modes: --self-check (contracts parse), --fixtures (CI; goldens against the baseline), --run-dir <.migration/run> --skill <name> (users). - scripts/artifact-contracts.json maps artifact → contract per skill. - scripts/artifact-contracts-baseline.json: 21 explained entries covering 44 known gaps between what phases emit and what their contract documents say, each naming the document that must change. Stale entries are reported so the list only shrinks. Contract documents fixed where the document contradicted itself (its own prose named a key its example block omitted): - gcp schema-preferences.md: the shape block was not JSON (`{ "value", "chosen_by" }` set notation); rewritten as a JSONC row template. - gcp schema-estimate-infra.md: root block lacked complexity_tier, which estimation-infra.schema.json marks required. - gcp schema-discover-iac.md § Live discovery extensions: block now shows the metadata/resources[] additions the prose above it documents. - heroku schema-discover-heroku.md: resources[] example carries source/unmanaged_by_terraform/not_found_live (table at lines 93-95); addon config example carries the per-add-on and live-enrichment fields the bullets below it list. - azure schema-design-aws.md: services[] block carries routing_provenance/sizing_provenance/sizing_source/is_compute_unit/ hosted_app_azure_ids (the same document marks them REQUIRED); deferred[] carries routing_provenance. - azure schema-discover-azure.md: clusters[] carries pattern_status (discover-assemble.md § Status). - azure schema-preferences.md: top-level clarify_status (REQUIRED per clarify-assemble.md § clarify_status). Fixture: gcp-decision-gate's estimation-infra.json stub gains the four keys estimation-infra.schema.json requires; check_expected_decide.py still passes. Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it). README scripts/ entry updated. Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping, template inference (enum/required/placeholder/any-key), each finding code, JSON Schema subset, baseline matching + stale detection, CLI self-check and fixtures pass on the real repo, and two negatives: an injected off-contract key and a schema enum violation both exit 1. * fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails Review on aws#385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE. * ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/ Each migration skill is installable on its own (`npx skills add … --skill gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate schemas, …) is vendored into skills/<skill>/references/vendored/ from the canonical skills/shared/. The vendored READMEs said: "This repository has no automated sync task for these copies — keeping them byte-identical is part of the change that touches the canonical file." All 44 copies are identical today; nothing stopped the next edit to skills/shared/ from leaving a skill running stale logic. - tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py. Default mode copies skills/shared/<path> over every vendored copy that differs; --check exits 1 listing any copy that differs or has no canonical source (a vendored file nobody owns). README.md in a vendored dir is explanatory and skipped. The vendored tree is a subset of shared/, so files in shared/ that no skill vendors are fine. - mise `lint:vendored-parity`, added to `validate` (the CI build runs it). - The four vendored READMEs replace the manual cp + md5sum instructions with the tool and the CI rule. - tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and orphan detection (README excluded), sync repairs drift and leaves orphans as errors, CLI exit codes — plus the real repository is in sync. No skill content changes. Stacked on aws#385 only because both add a task to the same `validate` depends line. * fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees Review on aws#386 (Contradictions / Completeness, should-fix): the three migration READMEs claimed "a change to skills/shared/ that forgets the copies cannot merge", but the check only walked files that still existed — a deleted vendored copy, or a new shared file never vendored, passed. Rather than only weakening the sentence, the gate now uses the record of intent each vendored README already carries: its `| Vendored path | Canonical source |` table. - check: every row must exist on disk (deleted copy → FAIL), every file on disk must have a row (README stays honest), and each row's canonical path must be skills/shared/<same path>. A README with no table is an error. - sync: restores a listed-but-missing copy from canonical before comparing. - What the gate still cannot see — a NEW skills/shared/ file no skill vendors — is now stated in the tool docstring and in all four READMEs, with the instruction "add the copy AND a table row". The three READMEs were already complete tables (gcp 14/14, heroku 8/8, azure 21/21, agent-advisor 1/1); the real repo passes unchanged. Tests (7): deleted copy caught via the table and restored by sync; unlisted file and wrong canonical path caught; a new shared file nobody vendors is NOT an error (pins the documented limit); CLI exit codes; real repo in sync. Reviewer probes re-run on a throwaway copy of the plugin: deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode restores it; adding skills/shared/brand-new.md → no error, as documented. * fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on aws#386/aws#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale. * fix(aws-startup-advisor): close three artifact and vendored-parity gaps An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): require the Azure discovery verdict and check estimate snapshots A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep vendored-parity tooling plugin-local Move tools/sync-vendored.py and its test into plugins/aws-startup-advisor/tools/, since the vendored/shared pattern it gates is unique to this plugin (no other plugin uses references/vendored/). - Script now resolves PLUGIN_ROOT from its own location instead of scanning every plugins/ subdirectory; dropped the now-unneeded --plugin/--plugins-root multi-plugin CLI surface in favor of a single --plugin-root override for tests. - Updated the test suite to match (plugin-root fixtures instead of a multi-plugin tree; the MCP-only-plugin skip test now asserts a clean exit for a plugin root with no skills/ at all). - Updated mise.toml's lint:vendored-parity task and the four vendored/ README.md usage snippets (agent-advisor, heroku-to-aws, gcp-to-aws, azure-to-aws) to the new path. * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly Root mise.toml now matches main exactly. lint:artifacts and lint:vendored-parity are defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience. CI enforcement bypasses mise entirely, same pattern as the existing model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml (PR aws#385): two dedicated workflows invoke the plugin's scripts directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. Net effect: root mise.toml has zero diff against main. The only files outside the plugin are the two GitHub Actions workflow files themselves, which GitHub requires under .github/workflows/ — each only invokes the plugin's own script and carries no shared logic. * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(aws-startup-advisor): move vendored-parity check into plugin dispatcher --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # plugins/aws-startup-advisor/mise.toml
leon1418
approved these changes
Oct 12, 2026
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 13, 2026
* feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas
The migration skills pass ~17 JSON artifacts between phases, each with a
contract published as a schema-*.md "shape" block (JSON/JSONC example) or,
for two of them, a real JSON Schema under skills/shared/. Nothing checked
that a producer writes what the next phase reads: a key could be renamed,
emitted without being documented, or an enum widened, and every DSL
_assert still passed because the model that writes the artifact evaluates
the assertion. This is the precondition the backlog names for any engine
consolidation or AI-engine extraction — moves across adapters have no
drift detection without it.
scripts/validate-artifacts.py (stdlib, Python 3.9+)
- Parses each contract document's example blocks (JSONC comments, trailing
commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments,
`// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open
object, `"<key>": {...}` = template for every key) into a key-path
contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`)
refine paths; manifest can pin a block by `@L<line>` anchor, merge other
documents' additions (graviton, workshop), or declare a row-annotation
vocabulary (azure preferences' source/reason/context/...).
- Validates real JSON Schemas with a draft-07 subset (type, required,
properties, additionalProperties, enum, const, items, pattern, min/max,
minItems, allOf/anyOf/oneOf, $ref).
- Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION,
SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken),
NO_CONTRACT (info). `_`-prefixed keys are annotations.
- Modes: --self-check (contracts parse), --fixtures (CI; goldens against
the baseline), --run-dir <.migration/run> --skill <name> (users).
- scripts/artifact-contracts.json maps artifact → contract per skill.
- scripts/artifact-contracts-baseline.json: 21 explained entries covering
44 known gaps between what phases emit and what their contract documents
say, each naming the document that must change. Stale entries are
reported so the list only shrinks.
Contract documents fixed where the document contradicted itself (its own
prose named a key its example block omitted):
- gcp schema-preferences.md: the shape block was not JSON (`{ "value",
"chosen_by" }` set notation); rewritten as a JSONC row template.
- gcp schema-estimate-infra.md: root block lacked complexity_tier, which
estimation-infra.schema.json marks required.
- gcp schema-discover-iac.md § Live discovery extensions: block now shows
the metadata/resources[] additions the prose above it documents.
- heroku schema-discover-heroku.md: resources[] example carries
source/unmanaged_by_terraform/not_found_live (table at lines 93-95);
addon config example carries the per-add-on and live-enrichment fields
the bullets below it list.
- azure schema-design-aws.md: services[] block carries
routing_provenance/sizing_provenance/sizing_source/is_compute_unit/
hosted_app_azure_ids (the same document marks them REQUIRED); deferred[]
carries routing_provenance.
- azure schema-discover-azure.md: clusters[] carries pattern_status
(discover-assemble.md § Status).
- azure schema-preferences.md: top-level clarify_status (REQUIRED per
clarify-assemble.md § clarify_status).
Fixture: gcp-decision-gate's estimation-infra.json stub gains the four
keys estimation-infra.schema.json requires; check_expected_decide.py
still passes.
Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it).
README scripts/ entry updated.
Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping,
template inference (enum/required/placeholder/any-key), each finding
code, JSON Schema subset, baseline matching + stale detection, CLI
self-check and fixtures pass on the real repo, and two negatives: an
injected off-contract key and a schema enum violation both exit 1.
* fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails
Review on aws#385, both should-fix findings valid:
1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`;
the draft-07 subset did not implement it, so `phases: {}` passed. Fixed
the instance AND the class: the subset now enforces minProperties,
maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems,
minLength/maxLength and `not`; and `unsupported_keywords()` walks every
loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any
validation keyword that is neither enforced nor a known annotation. A
constraint can no longer be skipped silently — adding a keyword to a
schema without teaching the validator fails the gate.
2. [Completeness] a baseline entry that matched nothing was printed as a
warning and exited 0, so the burn-down list only shrank if someone read
the log. STALE_BASELINE is now an error code: `--fixtures` (what
`mise run lint:artifacts` runs) exits 1 on any stale entry.
Tests (19): empty `phases` fails with minProperties named; the two real
schemas use only enforced/annotation keywords and an invented
`contentEncoding` / `dependencies` schema is rejected; a never-matching
baseline entry exits 1 with STALE_BASELINE.
* ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/
Each migration skill is installable on its own (`npx skills add … --skill
gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate
schemas, …) is vendored into skills/<skill>/references/vendored/ from the
canonical skills/shared/. The vendored READMEs said: "This repository has
no automated sync task for these copies — keeping them byte-identical is
part of the change that touches the canonical file." All 44 copies are
identical today; nothing stopped the next edit to skills/shared/ from
leaving a skill running stale logic.
- tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py.
Default mode copies skills/shared/<path> over every vendored copy that
differs; --check exits 1 listing any copy that differs or has no
canonical source (a vendored file nobody owns). README.md in a vendored
dir is explanatory and skipped. The vendored tree is a subset of shared/,
so files in shared/ that no skill vendors are fine.
- mise `lint:vendored-parity`, added to `validate` (the CI build runs it).
- The four vendored READMEs replace the manual cp + md5sum instructions
with the tool and the CI rule.
- tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and
orphan detection (README excluded), sync repairs drift and leaves orphans
as errors, CLI exit codes — plus the real repository is in sync.
No skill content changes. Stacked on aws#385 only because both add a task to
the same `validate` depends line.
* fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees
Review on aws#386 (Contradictions / Completeness, should-fix): the three
migration READMEs claimed "a change to skills/shared/ that forgets the
copies cannot merge", but the check only walked files that still existed —
a deleted vendored copy, or a new shared file never vendored, passed.
Rather than only weakening the sentence, the gate now uses the record of
intent each vendored README already carries: its
`| Vendored path | Canonical source |` table.
- check: every row must exist on disk (deleted copy → FAIL), every file on
disk must have a row (README stays honest), and each row's canonical
path must be skills/shared/<same path>. A README with no table is an
error.
- sync: restores a listed-but-missing copy from canonical before comparing.
- What the gate still cannot see — a NEW skills/shared/ file no skill
vendors — is now stated in the tool docstring and in all four READMEs,
with the instruction "add the copy AND a table row".
The three READMEs were already complete tables (gcp 14/14, heroku 8/8,
azure 21/21, agent-advisor 1/1); the real repo passes unchanged.
Tests (7): deleted copy caught via the table and restored by sync;
unlisted file and wrong canonical path caught; a new shared file nobody
vendors is NOT an error (pins the documented limit); CLI exit codes; real
repo in sync. Reviewer probes re-run on a throwaway copy of the plugin:
deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode
restores it; adding skills/shared/brand-new.md → no error, as documented.
* ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist
The skills are prose state machines: a phase file says "Load
`references/phases/design/design-ai.md`" or "run `python3
\"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing
checked the target exists. A renamed or never-written file is a dead end
the user hits mid-migration, and any file move during consolidation would
break references silently. This is the third gate in the family
(validate-artifacts.py: artifact contracts; sync-vendored.py: vendored
parity) and the precondition for moving phase files between skills.
tools/check-skill-refs.py (stdlib)
- Scans skills/**/*.md and agents/*.md for path tokens into the skill
tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the
agent-advisor short forms phases/, shared/, design-refs/) with optional
$PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> /
$GCP_BASE / ./ / ../ prefixes.
- Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form> → plugin/repo-relative. Library-origin
files (skills/shared/*, */references/vendored/*, */references/shared/*)
speak from the consuming skill's point of view and resolve against
skills/shared and every skill; agent-advisor and llm-to-bedrock may name
gcp-to-aws's tree (documented inline execution); agents/ resolve against
llm-to-bedrock.
- Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…),
templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no
skill-tree root in them (generated outputs like ./deploy.sh).
- tools/skill-refs-baseline.json: `ignore` rules (permanent policy —
scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in
the user's app) and `entries` (known dead references with the fix;
STALE when they stop matching, failing the gate so the list only shrinks).
1118 references checked on the real repo; 7 dead ones baselined, each a
real user-facing defect: azure discover-iac.md points Bicep/ARM input at
extract-bicep.md / extract-arm.md that do not exist (so Bicep halts);
design-refs/patterns.md referenced from two azure files before it exists;
contextual-offers SKILL.md names references/offers.md unqualified when the
file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at
shared/ai-model-lifecycle.md, which in gcp is vendored at
references/vendored/ai/.
mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution
rule, each skip rule, library-origin resolution, gcp-dependent and agents
fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS.
* fix(aws-startup-advisor): check-skill-refs — agents/ resolve only against llm-to-bedrock
Review on aws#387 (Completeness / Contradictions, should-fix): the agents/
branch of the resolver searched every skill and kept the first hit, so
agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml`
resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a
tree with the file only under agent-advisor still passed — deleting the
llm-to-bedrock copy would not have failed the gate. The comment above the
loop said llm-to-bedrock; the loop did not.
- Agent references now resolve ONLY against skills/llm-to-bedrock
(_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to
that skill.
- Tests: scripts/pyproject.toml present only under agent-advisor is
MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy
even when agent-advisor also has it; the fixture tree's agents/worker.md
reference to another skill's file is now correctly MISSING.
- Real repo unchanged: 1118 references, 0 missing, 7 baselined.
* fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths
Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on
aws#386/aws#387 against the same code):
- The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it
now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no
baseline unless --baseline names one, and then every entry must still match (a valid
real run previously failed with 21 STALE_BASELINE errors).
- Node.merge keeps a declared enum when another example shows a concrete member
(`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document
shows joins the set, only a free-text example clears it. Adopted children are copied,
so refining `design_constraints.cpu_architecture` no longer widens the generic
`<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`.
- `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is
valid, `terraform+bogus` is not, closed enums elsewhere stay closed.
- Heading match is exact or word-boundary prefix: `graviton` no longer selects
`graviton_profile`; the Azure design manifest names `graviton block` explicitly.
- Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the
array level, so workload key/type drift is detected; bare item examples still merge
into the item.
- `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku
`metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged.
Two baseline entries record the hand-authored Heroku seed inventories that lack them.
- GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline
and breakdown.security_baseline_compliance bind to their object keys (line anchors
separate the two Security Baseline blocks); nested drift inside them is still caught.
- Heroku workshop `preferences_subset` is the documented open knob map (open_paths).
15 regression tests added (unit + CLI against the real contracts); each fails on the
previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale.
* fix(aws-startup-advisor): check-skill-refs — run-artifact skip is per token; explicit bases are authoritative
Review round 2 on aws#387 (the two findings unique to this tool):
- The run-artifact skip looked 40 characters back, so an earlier, separate token
(`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line
(azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the
token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…`
forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve).
- An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now
authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a
same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look
alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md`
traversal still resolves.
Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing,
7 baselined, 0 stale.
* fix(aws-startup-advisor): admit the unfired licensing stub in the preferences example
The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): close three artifact and vendored-parity gaps
An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): reject refined empty-array items and honor relative prefixes
An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): require the Azure discovery verdict and check estimate snapshots
A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): accept each Azure preference route and typed drift values
An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate.
* fix: ignore Bedrock run state references
* fix(aws-startup-advisor): keep vendored-parity tooling plugin-local
Move tools/sync-vendored.py and its test into
plugins/aws-startup-advisor/tools/, since the vendored/shared pattern it
gates is unique to this plugin (no other plugin uses references/vendored/).
- Script now resolves PLUGIN_ROOT from its own location instead of scanning
every plugins/ subdirectory; dropped the now-unneeded --plugin/--plugins-root
multi-plugin CLI surface in favor of a single --plugin-root override for
tests.
- Updated the test suite to match (plugin-root fixtures instead of a
multi-plugin tree; the MCP-only-plugin skip test now asserts a clean exit
for a plugin root with no skills/ at all).
- Updated mise.toml's lint:vendored-parity task and the four vendored/
README.md usage snippets (agent-advisor, heroku-to-aws, gcp-to-aws,
azure-to-aws) to the new path.
* fix(aws-startup-advisor): keep skill-refs and vendored-parity tooling plugin-local
Move tools/check-skill-refs.py, tools/skill-refs-baseline.json, and
tools/sync-vendored.py (plus their tests) into
plugins/aws-startup-advisor/tools/. Both gates are owned by, and only ever
scan, this plugin — check-skill-refs.py was already hardcoded to
DEFAULT_PLUGIN=aws-startup-advisor, and sync-vendored.py's vendored/shared
pattern has no other consumer in this repository.
- check-skill-refs.py and its baseline/test now resolve relative to the
script's own plugin-local position (PLUGIN_ROOT) instead of a root-level
tools/ directory; REPO_ROOT is still derived (two levels further up) for
the one genuinely repo-relative fallback candidate and for display paths.
- sync-vendored.py: same single-plugin fix as PR aws#386 (PLUGIN_ROOT instead
of scanning every plugins/ subdirectory; --plugin-root replaces the
removed multi-plugin --plugin/--plugins-root surface).
- Updated mise.toml's lint:skill-refs and lint:vendored-parity tasks and the
four vendored/README.md usage snippets to the new paths.
- Updated the baseline file's _what_this_is pointer to the new script path.
* test(aws-startup-advisor): pin AI migration matrix
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (6 passed)\n- mise run lint:artifacts (passed)\n- mise run lint:vendored-parity (passed)\n- mise run lint:skill-refs (passed)\n- mise run lint:md (passed)\n- mise run validate (passed)\n- mise run build (passed)
* fix(aws-startup-advisor): fail closed integration verdict
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (7 passed)\n- mise run lint:artifacts (70 checked, 0 errors)\n- mise run lint:vendored-parity (44 files passed)\n- mise run lint:skill-refs (1160 refs passed)\n- mise run lint:md (2475 files, 0 issues)\n- mise run validate (passed)\n- mise run build (passed; gitleaks found no leaks)\n- git diff --check (passed)
* refactor(aws-startup-advisor): separate Bedrock pricing cache
Extract provider-neutral Bedrock rates into the shared AI canonical/vendor pattern and repoint GCP, llm-to-bedrock, and agent-advisor consumers without changing pricing behavior.
Validation: parity script PASS (54 model keys, 24 static fallbacks, representative estimates unchanged); llm pricing 21 passed; agent-advisor 323 passed; Azure estimate fixture PASS; mise run build PASS.
* fix(aws-startup-advisor): preserve pricing accuracy metadata
* fix: align integration validation run IDs
Accept uppercase UUID hex in integration validation to match the phase-status contract, cover uppercase/lowercase/malformed IDs through the artifact validator, and document the distinct OpenRouter fixture pair.
Validated: targeted matrix suite (8 passed); uppercase artifact probe; empty-reference rejection; failed infrastructure/AI/IAM children; phase-status schema probe; mise run build; git diff --check; clean merge-tree against upstream/main.
* fix(aws-startup-advisor): correct Bedrock cache ownership
Route Azure Bedrock estimates through the canonical vendored cache, remove duplicate Azure Bedrock rows, and align infrastructure and AI accuracy contracts.
Validated: pricing tests 21 passed; agent-advisor 323 passed; model/vendor tests 19 passed; Azure/GCP fixtures passed; parity checker and four mutation probes passed; artifact, vendored, skill-ref, Markdown, aggregate validation, full build, and gitleaks passed.
* test(aws-startup-advisor): make AI matrix composable
Validated 31 contract tests and all 14 scenario snapshots. Full build, artifact lint, vendored parity, skill refs, markdown lint, and gitleaks pass.
* test(aws-startup-advisor): expand AI migration matrix
* fix: enforce exact gateway source contracts
* fix: validate shared neutral fixtures
* fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly
Root mise.toml now matches main exactly. lint:artifacts and
lint:vendored-parity are defined in a new, fully self-contained
plugins/aws-startup-advisor/mise.toml instead, for local dev convenience.
CI enforcement bypasses mise entirely, same pattern as the existing
model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml
(PR aws#385): two dedicated workflows invoke the plugin's scripts directly
via python3, scoped to pull requests touching plugins/aws-startup-advisor/**.
Net effect: root mise.toml has zero diff against main. The only files
outside the plugin are the two GitHub Actions workflow files themselves,
which GitHub requires under .github/workflows/ — each only invokes the
plugin's own script and carries no shared logic.
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): address PR aws#410 Cursor review findings
Three independently reproduced findings:
1. clarify-ai-only.md's "preferences.json (AI-only)" example contradicted
its own prose: ai_framework.value was a bare string while Q1 is
documented as multi-select with a ["direct"] default, and the example
omitted the mandated metadata.report_caveats[] and never used the
source/question_id provenance alternative the prose referenced. Fixed
the example to match an array value and added report_caveats; clarified
the prose that question_id is this flow's sole provenance field.
2. _assert_scenario_semantics deduplicated gateway upstreams on their raw
(provider, source_service-or-None) key BEFORE resolving a service-less
upstream to its actual source, so an explicit-service upstream and an
overlapping service-less upstream for the same single retained source
had different raw keys and both passed. Now dedupes on the resolved
source's own (provider, source_service) key. Added a parametrized
regression test covering both insertion orderings.
3. strip_jsonc attributed a trailing // comment to the textually-last
"key": match on a line, which broke for an inline array/object value
that closes before the comment ("call_sites": [{ "file": ..., "line":
1 }], // REQUIRED resolved to "line", not "call_sites"), so omitting
the entire required field passed silently. Now tracks bracket depth and
binds the comment to the key whose own value's depth matches (or, for a
value still open at end of line, is one level shallower than) the
depth at the comment. Added unit tests on strip_jsonc directly plus an
end-to-end MISSING_REQUIRED regression against the real contract.
All three reproduced against the live PR head before fixing. 104 tests
pass in the two affected files; full mise run build, lint:artifacts (74
checked, 0 errors), lint:vendored-parity, diff --check, and merge-tree
against upstream/main are all clean.
* fix(aws-startup-advisor): address PR aws#411 Cursor review findings
Four independently reproduced findings:
1. The pre-split Azure cache had its own "### Embeddings — Bedrock" target
table (Titan Text Embeddings v1/v2, Cohere Embed v4/English v3/Multilingual
v3) nested under GCP's "## Bedrock Models (On-Demand)" heading, but it is a
different subsection than the one the parity script's byte-equality check
covers, so the split dropped it entirely. estimate-ai.md's design flow still
references this table and still recommends Titan v2 as the default embedding
target. Moved the table into the canonical cache (scoped out of the main
section's byte-equality check, matching the existing Claude Fable exception
pattern) and the model_rows() quick-reference scanner (its Dimensions column
coincidentally parses as a numeric output price for three rows), synced all
three vendors, and added a dedicated row-level parity check.
2. clarify-ai-only.md's AI-only example contradicted its own prose (bare-string
ai_framework vs. the documented ["direct"] array default, missing
metadata.report_caveats) and route_shapes does not inherit the parent
preferences.json entry's allowed_anywhere -- so (REQUIRED on every
DETECTED-equivalent row per schema-preferences.md's own fundamental rule)
was rejected on this route even though the general contract requires it.
Fixed the example and added allowed_anywhere to the AI-only route_shapes
entry; added source to this route's extracted rows and corrected the prose
accordingly. Added three documented-shape acceptance cases to the existing
producer-route test.
3. artifact-contracts.json's extra_shapes pointed preferences.json's mixed-flow
AI fields at clarify-ai.md's raw per-fragment "Step 3: Rows returned" values
instead of the final assembled {disposition, value, default} envelope
clarify-assemble.md's rules require and every sibling fragment (e.g.
clarify-global.md) already documents in that wrapped form; the example also
omitted the documented Q21 ai_latency row. Rewrote the example to match the
established convention, with source on DETECTED rows per finding 2's rule.
4. strip_jsonc attributed a trailing comment to the textually-last "key":
match on a line, which broke for the committed workloads shape
"call_sites": [{ "file": ..., "line": 1 }], // REQUIRED (resolved to
"line", not "call_sites"), so omitting the entire required field passed
silently in both Azure preferences routes. Now tracks bracket depth and
binds the comment to the key whose own depth matches the end-of-line depth.
Added strip_jsonc unit tests plus end-to-end MISSING_REQUIRED regressions
on the real contract.
All four reproduced against the live PR head before fixing. 73 tests pass in
the plugin's full suite plus 23 in llm-to-bedrock's pricing suite; mise run
build, lint:artifacts (53 checked, 0 errors), lint:vendored-parity (47 files),
diff --check, and merge-tree against upstream/main are all clean.
* fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows
Correction to an earlier fix on this PR. While fixing the identical finding
on PR aws#411, confirmed via schema-preferences.md's own fundamental rule
("the justification key is source, and it is REQUIRED on DETECTED") that
source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's
chosen_by: "extracted" rows -- the AI-only route's naming for the same
DETECTED concept. The earlier fix on this PR incorrectly concluded source
was "not part of this AI-only shape" and only clarified the prose to say
so, rather than fixing the real root cause: route_shapes builds a shape
contract from only the route object, not merged with the parent
preferences.json entry, so it never inherited that entry's
allowed_anywhere vocabulary (source, reason, context, etc.).
Added allowed_anywhere to the AI-only route_shapes manifest entry
(mirroring the parent preferences.json entry's), added source to the
example's chosen_by: "extracted" rows (ai_framework, ai_monthly_spend,
ai_model_baseline, ai_vision), and corrected the prose to say source is
required on those rows. Added three acceptance cases (multi-value
framework, report_caveats present, source present) to the existing
producer-route test, replacing the test case that had asserted source
must still fail.
136 tests pass in the plugin's full suite; mise run build, lint:artifacts
(74 checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
* fix(aws-startup-advisor): address PR aws#410 follow-up Cursor review findings
Three independently reproduced findings:
1. Regression in the strip_jsonc depth-tracking fix: the unconditional
'if ch not in "\"\": key_start = None' reset the pending key on ANY
non-quote character, including whitespace between a closed key string
and its ':'. 'priority' : "balanced", // REQUIRED (space or tab before
the colon) lost the key entirely, so omitting workloads[0].priority
stopped producing MISSING_REQUIRED. Fixed by exempting whitespace from
the reset; only a genuinely different character invalidates the pending
key. Added parametrized unit tests for space/tab/double-space before the
colon plus a control proving a real intervening token still invalidates.
2. artifact-contracts.json's extra_shapes still pointed preferences.json's
mixed-flow AI fields at clarify-ai.md's raw per-fragment 'Step 3: Rows
returned' values instead of the final assembled {disposition, value,
default} envelope clarify-assemble.md's rules require (the same finding
already fixed on aws#411/aws#416, not yet addressed here); the example also
omitted the documented Q21 ai_latency row. Rewrote the example to match
the established sibling-fragment convention, with source on DETECTED
rows per the aws#411 correction.
3. Both clarify-ai.md and clarify-ai-only.md persisted
workloads[].capability: "chat", but schema-discover-ai.md's capability
enum is text_generation/structured_output/image_generation/embedding/…
and design-ai.md has no routing case for "chat". Changed both examples
to text_generation and added a test asserting both producer examples
use a value from the discovery/design vocabulary.
All three reproduced against the live PR head before fixing. 141 tests
pass in the plugin's full suite; mise run build, lint:artifacts (74
checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
* refactor(aws-startup-advisor): move vendored-parity check into plugin dispatcher
---------
Co-authored-by: Logan Kleier <lkleier@amazon.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Gen Li <i.ligen@gmail.com>
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 14, 2026
* feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas
The migration skills pass ~17 JSON artifacts between phases, each with a
contract published as a schema-*.md "shape" block (JSON/JSONC example) or,
for two of them, a real JSON Schema under skills/shared/. Nothing checked
that a producer writes what the next phase reads: a key could be renamed,
emitted without being documented, or an enum widened, and every DSL
_assert still passed because the model that writes the artifact evaluates
the assertion. This is the precondition the backlog names for any engine
consolidation or AI-engine extraction — moves across adapters have no
drift detection without it.
scripts/validate-artifacts.py (stdlib, Python 3.9+)
- Parses each contract document's example blocks (JSONC comments, trailing
commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments,
`// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open
object, `"<key>": {...}` = template for every key) into a key-path
contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`)
refine paths; manifest can pin a block by `@L<line>` anchor, merge other
documents' additions (graviton, workshop), or declare a row-annotation
vocabulary (azure preferences' source/reason/context/...).
- Validates real JSON Schemas with a draft-07 subset (type, required,
properties, additionalProperties, enum, const, items, pattern, min/max,
minItems, allOf/anyOf/oneOf, $ref).
- Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION,
SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken),
NO_CONTRACT (info). `_`-prefixed keys are annotations.
- Modes: --self-check (contracts parse), --fixtures (CI; goldens against
the baseline), --run-dir <.migration/run> --skill <name> (users).
- scripts/artifact-contracts.json maps artifact → contract per skill.
- scripts/artifact-contracts-baseline.json: 21 explained entries covering
44 known gaps between what phases emit and what their contract documents
say, each naming the document that must change. Stale entries are
reported so the list only shrinks.
Contract documents fixed where the document contradicted itself (its own
prose named a key its example block omitted):
- gcp schema-preferences.md: the shape block was not JSON (`{ "value",
"chosen_by" }` set notation); rewritten as a JSONC row template.
- gcp schema-estimate-infra.md: root block lacked complexity_tier, which
estimation-infra.schema.json marks required.
- gcp schema-discover-iac.md § Live discovery extensions: block now shows
the metadata/resources[] additions the prose above it documents.
- heroku schema-discover-heroku.md: resources[] example carries
source/unmanaged_by_terraform/not_found_live (table at lines 93-95);
addon config example carries the per-add-on and live-enrichment fields
the bullets below it list.
- azure schema-design-aws.md: services[] block carries
routing_provenance/sizing_provenance/sizing_source/is_compute_unit/
hosted_app_azure_ids (the same document marks them REQUIRED); deferred[]
carries routing_provenance.
- azure schema-discover-azure.md: clusters[] carries pattern_status
(discover-assemble.md § Status).
- azure schema-preferences.md: top-level clarify_status (REQUIRED per
clarify-assemble.md § clarify_status).
Fixture: gcp-decision-gate's estimation-infra.json stub gains the four
keys estimation-infra.schema.json requires; check_expected_decide.py
still passes.
Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it).
README scripts/ entry updated.
Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping,
template inference (enum/required/placeholder/any-key), each finding
code, JSON Schema subset, baseline matching + stale detection, CLI
self-check and fixtures pass on the real repo, and two negatives: an
injected off-contract key and a schema enum violation both exit 1.
* fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails
Review on aws#385, both should-fix findings valid:
1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`;
the draft-07 subset did not implement it, so `phases: {}` passed. Fixed
the instance AND the class: the subset now enforces minProperties,
maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems,
minLength/maxLength and `not`; and `unsupported_keywords()` walks every
loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any
validation keyword that is neither enforced nor a known annotation. A
constraint can no longer be skipped silently — adding a keyword to a
schema without teaching the validator fails the gate.
2. [Completeness] a baseline entry that matched nothing was printed as a
warning and exited 0, so the burn-down list only shrank if someone read
the log. STALE_BASELINE is now an error code: `--fixtures` (what
`mise run lint:artifacts` runs) exits 1 on any stale entry.
Tests (19): empty `phases` fails with minProperties named; the two real
schemas use only enforced/annotation keywords and an invented
`contentEncoding` / `dependencies` schema is rejected; a never-matching
baseline entry exits 1 with STALE_BASELINE.
* ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/
Each migration skill is installable on its own (`npx skills add … --skill
gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate
schemas, …) is vendored into skills/<skill>/references/vendored/ from the
canonical skills/shared/. The vendored READMEs said: "This repository has
no automated sync task for these copies — keeping them byte-identical is
part of the change that touches the canonical file." All 44 copies are
identical today; nothing stopped the next edit to skills/shared/ from
leaving a skill running stale logic.
- tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py.
Default mode copies skills/shared/<path> over every vendored copy that
differs; --check exits 1 listing any copy that differs or has no
canonical source (a vendored file nobody owns). README.md in a vendored
dir is explanatory and skipped. The vendored tree is a subset of shared/,
so files in shared/ that no skill vendors are fine.
- mise `lint:vendored-parity`, added to `validate` (the CI build runs it).
- The four vendored READMEs replace the manual cp + md5sum instructions
with the tool and the CI rule.
- tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and
orphan detection (README excluded), sync repairs drift and leaves orphans
as errors, CLI exit codes — plus the real repository is in sync.
No skill content changes. Stacked on aws#385 only because both add a task to
the same `validate` depends line.
* fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees
Review on aws#386 (Contradictions / Completeness, should-fix): the three
migration READMEs claimed "a change to skills/shared/ that forgets the
copies cannot merge", but the check only walked files that still existed —
a deleted vendored copy, or a new shared file never vendored, passed.
Rather than only weakening the sentence, the gate now uses the record of
intent each vendored README already carries: its
`| Vendored path | Canonical source |` table.
- check: every row must exist on disk (deleted copy → FAIL), every file on
disk must have a row (README stays honest), and each row's canonical
path must be skills/shared/<same path>. A README with no table is an
error.
- sync: restores a listed-but-missing copy from canonical before comparing.
- What the gate still cannot see — a NEW skills/shared/ file no skill
vendors — is now stated in the tool docstring and in all four READMEs,
with the instruction "add the copy AND a table row".
The three READMEs were already complete tables (gcp 14/14, heroku 8/8,
azure 21/21, agent-advisor 1/1); the real repo passes unchanged.
Tests (7): deleted copy caught via the table and restored by sync;
unlisted file and wrong canonical path caught; a new shared file nobody
vendors is NOT an error (pins the documented limit); CLI exit codes; real
repo in sync. Reviewer probes re-run on a throwaway copy of the plugin:
deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode
restores it; adding skills/shared/brand-new.md → no error, as documented.
* ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist
The skills are prose state machines: a phase file says "Load
`references/phases/design/design-ai.md`" or "run `python3
\"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing
checked the target exists. A renamed or never-written file is a dead end
the user hits mid-migration, and any file move during consolidation would
break references silently. This is the third gate in the family
(validate-artifacts.py: artifact contracts; sync-vendored.py: vendored
parity) and the precondition for moving phase files between skills.
tools/check-skill-refs.py (stdlib)
- Scans skills/**/*.md and agents/*.md for path tokens into the skill
tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the
agent-advisor short forms phases/, shared/, design-refs/) with optional
$PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> /
$GCP_BASE / ./ / ../ prefixes.
- Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form> → plugin/repo-relative. Library-origin
files (skills/shared/*, */references/vendored/*, */references/shared/*)
speak from the consuming skill's point of view and resolve against
skills/shared and every skill; agent-advisor and llm-to-bedrock may name
gcp-to-aws's tree (documented inline execution); agents/ resolve against
llm-to-bedrock.
- Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…),
templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no
skill-tree root in them (generated outputs like ./deploy.sh).
- tools/skill-refs-baseline.json: `ignore` rules (permanent policy —
scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in
the user's app) and `entries` (known dead references with the fix;
STALE when they stop matching, failing the gate so the list only shrinks).
1118 references checked on the real repo; 7 dead ones baselined, each a
real user-facing defect: azure discover-iac.md points Bicep/ARM input at
extract-bicep.md / extract-arm.md that do not exist (so Bicep halts);
design-refs/patterns.md referenced from two azure files before it exists;
contextual-offers SKILL.md names references/offers.md unqualified when the
file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at
shared/ai-model-lifecycle.md, which in gcp is vendored at
references/vendored/ai/.
mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution
rule, each skip rule, library-origin resolution, gcp-dependent and agents
fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS.
* fix(aws-startup-advisor): check-skill-refs — agents/ resolve only against llm-to-bedrock
Review on aws#387 (Completeness / Contradictions, should-fix): the agents/
branch of the resolver searched every skill and kept the first hit, so
agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml`
resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a
tree with the file only under agent-advisor still passed — deleting the
llm-to-bedrock copy would not have failed the gate. The comment above the
loop said llm-to-bedrock; the loop did not.
- Agent references now resolve ONLY against skills/llm-to-bedrock
(_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to
that skill.
- Tests: scripts/pyproject.toml present only under agent-advisor is
MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy
even when agent-advisor also has it; the fixture tree's agents/worker.md
reference to another skill's file is now correctly MISSING.
- Real repo unchanged: 1118 references, 0 missing, 7 baselined.
* fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths
Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on
aws#386/aws#387 against the same code):
- The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it
now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no
baseline unless --baseline names one, and then every entry must still match (a valid
real run previously failed with 21 STALE_BASELINE errors).
- Node.merge keeps a declared enum when another example shows a concrete member
(`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document
shows joins the set, only a free-text example clears it. Adopted children are copied,
so refining `design_constraints.cpu_architecture` no longer widens the generic
`<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`.
- `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is
valid, `terraform+bogus` is not, closed enums elsewhere stay closed.
- Heading match is exact or word-boundary prefix: `graviton` no longer selects
`graviton_profile`; the Azure design manifest names `graviton block` explicitly.
- Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the
array level, so workload key/type drift is detected; bare item examples still merge
into the item.
- `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku
`metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged.
Two baseline entries record the hand-authored Heroku seed inventories that lack them.
- GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline
and breakdown.security_baseline_compliance bind to their object keys (line anchors
separate the two Security Baseline blocks); nested drift inside them is still caught.
- Heroku workshop `preferences_subset` is the documented open knob map (open_paths).
15 regression tests added (unit + CLI against the real contracts); each fails on the
previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale.
* fix(aws-startup-advisor): check-skill-refs — run-artifact skip is per token; explicit bases are authoritative
Review round 2 on aws#387 (the two findings unique to this tool):
- The run-artifact skip looked 40 characters back, so an earlier, separate token
(`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line
(azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the
token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…`
forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve).
- An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now
authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a
same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look
alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md`
traversal still resolves.
Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing,
7 baselined, 0 stale.
* fix(aws-startup-advisor): admit the unfired licensing stub in the preferences example
The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): close three artifact and vendored-parity gaps
An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): reject refined empty-array items and honor relative prefixes
An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): require the Azure discovery verdict and check estimate snapshots
A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): accept each Azure preference route and typed drift values
An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate.
* fix: ignore Bedrock run state references
* fix(aws-startup-advisor): keep vendored-parity tooling plugin-local
Move tools/sync-vendored.py and its test into
plugins/aws-startup-advisor/tools/, since the vendored/shared pattern it
gates is unique to this plugin (no other plugin uses references/vendored/).
- Script now resolves PLUGIN_ROOT from its own location instead of scanning
every plugins/ subdirectory; dropped the now-unneeded --plugin/--plugins-root
multi-plugin CLI surface in favor of a single --plugin-root override for
tests.
- Updated the test suite to match (plugin-root fixtures instead of a
multi-plugin tree; the MCP-only-plugin skip test now asserts a clean exit
for a plugin root with no skills/ at all).
- Updated mise.toml's lint:vendored-parity task and the four vendored/
README.md usage snippets (agent-advisor, heroku-to-aws, gcp-to-aws,
azure-to-aws) to the new path.
* fix(aws-startup-advisor): keep skill-refs and vendored-parity tooling plugin-local
Move tools/check-skill-refs.py, tools/skill-refs-baseline.json, and
tools/sync-vendored.py (plus their tests) into
plugins/aws-startup-advisor/tools/. Both gates are owned by, and only ever
scan, this plugin — check-skill-refs.py was already hardcoded to
DEFAULT_PLUGIN=aws-startup-advisor, and sync-vendored.py's vendored/shared
pattern has no other consumer in this repository.
- check-skill-refs.py and its baseline/test now resolve relative to the
script's own plugin-local position (PLUGIN_ROOT) instead of a root-level
tools/ directory; REPO_ROOT is still derived (two levels further up) for
the one genuinely repo-relative fallback candidate and for display paths.
- sync-vendored.py: same single-plugin fix as PR aws#386 (PLUGIN_ROOT instead
of scanning every plugins/ subdirectory; --plugin-root replaces the
removed multi-plugin --plugin/--plugins-root surface).
- Updated mise.toml's lint:skill-refs and lint:vendored-parity tasks and the
four vendored/README.md usage snippets to the new paths.
- Updated the baseline file's _what_this_is pointer to the new script path.
* test(aws-startup-advisor): pin AI migration matrix
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (6 passed)\n- mise run lint:artifacts (passed)\n- mise run lint:vendored-parity (passed)\n- mise run lint:skill-refs (passed)\n- mise run lint:md (passed)\n- mise run validate (passed)\n- mise run build (passed)
* fix(aws-startup-advisor): fail closed integration verdict
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (7 passed)\n- mise run lint:artifacts (70 checked, 0 errors)\n- mise run lint:vendored-parity (44 files passed)\n- mise run lint:skill-refs (1160 refs passed)\n- mise run lint:md (2475 files, 0 issues)\n- mise run validate (passed)\n- mise run build (passed; gitleaks found no leaks)\n- git diff --check (passed)
* refactor(aws-startup-advisor): separate Bedrock pricing cache
Extract provider-neutral Bedrock rates into the shared AI canonical/vendor pattern and repoint GCP, llm-to-bedrock, and agent-advisor consumers without changing pricing behavior.
Validation: parity script PASS (54 model keys, 24 static fallbacks, representative estimates unchanged); llm pricing 21 passed; agent-advisor 323 passed; Azure estimate fixture PASS; mise run build PASS.
* fix(aws-startup-advisor): preserve pricing accuracy metadata
* fix: align integration validation run IDs
Accept uppercase UUID hex in integration validation to match the phase-status contract, cover uppercase/lowercase/malformed IDs through the artifact validator, and document the distinct OpenRouter fixture pair.
Validated: targeted matrix suite (8 passed); uppercase artifact probe; empty-reference rejection; failed infrastructure/AI/IAM children; phase-status schema probe; mise run build; git diff --check; clean merge-tree against upstream/main.
* fix(aws-startup-advisor): correct Bedrock cache ownership
Route Azure Bedrock estimates through the canonical vendored cache, remove duplicate Azure Bedrock rows, and align infrastructure and AI accuracy contracts.
Validated: pricing tests 21 passed; agent-advisor 323 passed; model/vendor tests 19 passed; Azure/GCP fixtures passed; parity checker and four mutation probes passed; artifact, vendored, skill-ref, Markdown, aggregate validation, full build, and gitleaks passed.
* test(aws-startup-advisor): make AI matrix composable
Validated 31 contract tests and all 14 scenario snapshots. Full build, artifact lint, vendored parity, skill refs, markdown lint, and gitleaks pass.
* test(aws-startup-advisor): expand AI migration matrix
* fix: enforce exact gateway source contracts
* fix: validate shared neutral fixtures
* fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly
Root mise.toml now matches main exactly. lint:artifacts and
lint:vendored-parity are defined in a new, fully self-contained
plugins/aws-startup-advisor/mise.toml instead, for local dev convenience.
CI enforcement bypasses mise entirely, same pattern as the existing
model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml
(PR aws#385): two dedicated workflows invoke the plugin's scripts directly
via python3, scoped to pull requests touching plugins/aws-startup-advisor/**.
Net effect: root mise.toml has zero diff against main. The only files
outside the plugin are the two GitHub Actions workflow files themselves,
which GitHub requires under .github/workflows/ — each only invokes the
plugin's own script and carries no shared logic.
* fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly
Root mise.toml now matches main exactly. lint:artifacts, lint:vendored-parity,
and lint:skill-refs are defined in a new, fully self-contained
plugins/aws-startup-advisor/mise.toml instead, for local dev convenience.
CI enforcement bypasses mise entirely, same pattern as the existing
model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml /
vendored-parity.yml (PRs aws#385/aws#386): three dedicated workflows invoke the
plugin's scripts directly via python3, scoped to pull requests touching
plugins/aws-startup-advisor/**.
Net effect: root mise.toml has zero diff against main. The only files
outside the plugin are the three GitHub Actions workflow files themselves,
which GitHub requires under .github/workflows/ — each only invokes the
plugin's own script and carries no shared logic.
* test(aws-startup-advisor): define neutral AI workload profile
Add a target-only, provider-neutral ai-workload-profile.json contract
plus representative fixtures and contract tests, so a future
normalization output can preserve multiple AI providers/gateways
instead of collapsing them into a lossy both/single-gateway-type
shape. No producer or runtime behavior change; schema validity does
not imply an implemented adapter or route.
* test(aws-startup-advisor): lock observed-source evidence trail and cover ai-workload fixtures in lint:artifacts
- Add regression test asserting _assert_profile_semantics rejects an observed
ai_source whose model_observations has been emptied while models[] and
model_evidence stay intact (the schema alone cannot express that correspondence).
- Move the four ai-workload-profiles fixtures into <case>/ai-workload-profile.json
subdirectories and register ai-workload-profile.json in artifact-contracts.json's
shared section so lint:artifacts (and mise run build) actually open these files.
* fix(aws-startup-advisor): require model_observations on observed ai-workload-profile sources
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): address PR aws#410 Cursor review findings
Three independently reproduced findings:
1. clarify-ai-only.md's "preferences.json (AI-only)" example contradicted
its own prose: ai_framework.value was a bare string while Q1 is
documented as multi-select with a ["direct"] default, and the example
omitted the mandated metadata.report_caveats[] and never used the
source/question_id provenance alternative the prose referenced. Fixed
the example to match an array value and added report_caveats; clarified
the prose that question_id is this flow's sole provenance field.
2. _assert_scenario_semantics deduplicated gateway upstreams on their raw
(provider, source_service-or-None) key BEFORE resolving a service-less
upstream to its actual source, so an explicit-service upstream and an
overlapping service-less upstream for the same single retained source
had different raw keys and both passed. Now dedupes on the resolved
source's own (provider, source_service) key. Added a parametrized
regression test covering both insertion orderings.
3. strip_jsonc attributed a trailing // comment to the textually-last
"key": match on a line, which broke for an inline array/object value
that closes before the comment ("call_sites": [{ "file": ..., "line":
1 }], // REQUIRED resolved to "line", not "call_sites"), so omitting
the entire required field passed silently. Now tracks bracket depth and
binds the comment to the key whose own value's depth matches (or, for a
value still open at end of line, is one level shallower than) the
depth at the comment. Added unit tests on strip_jsonc directly plus an
end-to-end MISSING_REQUIRED regression against the real contract.
All three reproduced against the live PR head before fixing. 104 tests
pass in the two affected files; full mise run build, lint:artifacts (74
checked, 0 errors), lint:vendored-parity, diff --check, and merge-tree
against upstream/main are all clean.
* fix(aws-startup-advisor): address PR aws#411 Cursor review findings
Four independently reproduced findings:
1. The pre-split Azure cache had its own "### Embeddings — Bedrock" target
table (Titan Text Embeddings v1/v2, Cohere Embed v4/English v3/Multilingual
v3) nested under GCP's "## Bedrock Models (On-Demand)" heading, but it is a
different subsection than the one the parity script's byte-equality check
covers, so the split dropped it entirely. estimate-ai.md's design flow still
references this table and still recommends Titan v2 as the default embedding
target. Moved the table into the canonical cache (scoped out of the main
section's byte-equality check, matching the existing Claude Fable exception
pattern) and the model_rows() quick-reference scanner (its Dimensions column
coincidentally parses as a numeric output price for three rows), synced all
three vendors, and added a dedicated row-level parity check.
2. clarify-ai-only.md's AI-only example contradicted its own prose (bare-string
ai_framework vs. the documented ["direct"] array default, missing
metadata.report_caveats) and route_shapes does not inherit the parent
preferences.json entry's allowed_anywhere -- so (REQUIRED on every
DETECTED-equivalent row per schema-preferences.md's own fundamental rule)
was rejected on this route even though the general contract requires it.
Fixed the example and added allowed_anywhere to the AI-only route_shapes
entry; added source to this route's extracted rows and corrected the prose
accordingly. Added three documented-shape acceptance cases to the existing
producer-route test.
3. artifact-contracts.json's extra_shapes pointed preferences.json's mixed-flow
AI fields at clarify-ai.md's raw per-fragment "Step 3: Rows returned" values
instead of the final assembled {disposition, value, default} envelope
clarify-assemble.md's rules require and every sibling fragment (e.g.
clarify-global.md) already documents in that wrapped form; the example also
omitted the documented Q21 ai_latency row. Rewrote the example to match the
established convention, with source on DETECTED rows per finding 2's rule.
4. strip_jsonc attributed a trailing comment to the textually-last "key":
match on a line, which broke for the committed workloads shape
"call_sites": [{ "file": ..., "line": 1 }], // REQUIRED (resolved to
"line", not "call_sites"), so omitting the entire required field passed
silently in both Azure preferences routes. Now tracks bracket depth and
binds the comment to the key whose own depth matches the end-of-line depth.
Added strip_jsonc unit tests plus end-to-end MISSING_REQUIRED regressions
on the real contract.
All four reproduced against the live PR head before fixing. 73 tests pass in
the plugin's full suite plus 23 in llm-to-bedrock's pricing suite; mise run
build, lint:artifacts (53 checked, 0 errors), lint:vendored-parity (47 files),
diff --check, and merge-tree against upstream/main are all clean.
* fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows
Correction to an earlier fix on this PR. While fixing the identical finding
on PR aws#411, confirmed via schema-preferences.md's own fundamental rule
("the justification key is source, and it is REQUIRED on DETECTED") that
source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's
chosen_by: "extracted" rows -- the AI-only route's naming for the same
DETECTED concept. The earlier fix on this PR incorrectly concluded source
was "not part of this AI-only shape" and only clarified the prose to say
so, rather than fixing the real root cause: route_shapes builds a shape
contract from only the route object, not merged with the parent
preferences.json entry, so it never inherited that entry's
allowed_anywhere vocabulary (source, reason, context, etc.).
Added allowed_anywhere to the AI-only route_shapes manifest entry
(mirroring the parent preferences.json entry's), added source to the
example's chosen_by: "extracted" rows (ai_framework, ai_monthly_spend,
ai_model_baseline, ai_vision), and corrected the prose to say source is
required on those rows. Added three acceptance cases (multi-value
framework, report_caveats present, source present) to the existing
producer-route test, replacing the test case that had asserted source
must still fail.
136 tests pass in the plugin's full suite; mise run build, lint:artifacts
(74 checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
* fix(aws-startup-advisor): address PR aws#410 follow-up Cursor review findings
Three independently reproduced findings:
1. Regression in the strip_jsonc depth-tracking fix: the unconditional
'if ch not in "\"\": key_start = None' reset the pending key on ANY
non-quote character, including whitespace between a closed key string
and its ':'. 'priority' : "balanced", // REQUIRED (space or tab before
the colon) lost the key entirely, so omitting workloads[0].priority
stopped producing MISSING_REQUIRED. Fixed by exempting whitespace from
the reset; only a genuinely different character invalidates the pending
key. Added parametrized unit tests for space/tab/double-space before the
colon plus a control proving a real intervening token still invalidates.
2. artifact-contracts.json's extra_shapes still pointed preferences.json's
mixed-flow AI fields at clarify-ai.md's raw per-fragment 'Step 3: Rows
returned' values instead of the final assembled {disposition, value,
default} envelope clarify-assemble.md's rules require (the same finding
already fixed on aws#411/aws#416, not yet addressed here); the example also
omitted the documented Q21 ai_latency row. Rewrote the example to match
the established sibling-fragment convention, with source on DETECTED
rows per the aws#411 correction.
3. Both clarify-ai.md and clarify-ai-only.md persisted
workloads[].capability: "chat", but schema-discover-ai.md's capability
enum is text_generation/structured_output/image_generation/embedding/…
and design-ai.md has no routing case for "chat". Changed both examples
to text_generation and added a test asserting both producer examples
use a value from the discovery/design vocabulary.
All three reproduced against the live PR head before fixing. 141 tests
pass in the plugin's full suite; mise run build, lint:artifacts (74
checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
* refactor(aws-startup-advisor): move vendored-parity check into plugin dispatcher
* fix(aws-startup-advisor): account for every neutral fixture directory, not just ai-migration-matrix
test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count
hardcoded the assumption that ai-migration-matrix was the only fixture
directory with no skill prefix (gcp-/heroku-/azure-), so --fixtures'
checked_artifacts total only needed one 'shared artifacts' bucket
subtracted out. This PR's own fixtures/ai-workload-profiles/ directory
is a second such neutral directory (its 4 ai-workload-profile.json
files are themselves registered as a 'shared' contract in
artifact-contracts.json), so --fixtures counts them in
checked_artifacts, but the test's formula never added them to the
subtracted total -- failing with 'assert (78 - 53) == 21' instead of
the correct 25.
Generalized the test to sum shared-artifact counts and validate_run_dir
checks across every fixture directory that skill_for_fixture maps to
None, not just the matrix by name. This also guards against the same
accounting gap recurring if a third neutral fixture directory is added
later.
Verification: the fixed test passes; mise -C plugins/aws-startup-advisor
run ci passes in full (vendored-parity 44 files, artifacts 78 checked/0
errors, model-id, skill-refs, 246 tests); mise run build from repo root
clean; git diff --check clean.
---------
Co-authored-by: Logan Kleier <lkleier@amazon.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Gen Li <i.ligen@gmail.com>
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 14, 2026
…ct (aws#416) * feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas The migration skills pass ~17 JSON artifacts between phases, each with a contract published as a schema-*.md "shape" block (JSON/JSONC example) or, for two of them, a real JSON Schema under skills/shared/. Nothing checked that a producer writes what the next phase reads: a key could be renamed, emitted without being documented, or an enum widened, and every DSL _assert still passed because the model that writes the artifact evaluates the assertion. This is the precondition the backlog names for any engine consolidation or AI-engine extraction — moves across adapters have no drift detection without it. scripts/validate-artifacts.py (stdlib, Python 3.9+) - Parses each contract document's example blocks (JSONC comments, trailing commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments, `// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open object, `"<key>": {...}` = template for every key) into a key-path contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`) refine paths; manifest can pin a block by `@L<line>` anchor, merge other documents' additions (graviton, workshop), or declare a row-annotation vocabulary (azure preferences' source/reason/context/...). - Validates real JSON Schemas with a draft-07 subset (type, required, properties, additionalProperties, enum, const, items, pattern, min/max, minItems, allOf/anyOf/oneOf, $ref). - Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION, SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken), NO_CONTRACT (info). `_`-prefixed keys are annotations. - Modes: --self-check (contracts parse), --fixtures (CI; goldens against the baseline), --run-dir <.migration/run> --skill <name> (users). - scripts/artifact-contracts.json maps artifact → contract per skill. - scripts/artifact-contracts-baseline.json: 21 explained entries covering 44 known gaps between what phases emit and what their contract documents say, each naming the document that must change. Stale entries are reported so the list only shrinks. Contract documents fixed where the document contradicted itself (its own prose named a key its example block omitted): - gcp schema-preferences.md: the shape block was not JSON (`{ "value", "chosen_by" }` set notation); rewritten as a JSONC row template. - gcp schema-estimate-infra.md: root block lacked complexity_tier, which estimation-infra.schema.json marks required. - gcp schema-discover-iac.md § Live discovery extensions: block now shows the metadata/resources[] additions the prose above it documents. - heroku schema-discover-heroku.md: resources[] example carries source/unmanaged_by_terraform/not_found_live (table at lines 93-95); addon config example carries the per-add-on and live-enrichment fields the bullets below it list. - azure schema-design-aws.md: services[] block carries routing_provenance/sizing_provenance/sizing_source/is_compute_unit/ hosted_app_azure_ids (the same document marks them REQUIRED); deferred[] carries routing_provenance. - azure schema-discover-azure.md: clusters[] carries pattern_status (discover-assemble.md § Status). - azure schema-preferences.md: top-level clarify_status (REQUIRED per clarify-assemble.md § clarify_status). Fixture: gcp-decision-gate's estimation-infra.json stub gains the four keys estimation-infra.schema.json requires; check_expected_decide.py still passes. Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it). README scripts/ entry updated. Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping, template inference (enum/required/placeholder/any-key), each finding code, JSON Schema subset, baseline matching + stale detection, CLI self-check and fixtures pass on the real repo, and two negatives: an injected off-contract key and a schema enum violation both exit 1. * fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails Review on aws#385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE. * ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/ Each migration skill is installable on its own (`npx skills add … --skill gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate schemas, …) is vendored into skills/<skill>/references/vendored/ from the canonical skills/shared/. The vendored READMEs said: "This repository has no automated sync task for these copies — keeping them byte-identical is part of the change that touches the canonical file." All 44 copies are identical today; nothing stopped the next edit to skills/shared/ from leaving a skill running stale logic. - tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py. Default mode copies skills/shared/<path> over every vendored copy that differs; --check exits 1 listing any copy that differs or has no canonical source (a vendored file nobody owns). README.md in a vendored dir is explanatory and skipped. The vendored tree is a subset of shared/, so files in shared/ that no skill vendors are fine. - mise `lint:vendored-parity`, added to `validate` (the CI build runs it). - The four vendored READMEs replace the manual cp + md5sum instructions with the tool and the CI rule. - tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and orphan detection (README excluded), sync repairs drift and leaves orphans as errors, CLI exit codes — plus the real repository is in sync. No skill content changes. Stacked on aws#385 only because both add a task to the same `validate` depends line. * fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees Review on aws#386 (Contradictions / Completeness, should-fix): the three migration READMEs claimed "a change to skills/shared/ that forgets the copies cannot merge", but the check only walked files that still existed — a deleted vendored copy, or a new shared file never vendored, passed. Rather than only weakening the sentence, the gate now uses the record of intent each vendored README already carries: its `| Vendored path | Canonical source |` table. - check: every row must exist on disk (deleted copy → FAIL), every file on disk must have a row (README stays honest), and each row's canonical path must be skills/shared/<same path>. A README with no table is an error. - sync: restores a listed-but-missing copy from canonical before comparing. - What the gate still cannot see — a NEW skills/shared/ file no skill vendors — is now stated in the tool docstring and in all four READMEs, with the instruction "add the copy AND a table row". The three READMEs were already complete tables (gcp 14/14, heroku 8/8, azure 21/21, agent-advisor 1/1); the real repo passes unchanged. Tests (7): deleted copy caught via the table and restored by sync; unlisted file and wrong canonical path caught; a new shared file nobody vendors is NOT an error (pins the documented limit); CLI exit codes; real repo in sync. Reviewer probes re-run on a throwaway copy of the plugin: deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode restores it; adding skills/shared/brand-new.md → no error, as documented. * ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist The skills are prose state machines: a phase file says "Load `references/phases/design/design-ai.md`" or "run `python3 \"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing checked the target exists. A renamed or never-written file is a dead end the user hits mid-migration, and any file move during consolidation would break references silently. This is the third gate in the family (validate-artifacts.py: artifact contracts; sync-vendored.py: vendored parity) and the precondition for moving phase files between skills. tools/check-skill-refs.py (stdlib) - Scans skills/**/*.md and agents/*.md for path tokens into the skill tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the agent-advisor short forms phases/, shared/, design-refs/) with optional $PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> / $GCP_BASE / ./ / ../ prefixes. - Resolution: explicit prefix → skill-relative → file-relative → skill/references/<short form> → plugin/repo-relative. Library-origin files (skills/shared/*, */references/vendored/*, */references/shared/*) speak from the consuming skill's point of view and resolve against skills/shared and every skill; agent-advisor and llm-to-bedrock may name gcp-to-aws's tree (documented inline execution); agents/ resolve against llm-to-bedrock. - Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…), templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no skill-tree root in them (generated outputs like ./deploy.sh). - tools/skill-refs-baseline.json: `ignore` rules (permanent policy — scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in the user's app) and `entries` (known dead references with the fix; STALE when they stop matching, failing the gate so the list only shrinks). 1118 references checked on the real repo; 7 dead ones baselined, each a real user-facing defect: azure discover-iac.md points Bicep/ARM input at extract-bicep.md / extract-arm.md that do not exist (so Bicep halts); design-refs/patterns.md referenced from two azure files before it exists; contextual-offers SKILL.md names references/offers.md unqualified when the file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at shared/ai-model-lifecycle.md, which in gcp is vendored at references/vendored/ai/. mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution rule, each skip rule, library-origin resolution, gcp-dependent and agents fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS. * fix(aws-startup-advisor): check-skill-refs — agents/ resolve only against llm-to-bedrock Review on aws#387 (Completeness / Contradictions, should-fix): the agents/ branch of the resolver searched every skill and kept the first hit, so agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml` resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a tree with the file only under agent-advisor still passed — deleting the llm-to-bedrock copy would not have failed the gate. The comment above the loop said llm-to-bedrock; the loop did not. - Agent references now resolve ONLY against skills/llm-to-bedrock (_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to that skill. - Tests: scripts/pyproject.toml present only under agent-advisor is MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy even when agent-advisor also has it; the fixture tree's agents/worker.md reference to another skill's file is now correctly MISSING. - Real repo unchanged: 1118 references, 0 missing, 7 baselined. * fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on aws#386/aws#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale. * fix(aws-startup-advisor): check-skill-refs — run-artifact skip is per token; explicit bases are authoritative Review round 2 on aws#387 (the two findings unique to this tool): - The run-artifact skip looked 40 characters back, so an earlier, separate token (`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line (azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…` forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve). - An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md` traversal still resolves. Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing, 7 baselined, 0 stale. * fix(aws-startup-advisor): admit the unfired licensing stub in the preferences example The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): close three artifact and vendored-parity gaps An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): reject refined empty-array items and honor relative prefixes An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): require the Azure discovery verdict and check estimate snapshots A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): accept each Azure preference route and typed drift values An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate. * fix: ignore Bedrock run state references * fix(aws-startup-advisor): keep vendored-parity tooling plugin-local Move tools/sync-vendored.py and its test into plugins/aws-startup-advisor/tools/, since the vendored/shared pattern it gates is unique to this plugin (no other plugin uses references/vendored/). - Script now resolves PLUGIN_ROOT from its own location instead of scanning every plugins/ subdirectory; dropped the now-unneeded --plugin/--plugins-root multi-plugin CLI surface in favor of a single --plugin-root override for tests. - Updated the test suite to match (plugin-root fixtures instead of a multi-plugin tree; the MCP-only-plugin skip test now asserts a clean exit for a plugin root with no skills/ at all). - Updated mise.toml's lint:vendored-parity task and the four vendored/ README.md usage snippets (agent-advisor, heroku-to-aws, gcp-to-aws, azure-to-aws) to the new path. * fix(aws-startup-advisor): keep skill-refs and vendored-parity tooling plugin-local Move tools/check-skill-refs.py, tools/skill-refs-baseline.json, and tools/sync-vendored.py (plus their tests) into plugins/aws-startup-advisor/tools/. Both gates are owned by, and only ever scan, this plugin — check-skill-refs.py was already hardcoded to DEFAULT_PLUGIN=aws-startup-advisor, and sync-vendored.py's vendored/shared pattern has no other consumer in this repository. - check-skill-refs.py and its baseline/test now resolve relative to the script's own plugin-local position (PLUGIN_ROOT) instead of a root-level tools/ directory; REPO_ROOT is still derived (two levels further up) for the one genuinely repo-relative fallback candidate and for display paths. - sync-vendored.py: same single-plugin fix as PR aws#386 (PLUGIN_ROOT instead of scanning every plugins/ subdirectory; --plugin-root replaces the removed multi-plugin --plugin/--plugins-root surface). - Updated mise.toml's lint:skill-refs and lint:vendored-parity tasks and the four vendored/README.md usage snippets to the new paths. - Updated the baseline file's _what_this_is pointer to the new script path. * test(aws-startup-advisor): pin AI migration matrix Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (6 passed)\n- mise run lint:artifacts (passed)\n- mise run lint:vendored-parity (passed)\n- mise run lint:skill-refs (passed)\n- mise run lint:md (passed)\n- mise run validate (passed)\n- mise run build (passed) * fix(aws-startup-advisor): fail closed integration verdict Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (7 passed)\n- mise run lint:artifacts (70 checked, 0 errors)\n- mise run lint:vendored-parity (44 files passed)\n- mise run lint:skill-refs (1160 refs passed)\n- mise run lint:md (2475 files, 0 issues)\n- mise run validate (passed)\n- mise run build (passed; gitleaks found no leaks)\n- git diff --check (passed) * refactor(aws-startup-advisor): separate Bedrock pricing cache Extract provider-neutral Bedrock rates into the shared AI canonical/vendor pattern and repoint GCP, llm-to-bedrock, and agent-advisor consumers without changing pricing behavior. Validation: parity script PASS (54 model keys, 24 static fallbacks, representative estimates unchanged); llm pricing 21 passed; agent-advisor 323 passed; Azure estimate fixture PASS; mise run build PASS. * fix(aws-startup-advisor): preserve pricing accuracy metadata * fix: align integration validation run IDs Accept uppercase UUID hex in integration validation to match the phase-status contract, cover uppercase/lowercase/malformed IDs through the artifact validator, and document the distinct OpenRouter fixture pair. Validated: targeted matrix suite (8 passed); uppercase artifact probe; empty-reference rejection; failed infrastructure/AI/IAM children; phase-status schema probe; mise run build; git diff --check; clean merge-tree against upstream/main. * fix(aws-startup-advisor): correct Bedrock cache ownership Route Azure Bedrock estimates through the canonical vendored cache, remove duplicate Azure Bedrock rows, and align infrastructure and AI accuracy contracts. Validated: pricing tests 21 passed; agent-advisor 323 passed; model/vendor tests 19 passed; Azure/GCP fixtures passed; parity checker and four mutation probes passed; artifact, vendored, skill-ref, Markdown, aggregate validation, full build, and gitleaks passed. * test(aws-startup-advisor): make AI matrix composable Validated 31 contract tests and all 14 scenario snapshots. Full build, artifact lint, vendored parity, skill refs, markdown lint, and gitleaks pass. * test(aws-startup-advisor): expand AI migration matrix * fix: enforce exact gateway source contracts * fix: validate shared neutral fixtures * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly Root mise.toml now matches main exactly. lint:artifacts and lint:vendored-parity are defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience. CI enforcement bypasses mise entirely, same pattern as the existing model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml (PR aws#385): two dedicated workflows invoke the plugin's scripts directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. Net effect: root mise.toml has zero diff against main. The only files outside the plugin are the two GitHub Actions workflow files themselves, which GitHub requires under .github/workflows/ — each only invokes the plugin's own script and carries no shared logic. * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly Root mise.toml now matches main exactly. lint:artifacts, lint:vendored-parity, and lint:skill-refs are defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience. CI enforcement bypasses mise entirely, same pattern as the existing model-staleness.yml precedent (PR aws#348) and artifact-contracts.yml / vendored-parity.yml (PRs aws#385/aws#386): three dedicated workflows invoke the plugin's scripts directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. Net effect: root mise.toml has zero diff against main. The only files outside the plugin are the three GitHub Actions workflow files themselves, which GitHub requires under .github/workflows/ — each only invokes the plugin's own script and carries no shared logic. * test(aws-startup-advisor): define neutral AI workload profile Add a target-only, provider-neutral ai-workload-profile.json contract plus representative fixtures and contract tests, so a future normalization output can preserve multiple AI providers/gateways instead of collapsing them into a lossy both/single-gateway-type shape. No producer or runtime behavior change; schema validity does not imply an implemented adapter or route. * test(aws-startup-advisor): lock observed-source evidence trail and cover ai-workload fixtures in lint:artifacts - Add regression test asserting _assert_profile_semantics rejects an observed ai_source whose model_observations has been emptied while models[] and model_evidence stay intact (the schema alone cannot express that correspondence). - Move the four ai-workload-profiles fixtures into <case>/ai-workload-profile.json subdirectories and register ai-workload-profile.json in artifact-contracts.json's shared section so lint:artifacts (and mise run build) actually open these files. * refactor(aws-startup-advisor): canonicalize aws-design-ai.json contract Canonicalizes Azure's prior schema-design-aws-ai.md into skills/shared/ai/schema-design-aws-ai.md, replacing the retired single-scalar metadata.ai_source (MUST equal summary.ai_source) with many-valued metadata.ai_sources[]/metadata.gateways[] mirroring aws#415's ai-workload-profile.schema.json exactly. Adds the matching target-only scripts/contracts/aws-design-ai.schema.json, vendors the canonical prose into both gcp-to-aws and azure-to-aws references/vendored/ai/, repoints artifact-contracts.json's aws-design-ai.json entry to the shared canonical shape (adding the GCP entry it previously lacked), and reduces Azure's old schema-design-aws-ai.md to a redirect pointer. Target-only / non-runtime, per aws#410 and aws#415's own framing: no producer is repointed in this change (design-ai.md is unchanged in both skills); structural validity does not imply runtime support. Adds plugins/aws-startup-advisor/tests/test_aws_design_ai_contracts.py, a projection/equivalence suite that reuses aws#415's own semantic oracle (_assert_profile_semantics / MATRIX._assert_scenario_semantics) to prove metadata.ai_sources[]/gateways[] project exactly into ai-workload-profile.schema.json's shape, covering both a multi-provider direct case (Anthropic + OpenAI in one design) and a gateway-fronted multi-upstream case (one OpenRouter gateway fronting Anthropic + OpenAI simultaneously). Fixtures are nested as fixtures/aws-design-ai/<case>/ aws-design-ai.json (not flat) and registered in artifact-contracts.json's shared section from the start, per aws#415's own Cursor-review finding about lint:artifacts fixture visibility. Validation (run in the worktree .worktrees/aws-design-ai-contract): - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_aws_design_ai_contracts.py -> 16 passed - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py -> 57 passed (unchanged, regression-clean) - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_workload_profile_contracts.py -> 24 passed (unchanged, regression-clean) - mise run lint:artifacts -> PASS, 80 artifact(s) checked (up from 74 before this change), 0 errors, 50 baselined (pre-existing), 14 info (pre-existing NO_CONTRACT entries, unrelated to this change) - mise run lint:vendored-parity -> PASS, 46 vendored file(s) byte-identical to skills/shared/ and matching README tables - mise run lint:skill-refs -> PASS, 1181 references checked, 0 missing, 7 baselined (pre-existing), 0 stale - mise run lint:md -> 0 issues in 2478 files - mise run validate -> all sub-tasks (lint:manifests, lint:spec, lint:skill-parity, lint:artifacts, lint:vendored-parity, lint:skill-refs) pass - mise run build -> all validate tasks pass; security (gitleaks) scanned 503 commits, ~35.6MB, no leaks found - git diff --check -> clean (exit 0) - git merge-tree --write-tree origin/main HEAD -> clean tree SHA, no conflict markers - git diff against PR aws#415 head (4958a2c) touches only: plugins/aws-startup-advisor/README.md, fixtures/README.md, scripts/artifact-contracts.json, scripts/contracts/aws-design-ai.schema.json, skills/azure-to-aws/references/shared/schema-design-aws-ai.md, skills/{azure,gcp}-to-aws/references/vendored/README.md, skills/{azure,gcp}-to-aws/references/vendored/ai/schema-design-aws-ai.md, skills/shared/ai/README.md, skills/shared/ai/schema-design-aws-ai.md, tests/test_aws_design_ai_contracts.py, and the two new fixtures/aws-design-ai/<case>/aws-design-ai.json files — contracts, fixtures, tests, docs, and vendored files only. No design-ai.md, estimate-ai.md, generate-artifacts-ai.md, or other phase/runtime file touched. * fix(aws-startup-advisor): address PR aws#416 Cursor findings on aws-design-ai contract Finding 1 (blocking): restore Azure's aws-design-ai.json emission contract. The manifest had dropped azure-to-aws's shape/root_heading entry outright instead of repointing it, so Azure's current scalar metadata.ai_source output silently failed the shared target-only schema. Restore skills/azure-to-aws/references/shared/schema-design-aws-ai.md to its PR aws#415 content and restore the azure-to-aws artifact-contracts.json entry so it shadows the shared schema until design-ai.md is repointed. Rewrite the canonical-status section in skills/shared/ai/schema-design-aws-ai.md to name the restored Azure file as the current emission contract and this file as the target. Vendored copies re-synced for lint:vendored-parity. Finding 2 (should-fix): add a regression test proving an observed source with empty model_observations is rejected by the reused aws#415 _assert_profile_semantics helper, locking the completeness gap the schema itself cannot express without $data. Finding 3 (should-fix): add _assert_model_traceability and tests proving every design_blocks[].model_id traces to a metadata.ai_sources[] entry or a gateway upstream, with a negative case for an untraceable model id. Verified: pytest tests/test_aws_design_ai_contracts.py (22 passed), mise run lint:artifacts (0 errors), mise run lint:vendored-parity, mise run build all green. * fix(aws-startup-advisor): close GCP shadow gap, model_observations gap, gateway traceability test gap - Add gcp-to-aws manifest shadow entry for aws-design-ai.json mirroring azure-to-aws's pattern, with a new schema-design-aws-ai.md documenting GCP's current scalar emission; add the matching accept/reject test pair. - Require model_observations minItems:1 on the aiSource observed branch in aws-design-ai.schema.json, closing the schema-level gap the semantic oracle alone used to cover; update the existing test to assert the schema now rejects the mutation directly. - Add two tests exercising the gateway-upstream-walk branch of _assert_model_traceability (previously unexercised by any positive fixture). * fix(aws-startup-advisor): require model_observations on observed ai-workload-profile sources * fix(aws-startup-advisor): sync vendored schema-design-aws-ai.md copies The Azure and GCP vendored copies of schema-design-aws-ai.md had drifted from the canonical skills/shared/ai/schema-design-aws-ai.md — missing the PR's own round-2 fixes (the model_observations minItems note and the GCP manifest-shadow paragraph), which caused lint:vendored-parity to fail in CI. Re-ran tools/sync-vendored.py to bring both copies back to byte-identical with the canonical file. Verified: mise run build passes clean (lint:md, lint:manifests, lint:skill-parity, lint:spec, lint:skill-refs, lint:vendored-parity, lint:artifacts, gitleaks); pytest on test_aws_design_ai_contracts.py and test_ai_workload_profile_contracts.py — 50 passed. * fix(aws-startup-advisor): add missing canonical source for telemetry vendoring PR aws#407 (merged to main) vendored identical telemetry/ files into azure-to-aws, gcp-to-aws, heroku-to-aws, and llm-to-bedrock but the lint:vendored-parity gate did not yet exist on main at that time, so no skills/shared/telemetry/ canonical source or README rows were ever added. Merging upstream/main into this branch (which carries the gate from PR aws#386) now surfaces that pre-existing gap. Add skills/shared/telemetry/ as the canonical source (identical across all four vendored copies, verified via md5sum), add README rows for the eleven telemetry files in the three skills with existing vendored READMEs, and create the vendored README for llm-to-bedrock (which vendors only the telemetry tree and had no README.md at all). Verified: python3 plugins/aws-startup-advisor/tools/sync-vendored.py --check passes (90 files), and mise run build passes clean. * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): match the shared telemetry client to the bundle The parity check treats skills/shared as canonical. That copy was missing the plugin User-Agent the vendored bundles already send, so the check failed. * fix(aws-startup-advisor): address PR aws#410 Cursor review findings Three independently reproduced findings: 1. clarify-ai-only.md's "preferences.json (AI-only)" example contradicted its own prose: ai_framework.value was a bare string while Q1 is documented as multi-select with a ["direct"] default, and the example omitted the mandated metadata.report_caveats[] and never used the source/question_id provenance alternative the prose referenced. Fixed the example to match an array value and added report_caveats; clarified the prose that question_id is this flow's sole provenance field. 2. _assert_scenario_semantics deduplicated gateway upstreams on their raw (provider, source_service-or-None) key BEFORE resolving a service-less upstream to its actual source, so an explicit-service upstream and an overlapping service-less upstream for the same single retained source had different raw keys and both passed. Now dedupes on the resolved source's own (provider, source_service) key. Added a parametrized regression test covering both insertion orderings. 3. strip_jsonc attributed a trailing // comment to the textually-last "key": match on a line, which broke for an inline array/object value that closes before the comment ("call_sites": [{ "file": ..., "line": 1 }], // REQUIRED resolved to "line", not "call_sites"), so omitting the entire required field passed silently. Now tracks bracket depth and binds the comment to the key whose own value's depth matches (or, for a value still open at end of line, is one level shallower than) the depth at the comment. Added unit tests on strip_jsonc directly plus an end-to-end MISSING_REQUIRED regression against the real contract. All three reproduced against the live PR head before fixing. 104 tests pass in the two affected files; full mise run build, lint:artifacts (74 checked, 0 errors), lint:vendored-parity, diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR aws#416 Cursor review findings Five independently reproduced findings: 1. The GCP shadow contract (schema-design-aws-ai.md) put phase/focus/timestamp at the top level, but design-ai.md's own "top-level fields" table nests all three (plus ai_source, bedrock_models_selected) inside metadata -- GCP does not split them the way Azure's parallel shape does. The example also omitted design_blocks[].honest_assessment, required ("not_applicable") on the three traditional-AI capability rows per design-ai.md's Validation Checklist. Fixed the example's nesting and added the conditional honest_assessment field; corrected test_aws_design_ai_contracts.py's _gcp_scalar_design() fixture, which built from the canonical target shape with only metadata swapped and so never exercised the actual producer envelope. 2. test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count assumed only ai-migration-matrix contributes unmapped "shared" fixtures, but ai-workload-profiles/ (aws#415) and aws-design-ai/ (aws#416) are unmapped too. Generalized the assertion to count every unmapped fixture family. 3. clarify-ai-only.md's AI-only example contradicted its own prose (ai_framework as a bare string vs. the documented ["direct"] multi-select default, missing metadata.report_caveats); clarify-ai.md's mixed-flow example omitted its own documented Q21 ai_latency row. Fixed both examples; clarified clarify-ai-only .md's prose that question_id is this flow's sole provenance field. 4. test_positive_fixtures_project_exactly_into_pr415_semantics only ran the weaker profile-level oracle (_assert_profile_semantics), not the stronger matrix-level gateway/source correspondence oracle (_assert_scenario_semantics) that the dedicated negative mutation test already proves is strict. A model dropped from only a gateway upstream in a copied positive fixture passed all existing positive-fixture checks. Now runs both oracles on every positive fixture. 5. artifact-contracts.json's extra_shapes entries point preferences.json's AI fields at clarify-ai.md's "Step 3: Rows returned" example, which showed bare per-fragment values ("ai_priority": "balanced") instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's own rules require and every sibling fragment (e.g. clarify-global.md) already documents in that wrapped form. A properly assembled mixed-Azure artifact failed TYPE_MISMATCH/UNKNOWN_KEY on every AI row. Rewrote the example to show the wrapped envelope, matching the established sibling-fragment convention. All five reproduced against the live PR head before fixing. 181 tests pass in the plugin's full suite; mise run build, lint:artifacts (80 checked, 0 errors), lint:vendored-parity (90 files), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR aws#411 Cursor review findings Four independently reproduced findings: 1. The pre-split Azure cache had its own "### Embeddings — Bedrock" target table (Titan Text Embeddings v1/v2, Cohere Embed v4/English v3/Multilingual v3) nested under GCP's "## Bedrock Models (On-Demand)" heading, but it is a different subsection than the one the parity script's byte-equality check covers, so the split dropped it entirely. estimate-ai.md's design flow still references this table and still recommends Titan v2 as the default embedding target. Moved the table into the canonical cache (scoped out of the main section's byte-equality check, matching the existing Claude Fable exception pattern) and the model_rows() quick-reference scanner (its Dimensions column coincidentally parses as a numeric output price for three rows), synced all three vendors, and added a dedicated row-level parity check. 2. clarify-ai-only.md's AI-only example contradicted its own prose (bare-string ai_framework vs. the documented ["direct"] array default, missing metadata.report_caveats) and route_shapes does not inherit the parent preferences.json entry's allowed_anywhere -- so (REQUIRED on every DETECTED-equivalent row per schema-preferences.md's own fundamental rule) was rejected on this route even though the general contract requires it. Fixed the example and added allowed_anywhere to the AI-only route_shapes entry; added source to this route's extracted rows and corrected the prose accordingly. Added three documented-shape acceptance cases to the existing producer-route test. 3. artifact-contracts.json's extra_shapes pointed preferences.json's mixed-flow AI fields at clarify-ai.md's raw per-fragment "Step 3: Rows returned" values instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's rules require and every sibling fragment (e.g. clarify-global.md) already documents in that wrapped form; the example also omitted the documented Q21 ai_latency row. Rewrote the example to match the established convention, with source on DETECTED rows per finding 2's rule. 4. strip_jsonc attributed a trailing comment to the textually-last "key": match on a line, which broke for the committed workloads shape "call_sites": [{ "file": ..., "line": 1 }], // REQUIRED (resolved to "line", not "call_sites"), so omitting the entire required field passed silently in both Azure preferences routes. Now tracks bracket depth and binds the comment to the key whose own depth matches the end-of-line depth. Added strip_jsonc unit tests plus end-to-end MISSING_REQUIRED regressions on the real contract. All four reproduced against the live PR head before fixing. 73 tests pass in the plugin's full suite plus 23 in llm-to-bedrock's pricing suite; mise run build, lint:artifacts (53 checked, 0 errors), lint:vendored-parity (47 files), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows Correction to an earlier fix on this PR. While fixing the identical finding on PR aws#411, confirmed via schema-preferences.md's own fundamental rule ("the justification key is source, and it is REQUIRED on DETECTED") that source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's chosen_by: "extracted" rows -- the AI-only route's naming for the same DETECTED concept. The earlier fix on this PR incorrectly concluded source was "not part of this AI-only shape" and only clarified the prose to say so, rather than fixing the real root cause: route_shapes builds a shape contract from only the route object, not merged with the parent preferences.json entry, so it never inherited that entry's allowed_anywhere vocabulary (source, reason, context, etc.). Added allowed_anywhere to the AI-only route_shapes manifest entry (mirroring the parent preferences.json entry's), added source to the example's chosen_by: "extracted" rows (ai_framework, ai_monthly_spend, ai_model_baseline, ai_vision), and corrected the prose to say source is required on those rows. Added three acceptance cases (multi-value framework, report_caveats present, source present) to the existing producer-route test, replacing the test case that had asserted source must still fail. 136 tests pass in the plugin's full suite; mise run build, lint:artifacts (74 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows Correction to an earlier fix on this PR. While fixing the identical finding on PR aws#411, confirmed via schema-preferences.md's own fundamental rule ("the justification key is source, and it is REQUIRED on DETECTED") that source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's chosen_by: "extracted" rows -- the AI-only route's naming for the same DETECTED concept. The earlier fix on this PR incorrectly concluded source was "not part of this AI-only shape" and only clarified the prose to say so, rather than fixing the real root cause: route_shapes builds a shape contract from only the route object, not merged with the parent preferences.json entry, so it never inherited that entry's allowed_anywhere vocabulary (source, reason, context, etc.). Added allowed_anywhere to the AI-only route_shapes manifest entry (mirroring the parent preferences.json entry's), added source to the example's chosen_by: "extracted" rows, and corrected the prose. Added three acceptance cases (multi-value framework, report_caveats present, source present) to the existing producer-route test. 181 tests pass in the plugin's full suite; mise run build, lint:artifacts (80 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR aws#410 follow-up Cursor review findings Three independently reproduced findings: 1. Regression in the strip_jsonc depth-tracking fix: the unconditional 'if ch not in "\"\": key_start = None' reset the pending key on ANY non-quote character, including whitespace between a closed key string and its ':'. 'priority' : "balanced", // REQUIRED (space or tab before the colon) lost the key entirely, so omitting workloads[0].priority stopped producing MISSING_REQUIRED. Fixed by exempting whitespace from the reset; only a genuinely different character invalidates the pending key. Added parametrized unit tests for space/tab/double-space before the colon plus a control proving a real intervening token still invalidates. 2. artifact-contracts.json's extra_shapes still pointed preferences.json's mixed-flow AI fields at clarify-ai.md's raw per-fragment 'Step 3: Rows returned' values instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's rules require (the same finding already fixed on aws#411/aws#416, not yet addressed here); the example also omitted the documented Q21 ai_latency row. Rewrote the example to match the established sibling-fragment convention, with source on DETECTED rows per the aws#411 correction. 3. Both clarify-ai.md and clarify-ai-only.md persisted workloads[].capability: "chat", but schema-discover-ai.md's capability enum is text_generation/structured_output/image_generation/embedding/… and design-ai.md has no routing case for "chat". Changed both examples to text_generation and added a test asserting both producer examples use a value from the discovery/design vocabulary. All three reproduced against the live PR head before fixing. 141 tests pass in the plugin's full suite; mise run build, lint:artifacts (74 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * refactor(aws-startup-advisor): move vendored-parity check into plugin dispatcher * fix(aws-startup-advisor): drop stale telemetry rows from vendored READMEs The merged-in sync-vendored.py now excludes telemetry/ from its managed file set (that subtree has its own parity gate, generated from scripts/telemetry/ via sync_bundles.py, not copied from skills/shared/). This branch's vendored READMEs still carried telemetry rows from before that exclusion landed, so every telemetry row failed as 'listed but missing on disk' once rglob stopped counting those files as managed. Removed the telemetry rows from azure-to-aws, gcp-to-aws, and heroku-to-aws's vendored READMEs (each still vendors other, real skills/shared/ files). Deleted llm-to-bedrock's vendored README entirely, since telemetry was the only thing it ever listed and upstream/main carries no README for that skill's vendored dir either. Verified: mise -C plugins/aws-startup-advisor run ci passes (vendored- parity: 46 files OK; artifacts, model-id, skill-refs, and 272 tests all green). mise run build from repo root clean. * fix(aws-startup-advisor): account for every neutral fixture directory, not just ai-migration-matrix test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count hardcoded the assumption that ai-migration-matrix was the only fixture directory with no skill prefix (gcp-/heroku-/azure-), so --fixtures' checked_artifacts total only needed one 'shared artifacts' bucket subtracted out. This PR's own fixtures/ai-workload-profiles/ directory is a second such neutral directory (its 4 ai-workload-profile.json files are themselves registered as a 'shared' contract in artifact-contracts.json), so --fixtures counts them in checked_artifacts, but the test's formula never added them to the subtracted total -- failing with 'assert (78 - 53) == 21' instead of the correct 25. Generalized the test to sum shared-artifact counts and validate_run_dir checks across every fixture directory that skill_for_fixture maps to None, not just the matrix by name. This also guards against the same accounting gap recurring if a third neutral fixture directory is added later. Verification: the fixed test passes; mise -C plugins/aws-startup-advisor run ci passes in full (vendored-parity 44 files, artifacts 78 checked/0 errors, model-id, skill-refs, 246 tests); mise run build from repo root clean; git diff --check clean. --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Gen Li <i.ligen@gmail.com>
lamardealmaker
pushed a commit
to lamardealmaker/agent-toolkit-for-aws
that referenced
this pull request
Oct 14, 2026
* feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas
The migration skills pass ~17 JSON artifacts between phases, each with a
contract published as a schema-*.md "shape" block (JSON/JSONC example) or,
for two of them, a real JSON Schema under skills/shared/. Nothing checked
that a producer writes what the next phase reads: a key could be renamed,
emitted without being documented, or an enum widened, and every DSL
_assert still passed because the model that writes the artifact evaluates
the assertion. This is the precondition the backlog names for any engine
consolidation or AI-engine extraction — moves across adapters have no
drift detection without it.
scripts/validate-artifacts.py (stdlib, Python 3.9+)
- Parses each contract document's example blocks (JSONC comments, trailing
commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments,
`// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open
object, `"<key>": {...}` = template for every key) into a key-path
contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`)
refine paths; manifest can pin a block by `@L<line>` anchor, merge other
documents' additions (graviton, workshop), or declare a row-annotation
vocabulary (azure preferences' source/reason/context/...).
- Validates real JSON Schemas with a draft-07 subset (type, required,
properties, additionalProperties, enum, const, items, pattern, min/max,
minItems, allOf/anyOf/oneOf, $ref).
- Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION,
SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken),
NO_CONTRACT (info). `_`-prefixed keys are annotations.
- Modes: --self-check (contracts parse), --fixtures (CI; goldens against
the baseline), --run-dir <.migration/run> --skill <name> (users).
- scripts/artifact-contracts.json maps artifact → contract per skill.
- scripts/artifact-contracts-baseline.json: 21 explained entries covering
44 known gaps between what phases emit and what their contract documents
say, each naming the document that must change. Stale entries are
reported so the list only shrinks.
Contract documents fixed where the document contradicted itself (its own
prose named a key its example block omitted):
- gcp schema-preferences.md: the shape block was not JSON (`{ "value",
"chosen_by" }` set notation); rewritten as a JSONC row template.
- gcp schema-estimate-infra.md: root block lacked complexity_tier, which
estimation-infra.schema.json marks required.
- gcp schema-discover-iac.md § Live discovery extensions: block now shows
the metadata/resources[] additions the prose above it documents.
- heroku schema-discover-heroku.md: resources[] example carries
source/unmanaged_by_terraform/not_found_live (table at lines 93-95);
addon config example carries the per-add-on and live-enrichment fields
the bullets below it list.
- azure schema-design-aws.md: services[] block carries
routing_provenance/sizing_provenance/sizing_source/is_compute_unit/
hosted_app_azure_ids (the same document marks them REQUIRED); deferred[]
carries routing_provenance.
- azure schema-discover-azure.md: clusters[] carries pattern_status
(discover-assemble.md § Status).
- azure schema-preferences.md: top-level clarify_status (REQUIRED per
clarify-assemble.md § clarify_status).
Fixture: gcp-decision-gate's estimation-infra.json stub gains the four
keys estimation-infra.schema.json requires; check_expected_decide.py
still passes.
Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it).
README scripts/ entry updated.
Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping,
template inference (enum/required/placeholder/any-key), each finding
code, JSON Schema subset, baseline matching + stale detection, CLI
self-check and fixtures pass on the real repo, and two negatives: an
injected off-contract key and a schema enum violation both exit 1.
* fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails
Review on aws#385, both should-fix findings valid:
1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`;
the draft-07 subset did not implement it, so `phases: {}` passed. Fixed
the instance AND the class: the subset now enforces minProperties,
maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems,
minLength/maxLength and `not`; and `unsupported_keywords()` walks every
loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any
validation keyword that is neither enforced nor a known annotation. A
constraint can no longer be skipped silently — adding a keyword to a
schema without teaching the validator fails the gate.
2. [Completeness] a baseline entry that matched nothing was printed as a
warning and exited 0, so the burn-down list only shrank if someone read
the log. STALE_BASELINE is now an error code: `--fixtures` (what
`mise run lint:artifacts` runs) exits 1 on any stale entry.
Tests (19): empty `phases` fails with minProperties named; the two real
schemas use only enforced/annotation keywords and an invented
`contentEncoding` / `dependencies` schema is rejected; a never-matching
baseline entry exits 1 with STALE_BASELINE.
* ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/
Each migration skill is installable on its own (`npx skills add … --skill
gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate
schemas, …) is vendored into skills/<skill>/references/vendored/ from the
canonical skills/shared/. The vendored READMEs said: "This repository has
no automated sync task for these copies — keeping them byte-identical is
part of the change that touches the canonical file." All 44 copies are
identical today; nothing stopped the next edit to skills/shared/ from
leaving a skill running stale logic.
- tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py.
Default mode copies skills/shared/<path> over every vendored copy that
differs; --check exits 1 listing any copy that differs or has no
canonical source (a vendored file nobody owns). README.md in a vendored
dir is explanatory and skipped. The vendored tree is a subset of shared/,
so files in shared/ that no skill vendors are fine.
- mise `lint:vendored-parity`, added to `validate` (the CI build runs it).
- The four vendored READMEs replace the manual cp + md5sum instructions
with the tool and the CI rule.
- tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and
orphan detection (README excluded), sync repairs drift and leaves orphans
as errors, CLI exit codes — plus the real repository is in sync.
No skill content changes. Stacked on aws#385 only because both add a task to
the same `validate` depends line.
* fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees
Review on aws#386 (Contradictions / Completeness, should-fix): the three
migration READMEs claimed "a change to skills/shared/ that forgets the
copies cannot merge", but the check only walked files that still existed —
a deleted vendored copy, or a new shared file never vendored, passed.
Rather than only weakening the sentence, the gate now uses the record of
intent each vendored README already carries: its
`| Vendored path | Canonical source |` table.
- check: every row must exist on disk (deleted copy → FAIL), every file on
disk must have a row (README stays honest), and each row's canonical
path must be skills/shared/<same path>. A README with no table is an
error.
- sync: restores a listed-but-missing copy from canonical before comparing.
- What the gate still cannot see — a NEW skills/shared/ file no skill
vendors — is now stated in the tool docstring and in all four READMEs,
with the instruction "add the copy AND a table row".
The three READMEs were already complete tables (gcp 14/14, heroku 8/8,
azure 21/21, agent-advisor 1/1); the real repo passes unchanged.
Tests (7): deleted copy caught via the table and restored by sync;
unlisted file and wrong canonical path caught; a new shared file nobody
vendors is NOT an error (pins the documented limit); CLI exit codes; real
repo in sync. Reviewer probes re-run on a throwaway copy of the plugin:
deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode
restores it; adding skills/shared/brand-new.md → no error, as documented.
* ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist
The skills are prose state machines: a phase file says "Load
`references/phases/design/design-ai.md`" or "run `python3
\"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing
checked the target exists. A renamed or never-written file is a dead end
the user hits mid-migration, and any file move during consolidation would
break references silently. This is the third gate in the family
(validate-artifacts.py: artifact contracts; sync-vendored.py: vendored
parity) and the precondition for moving phase files between skills.
tools/check-skill-refs.py (stdlib)
- Scans skills/**/*.md and agents/*.md for path tokens into the skill
tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the
agent-advisor short forms phases/, shared/, design-refs/) with optional
$PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> /
$GCP_BASE / ./ / ../ prefixes.
- Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form> → plugin/repo-relative. Library-origin
files (skills/shared/*, */references/vendored/*, */references/shared/*)
speak from the consuming skill's point of view and resolve against
skills/shared and every skill; agent-advisor and llm-to-bedrock may name
gcp-to-aws's tree (documented inline execution); agents/ resolve against
llm-to-bedrock.
- Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…),
templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no
skill-tree root in them (generated outputs like ./deploy.sh).
- tools/skill-refs-baseline.json: `ignore` rules (permanent policy —
scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in
the user's app) and `entries` (known dead references with the fix;
STALE when they stop matching, failing the gate so the list only shrinks).
1118 references checked on the real repo; 7 dead ones baselined, each a
real user-facing defect: azure discover-iac.md points Bicep/ARM input at
extract-bicep.md / extract-arm.md that do not exist (so Bicep halts);
design-refs/patterns.md referenced from two azure files before it exists;
contextual-offers SKILL.md names references/offers.md unqualified when the
file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at
shared/ai-model-lifecycle.md, which in gcp is vendored at
references/vendored/ai/.
mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution
rule, each skip rule, library-origin resolution, gcp-dependent and agents
fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS.
* fix(aws-startup-advisor): check-skill-refs — agents/ resolve only against llm-to-bedrock
Review on aws#387 (Completeness / Contradictions, should-fix): the agents/
branch of the resolver searched every skill and kept the first hit, so
agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml`
resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a
tree with the file only under agent-advisor still passed — deleting the
llm-to-bedrock copy would not have failed the gate. The comment above the
loop said llm-to-bedrock; the loop did not.
- Agent references now resolve ONLY against skills/llm-to-bedrock
(_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to
that skill.
- Tests: scripts/pyproject.toml present only under agent-advisor is
MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy
even when agent-advisor also has it; the fixture tree's agents/worker.md
reference to another skill's file is now correctly MISSING.
- Real repo unchanged: 1118 references, 0 missing, 7 baselined.
* fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths
Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on
aws#386/aws#387 against the same code):
- The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it
now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no
baseline unless --baseline names one, and then every entry must still match (a valid
real run previously failed with 21 STALE_BASELINE errors).
- Node.merge keeps a declared enum when another example shows a concrete member
(`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document
shows joins the set, only a free-text example clears it. Adopted children are copied,
so refining `design_constraints.cpu_architecture` no longer widens the generic
`<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`.
- `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is
valid, `terraform+bogus` is not, closed enums elsewhere stay closed.
- Heading match is exact or word-boundary prefix: `graviton` no longer selects
`graviton_profile`; the Azure design manifest names `graviton block` explicitly.
- Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the
array level, so workload key/type drift is detected; bare item examples still merge
into the item.
- `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku
`metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged.
Two baseline entries record the hand-authored Heroku seed inventories that lack them.
- GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline
and breakdown.security_baseline_compliance bind to their object keys (line anchors
separate the two Security Baseline blocks); nested drift inside them is still caught.
- Heroku workshop `preferences_subset` is the documented open knob map (open_paths).
15 regression tests added (unit + CLI against the real contracts); each fails on the
previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale.
* fix(aws-startup-advisor): check-skill-refs — run-artifact skip is per token; explicit bases are authoritative
Review round 2 on aws#387 (the two findings unique to this tool):
- The run-artifact skip looked 40 characters back, so an earlier, separate token
(`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line
(azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the
token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…`
forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve).
- An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now
authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a
same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look
alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md`
traversal still resolves.
Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing,
7 baselined, 0 stale.
* fix(aws-startup-advisor): admit the unfired licensing stub in the preferences example
The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): reject refined empty-array items and honor relative prefixes
An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): accept each Azure preference route and typed drift values
An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate.
* fix: ignore Bedrock run state references
* fix(aws-startup-advisor): keep skill-refs and vendored-parity tooling plugin-local
Move tools/check-skill-refs.py, tools/skill-refs-baseline.json, and
tools/sync-vendored.py (plus their tests) into
plugins/aws-startup-advisor/tools/. Both gates are owned by, and only ever
scan, this plugin — check-skill-refs.py was already hardcoded to
DEFAULT_PLUGIN=aws-startup-advisor, and sync-vendored.py's vendored/shared
pattern has no other consumer in this repository.
- check-skill-refs.py and its baseline/test now resolve relative to the
script's own plugin-local position (PLUGIN_ROOT) instead of a root-level
tools/ directory; REPO_ROOT is still derived (two levels further up) for
the one genuinely repo-relative fallback candidate and for display paths.
- sync-vendored.py: same single-plugin fix as PR aws#386 (PLUGIN_ROOT instead
of scanning every plugins/ subdirectory; --plugin-root replaces the
removed multi-plugin --plugin/--plugins-root surface).
- Updated mise.toml's lint:skill-refs and lint:vendored-parity tasks and the
four vendored/README.md usage snippets to the new paths.
- Updated the baseline file's _what_this_is pointer to the new script path.
* test(aws-startup-advisor): pin AI migration matrix
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (6 passed)\n- mise run lint:artifacts (passed)\n- mise run lint:vendored-parity (passed)\n- mise run lint:skill-refs (passed)\n- mise run lint:md (passed)\n- mise run validate (passed)\n- mise run build (passed)
* fix(aws-startup-advisor): fail closed integration verdict
Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (7 passed)\n- mise run lint:artifacts (70 checked, 0 errors)\n- mise run lint:vendored-parity (44 files passed)\n- mise run lint:skill-refs (1160 refs passed)\n- mise run lint:md (2475 files, 0 issues)\n- mise run validate (passed)\n- mise run build (passed; gitleaks found no leaks)\n- git diff --check (passed)
* fix: align integration validation run IDs
Accept uppercase UUID hex in integration validation to match the phase-status contract, cover uppercase/lowercase/malformed IDs through the artifact validator, and document the distinct OpenRouter fixture pair.
Validated: targeted matrix suite (8 passed); uppercase artifact probe; empty-reference rejection; failed infrastructure/AI/IAM children; phase-status schema probe; mise run build; git diff --check; clean merge-tree against upstream/main.
* test(aws-startup-advisor): make AI matrix composable
Validated 31 contract tests and all 14 scenario snapshots. Full build, artifact lint, vendored parity, skill refs, markdown lint, and gitleaks pass.
* test(aws-startup-advisor): expand AI migration matrix
* fix: enforce exact gateway source contracts
* fix: validate shared neutral fixtures
* fix(aws-startup-advisor): keep telemetry under its bundle gate
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(aws-startup-advisor): address PR aws#410 Cursor review findings
Three independently reproduced findings:
1. clarify-ai-only.md's "preferences.json (AI-only)" example contradicted
its own prose: ai_framework.value was a bare string while Q1 is
documented as multi-select with a ["direct"] default, and the example
omitted the mandated metadata.report_caveats[] and never used the
source/question_id provenance alternative the prose referenced. Fixed
the example to match an array value and added report_caveats; clarified
the prose that question_id is this flow's sole provenance field.
2. _assert_scenario_semantics deduplicated gateway upstreams on their raw
(provider, source_service-or-None) key BEFORE resolving a service-less
upstream to its actual source, so an explicit-service upstream and an
overlapping service-less upstream for the same single retained source
had different raw keys and both passed. Now dedupes on the resolved
source's own (provider, source_service) key. Added a parametrized
regression test covering both insertion orderings.
3. strip_jsonc attributed a trailing // comment to the textually-last
"key": match on a line, which broke for an inline array/object value
that closes before the comment ("call_sites": [{ "file": ..., "line":
1 }], // REQUIRED resolved to "line", not "call_sites"), so omitting
the entire required field passed silently. Now tracks bracket depth and
binds the comment to the key whose own value's depth matches (or, for a
value still open at end of line, is one level shallower than) the
depth at the comment. Added unit tests on strip_jsonc directly plus an
end-to-end MISSING_REQUIRED regression against the real contract.
All three reproduced against the live PR head before fixing. 104 tests
pass in the two affected files; full mise run build, lint:artifacts (74
checked, 0 errors), lint:vendored-parity, diff --check, and merge-tree
against upstream/main are all clean.
* fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows
Correction to an earlier fix on this PR. While fixing the identical finding
on PR aws#411, confirmed via schema-preferences.md's own fundamental rule
("the justification key is source, and it is REQUIRED on DETECTED") that
source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's
chosen_by: "extracted" rows -- the AI-only route's naming for the same
DETECTED concept. The earlier fix on this PR incorrectly concluded source
was "not part of this AI-only shape" and only clarified the prose to say
so, rather than fixing the real root cause: route_shapes builds a shape
contract from only the route object, not merged with the parent
preferences.json entry, so it never inherited that entry's
allowed_anywhere vocabulary (source, reason, context, etc.).
Added allowed_anywhere to the AI-only route_shapes manifest entry
(mirroring the parent preferences.json entry's), added source to the
example's chosen_by: "extracted" rows (ai_framework, ai_monthly_spend,
ai_model_baseline, ai_vision), and corrected the prose to say source is
required on those rows. Added three acceptance cases (multi-value
framework, report_caveats present, source present) to the existing
producer-route test, replacing the test case that had asserted source
must still fail.
136 tests pass in the plugin's full suite; mise run build, lint:artifacts
(74 checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
* fix(aws-startup-advisor): address PR aws#410 follow-up Cursor review findings
Three independently reproduced findings:
1. Regression in the strip_jsonc depth-tracking fix: the unconditional
'if ch not in "\"\": key_start = None' reset the pending key on ANY
non-quote character, including whitespace between a closed key string
and its ':'. 'priority' : "balanced", // REQUIRED (space or tab before
the colon) lost the key entirely, so omitting workloads[0].priority
stopped producing MISSING_REQUIRED. Fixed by exempting whitespace from
the reset; only a genuinely different character invalidates the pending
key. Added parametrized unit tests for space/tab/double-space before the
colon plus a control proving a real intervening token still invalidates.
2. artifact-contracts.json's extra_shapes still pointed preferences.json's
mixed-flow AI fields at clarify-ai.md's raw per-fragment 'Step 3: Rows
returned' values instead of the final assembled {disposition, value,
default} envelope clarify-assemble.md's rules require (the same finding
already fixed on aws#411/aws#416, not yet addressed here); the example also
omitted the documented Q21 ai_latency row. Rewrote the example to match
the established sibling-fragment convention, with source on DETECTED
rows per the aws#411 correction.
3. Both clarify-ai.md and clarify-ai-only.md persisted
workloads[].capability: "chat", but schema-discover-ai.md's capability
enum is text_generation/structured_output/image_generation/embedding/…
and design-ai.md has no routing case for "chat". Changed both examples
to text_generation and added a test asserting both producer examples
use a value from the discovery/design vocabulary.
All three reproduced against the live PR head before fixing. 141 tests
pass in the plugin's full suite; mise run build, lint:artifacts (74
checked, 0 errors), diff --check, and merge-tree against upstream/main
are all clean.
---------
Co-authored-by: Logan Kleier <lkleier@amazon.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
herosjourney
pushed a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 23, 2026
This PR changed two root files only to make the Render scripts' local virtual environment pass the root spec check: a .venv skip in tools/validate_spec.py and a version pin on the root lint:spec task. Both are repo-wide tooling and are outside the plugin folder. Merged Startup Advisor gates (aws#348, aws#385, aws#386, aws#387) kept their checks in the plugin's own mise.toml and changed nothing outside the folder. Restore both files to main's content. Nothing in the Render change depends on them: the Render tests pin their own dependencies in the scripts project, and no venv is committed. Verified with root validate, root build (gitleaks clean), plugin ci (575 passed) and the Render suite (236 passed) on the restored tree.
lamardealmaker
pushed a commit
to lamardealmaker/agent-toolkit-for-aws
that referenced
this pull request
Oct 24, 2026
…s#449) * feat(aws-startup-advisor): artifact-contract validator — check phase artifacts against their published schemas The migration skills pass ~17 JSON artifacts between phases, each with a contract published as a schema-*.md "shape" block (JSON/JSONC example) or, for two of them, a real JSON Schema under skills/shared/. Nothing checked that a producer writes what the next phase reads: a key could be renamed, emitted without being documented, or an enum widened, and every DSL _assert still passed because the model that writes the artifact evaluates the assertion. This is the precondition the backlog names for any engine consolidation or AI-engine extraction — moves across adapters have no drift detection without it. scripts/validate-artifacts.py (stdlib, Python 3.9+) - Parses each contract document's example blocks (JSONC comments, trailing commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments, `// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open object, `"<key>": {...}` = template for every key) into a key-path contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`) refine paths; manifest can pin a block by `@L<line>` anchor, merge other documents' additions (graviton, workshop), or declare a row-annotation vocabulary (azure preferences' source/reason/context/...). - Validates real JSON Schemas with a draft-07 subset (type, required, properties, additionalProperties, enum, const, items, pattern, min/max, minItems, allOf/anyOf/oneOf, $ref). - Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION, SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken), NO_CONTRACT (info). `_`-prefixed keys are annotations. - Modes: --self-check (contracts parse), --fixtures (CI; goldens against the baseline), --run-dir <.migration/run> --skill <name> (users). - scripts/artifact-contracts.json maps artifact → contract per skill. - scripts/artifact-contracts-baseline.json: 21 explained entries covering 44 known gaps between what phases emit and what their contract documents say, each naming the document that must change. Stale entries are reported so the list only shrinks. Contract documents fixed where the document contradicted itself (its own prose named a key its example block omitted): - gcp schema-preferences.md: the shape block was not JSON (`{ "value", "chosen_by" }` set notation); rewritten as a JSONC row template. - gcp schema-estimate-infra.md: root block lacked complexity_tier, which estimation-infra.schema.json marks required. - gcp schema-discover-iac.md § Live discovery extensions: block now shows the metadata/resources[] additions the prose above it documents. - heroku schema-discover-heroku.md: resources[] example carries source/unmanaged_by_terraform/not_found_live (table at lines 93-95); addon config example carries the per-add-on and live-enrichment fields the bullets below it list. - azure schema-design-aws.md: services[] block carries routing_provenance/sizing_provenance/sizing_source/is_compute_unit/ hosted_app_azure_ids (the same document marks them REQUIRED); deferred[] carries routing_provenance. - azure schema-discover-azure.md: clusters[] carries pattern_status (discover-assemble.md § Status). - azure schema-preferences.md: top-level clarify_status (REQUIRED per clarify-assemble.md § clarify_status). Fixture: gcp-decision-gate's estimation-infra.json stub gains the four keys estimation-infra.schema.json requires; check_expected_decide.py still passes. Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it). README scripts/ entry updated. Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping, template inference (enum/required/placeholder/any-key), each finding code, JSON Schema subset, baseline matching + stale detection, CLI self-check and fixtures pass on the real repo, and two negatives: an injected off-contract key and a schema enum violation both exit 1. * fix(aws-startup-advisor): validate-artifacts — enforce minProperties (and reject unknown schema keywords); stale baseline fails Review on #385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE. * ci(aws-startup-advisor): vendored-parity gate — skills' references/vendored/ must match skills/shared/ Each migration skill is installable on its own (`npx skills add … --skill gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate schemas, …) is vendored into skills/<skill>/references/vendored/ from the canonical skills/shared/. The vendored READMEs said: "This repository has no automated sync task for these copies — keeping them byte-identical is part of the change that touches the canonical file." All 44 copies are identical today; nothing stopped the next edit to skills/shared/ from leaving a skill running stale logic. - tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py. Default mode copies skills/shared/<path> over every vendored copy that differs; --check exits 1 listing any copy that differs or has no canonical source (a vendored file nobody owns). README.md in a vendored dir is explanatory and skipped. The vendored tree is a subset of shared/, so files in shared/ that no skill vendors are fine. - mise `lint:vendored-parity`, added to `validate` (the CI build runs it). - The four vendored READMEs replace the manual cp + md5sum instructions with the tool and the CI rule. - tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and orphan detection (README excluded), sync repairs drift and leaves orphans as errors, CLI exit codes — plus the real repository is in sync. No skill content changes. Stacked on #385 only because both add a task to the same `validate` depends line. * fix(aws-startup-advisor): sync-vendored — README table is the manifest; deleted copies fail; READMEs say what the gate sees Review on #386 (Contradictions / Completeness, should-fix): the three migration READMEs claimed "a change to skills/shared/ that forgets the copies cannot merge", but the check only walked files that still existed — a deleted vendored copy, or a new shared file never vendored, passed. Rather than only weakening the sentence, the gate now uses the record of intent each vendored README already carries: its `| Vendored path | Canonical source |` table. - check: every row must exist on disk (deleted copy → FAIL), every file on disk must have a row (README stays honest), and each row's canonical path must be skills/shared/<same path>. A README with no table is an error. - sync: restores a listed-but-missing copy from canonical before comparing. - What the gate still cannot see — a NEW skills/shared/ file no skill vendors — is now stated in the tool docstring and in all four READMEs, with the instruction "add the copy AND a table row". The three READMEs were already complete tables (gcp 14/14, heroku 8/8, azure 21/21, agent-advisor 1/1); the real repo passes unchanged. Tests (7): deleted copy caught via the table and restored by sync; unlisted file and wrong canonical path caught; a new shared file nobody vendors is NOT an error (pins the documented limit); CLI exit codes; real repo in sync. Reviewer probes re-run on a throwaway copy of the plugin: deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode restores it; adding skills/shared/brand-new.md → no error, as documented. * ci(aws-startup-advisor): reference-path lint — every file path a skill mentions must exist The skills are prose state machines: a phase file says "Load `references/phases/design/design-ai.md`" or "run `python3 \"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing checked the target exists. A renamed or never-written file is a dead end the user hits mid-migration, and any file move during consolidation would break references silently. This is the third gate in the family (validate-artifacts.py: artifact contracts; sync-vendored.py: vendored parity) and the precondition for moving phase files between skills. tools/check-skill-refs.py (stdlib) - Scans skills/**/*.md and agents/*.md for path tokens into the skill tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the agent-advisor short forms phases/, shared/, design-refs/) with optional $PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> / $GCP_BASE / ./ / ../ prefixes. - Resolution: explicit prefix → skill-relative → file-relative → skill/references/<short form> → plugin/repo-relative. Library-origin files (skills/shared/*, */references/vendored/*, */references/shared/*) speak from the consuming skill's point of view and resolve against skills/shared and every skill; agent-advisor and llm-to-bedrock may name gcp-to-aws's tree (documented inline execution); agents/ resolve against llm-to-bedrock. - Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…), templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no skill-tree root in them (generated outputs like ./deploy.sh). - tools/skill-refs-baseline.json: `ignore` rules (permanent policy — scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in the user's app) and `entries` (known dead references with the fix; STALE when they stop matching, failing the gate so the list only shrinks). 1118 references checked on the real repo; 7 dead ones baselined, each a real user-facing defect: azure discover-iac.md points Bicep/ARM input at extract-bicep.md / extract-arm.md that do not exist (so Bicep halts); design-refs/patterns.md referenced from two azure files before it exists; contextual-offers SKILL.md names references/offers.md unqualified when the file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at shared/ai-model-lifecycle.md, which in gcp is vendored at references/vendored/ai/. mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution rule, each skip rule, library-origin resolution, gcp-dependent and agents fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS. * fix(aws-startup-advisor): check-skill-refs — agents/ resolve only against llm-to-bedrock Review on #387 (Completeness / Contradictions, should-fix): the agents/ branch of the resolver searched every skill and kept the first hit, so agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml` resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a tree with the file only under agent-advisor still passed — deleting the llm-to-bedrock copy would not have failed the gate. The comment above the loop said llm-to-bedrock; the loop did not. - Agent references now resolve ONLY against skills/llm-to-bedrock (_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to that skill. - Tests: scripts/pyproject.toml present only under agent-advisor is MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy even when agent-advisor also has it; the fixture tree's agents/worker.md reference to another skill's file is now correctly MISSING. - Real repo unchanged: 1118 references, 0 missing, 7 baselined. * fix(aws-startup-advisor): validate-artifacts — fixture baseline only for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on #385 (seven P2s, plus the Azure composite-source enum raised on #386/#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale. * fix(aws-startup-advisor): check-skill-refs — run-artifact skip is per token; explicit bases are authoritative Review round 2 on #387 (the two findings unique to this tool): - The run-artifact skip looked 40 characters back, so an earlier, separate token (`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line (azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…` forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve). - An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md` traversal still resolves. Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing, 7 baselined, 0 stale. * fix(aws-startup-advisor): admit the unfired licensing stub in the preferences example The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): close three artifact and vendored-parity gaps An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): reject refined empty-array items and honor relative prefixes An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): require the Azure discovery verdict and check estimate snapshots A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): accept each Azure preference route and typed drift values An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate. * fix: ignore Bedrock run state references * fix(aws-startup-advisor): keep vendored-parity tooling plugin-local Move tools/sync-vendored.py and its test into plugins/aws-startup-advisor/tools/, since the vendored/shared pattern it gates is unique to this plugin (no other plugin uses references/vendored/). - Script now resolves PLUGIN_ROOT from its own location instead of scanning every plugins/ subdirectory; dropped the now-unneeded --plugin/--plugins-root multi-plugin CLI surface in favor of a single --plugin-root override for tests. - Updated the test suite to match (plugin-root fixtures instead of a multi-plugin tree; the MCP-only-plugin skip test now asserts a clean exit for a plugin root with no skills/ at all). - Updated mise.toml's lint:vendored-parity task and the four vendored/ README.md usage snippets (agent-advisor, heroku-to-aws, gcp-to-aws, azure-to-aws) to the new path. * fix(aws-startup-advisor): keep skill-refs and vendored-parity tooling plugin-local Move tools/check-skill-refs.py, tools/skill-refs-baseline.json, and tools/sync-vendored.py (plus their tests) into plugins/aws-startup-advisor/tools/. Both gates are owned by, and only ever scan, this plugin — check-skill-refs.py was already hardcoded to DEFAULT_PLUGIN=aws-startup-advisor, and sync-vendored.py's vendored/shared pattern has no other consumer in this repository. - check-skill-refs.py and its baseline/test now resolve relative to the script's own plugin-local position (PLUGIN_ROOT) instead of a root-level tools/ directory; REPO_ROOT is still derived (two levels further up) for the one genuinely repo-relative fallback candidate and for display paths. - sync-vendored.py: same single-plugin fix as PR #386 (PLUGIN_ROOT instead of scanning every plugins/ subdirectory; --plugin-root replaces the removed multi-plugin --plugin/--plugins-root surface). - Updated mise.toml's lint:skill-refs and lint:vendored-parity tasks and the four vendored/README.md usage snippets to the new paths. - Updated the baseline file's _what_this_is pointer to the new script path. * test(aws-startup-advisor): pin AI migration matrix Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (6 passed)\n- mise run lint:artifacts (passed)\n- mise run lint:vendored-parity (passed)\n- mise run lint:skill-refs (passed)\n- mise run lint:md (passed)\n- mise run validate (passed)\n- mise run build (passed) * fix(aws-startup-advisor): fail closed integration verdict Validation:\n- uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py (7 passed)\n- mise run lint:artifacts (70 checked, 0 errors)\n- mise run lint:vendored-parity (44 files passed)\n- mise run lint:skill-refs (1160 refs passed)\n- mise run lint:md (2475 files, 0 issues)\n- mise run validate (passed)\n- mise run build (passed; gitleaks found no leaks)\n- git diff --check (passed) * refactor(aws-startup-advisor): separate Bedrock pricing cache Extract provider-neutral Bedrock rates into the shared AI canonical/vendor pattern and repoint GCP, llm-to-bedrock, and agent-advisor consumers without changing pricing behavior. Validation: parity script PASS (54 model keys, 24 static fallbacks, representative estimates unchanged); llm pricing 21 passed; agent-advisor 323 passed; Azure estimate fixture PASS; mise run build PASS. * fix(aws-startup-advisor): preserve pricing accuracy metadata * fix: align integration validation run IDs Accept uppercase UUID hex in integration validation to match the phase-status contract, cover uppercase/lowercase/malformed IDs through the artifact validator, and document the distinct OpenRouter fixture pair. Validated: targeted matrix suite (8 passed); uppercase artifact probe; empty-reference rejection; failed infrastructure/AI/IAM children; phase-status schema probe; mise run build; git diff --check; clean merge-tree against upstream/main. * fix(aws-startup-advisor): correct Bedrock cache ownership Route Azure Bedrock estimates through the canonical vendored cache, remove duplicate Azure Bedrock rows, and align infrastructure and AI accuracy contracts. Validated: pricing tests 21 passed; agent-advisor 323 passed; model/vendor tests 19 passed; Azure/GCP fixtures passed; parity checker and four mutation probes passed; artifact, vendored, skill-ref, Markdown, aggregate validation, full build, and gitleaks passed. * test(aws-startup-advisor): make AI matrix composable Validated 31 contract tests and all 14 scenario snapshots. Full build, artifact lint, vendored parity, skill refs, markdown lint, and gitleaks pass. * test(aws-startup-advisor): expand AI migration matrix * fix: enforce exact gateway source contracts * fix: validate shared neutral fixtures * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly Root mise.toml now matches main exactly. lint:artifacts and lint:vendored-parity are defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience. CI enforcement bypasses mise entirely, same pattern as the existing model-staleness.yml precedent (PR #348) and artifact-contracts.yml (PR #385): two dedicated workflows invoke the plugin's scripts directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. Net effect: root mise.toml has zero diff against main. The only files outside the plugin are the two GitHub Actions workflow files themselves, which GitHub requires under .github/workflows/ — each only invokes the plugin's own script and carries no shared logic. * fix(aws-startup-advisor): keep mise.toml plugin-local; CI runs scripts directly Root mise.toml now matches main exactly. lint:artifacts, lint:vendored-parity, and lint:skill-refs are defined in a new, fully self-contained plugins/aws-startup-advisor/mise.toml instead, for local dev convenience. CI enforcement bypasses mise entirely, same pattern as the existing model-staleness.yml precedent (PR #348) and artifact-contracts.yml / vendored-parity.yml (PRs #385/#386): three dedicated workflows invoke the plugin's scripts directly via python3, scoped to pull requests touching plugins/aws-startup-advisor/**. Net effect: root mise.toml has zero diff against main. The only files outside the plugin are the three GitHub Actions workflow files themselves, which GitHub requires under .github/workflows/ — each only invokes the plugin's own script and carries no shared logic. * test(aws-startup-advisor): define neutral AI workload profile Add a target-only, provider-neutral ai-workload-profile.json contract plus representative fixtures and contract tests, so a future normalization output can preserve multiple AI providers/gateways instead of collapsing them into a lossy both/single-gateway-type shape. No producer or runtime behavior change; schema validity does not imply an implemented adapter or route. * test(aws-startup-advisor): lock observed-source evidence trail and cover ai-workload fixtures in lint:artifacts - Add regression test asserting _assert_profile_semantics rejects an observed ai_source whose model_observations has been emptied while models[] and model_evidence stay intact (the schema alone cannot express that correspondence). - Move the four ai-workload-profiles fixtures into <case>/ai-workload-profile.json subdirectories and register ai-workload-profile.json in artifact-contracts.json's shared section so lint:artifacts (and mise run build) actually open these files. * refactor(aws-startup-advisor): canonicalize aws-design-ai.json contract Canonicalizes Azure's prior schema-design-aws-ai.md into skills/shared/ai/schema-design-aws-ai.md, replacing the retired single-scalar metadata.ai_source (MUST equal summary.ai_source) with many-valued metadata.ai_sources[]/metadata.gateways[] mirroring #415's ai-workload-profile.schema.json exactly. Adds the matching target-only scripts/contracts/aws-design-ai.schema.json, vendors the canonical prose into both gcp-to-aws and azure-to-aws references/vendored/ai/, repoints artifact-contracts.json's aws-design-ai.json entry to the shared canonical shape (adding the GCP entry it previously lacked), and reduces Azure's old schema-design-aws-ai.md to a redirect pointer. Target-only / non-runtime, per #410 and #415's own framing: no producer is repointed in this change (design-ai.md is unchanged in both skills); structural validity does not imply runtime support. Adds plugins/aws-startup-advisor/tests/test_aws_design_ai_contracts.py, a projection/equivalence suite that reuses #415's own semantic oracle (_assert_profile_semantics / MATRIX._assert_scenario_semantics) to prove metadata.ai_sources[]/gateways[] project exactly into ai-workload-profile.schema.json's shape, covering both a multi-provider direct case (Anthropic + OpenAI in one design) and a gateway-fronted multi-upstream case (one OpenRouter gateway fronting Anthropic + OpenAI simultaneously). Fixtures are nested as fixtures/aws-design-ai/<case>/ aws-design-ai.json (not flat) and registered in artifact-contracts.json's shared section from the start, per #415's own Cursor-review finding about lint:artifacts fixture visibility. Validation (run in the worktree .worktrees/aws-design-ai-contract): - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_aws_design_ai_contracts.py -> 16 passed - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_migration_matrix_contracts.py -> 57 passed (unchanged, regression-clean) - uv run --with pytest python -m pytest plugins/aws-startup-advisor/tests/test_ai_workload_profile_contracts.py -> 24 passed (unchanged, regression-clean) - mise run lint:artifacts -> PASS, 80 artifact(s) checked (up from 74 before this change), 0 errors, 50 baselined (pre-existing), 14 info (pre-existing NO_CONTRACT entries, unrelated to this change) - mise run lint:vendored-parity -> PASS, 46 vendored file(s) byte-identical to skills/shared/ and matching README tables - mise run lint:skill-refs -> PASS, 1181 references checked, 0 missing, 7 baselined (pre-existing), 0 stale - mise run lint:md -> 0 issues in 2478 files - mise run validate -> all sub-tasks (lint:manifests, lint:spec, lint:skill-parity, lint:artifacts, lint:vendored-parity, lint:skill-refs) pass - mise run build -> all validate tasks pass; security (gitleaks) scanned 503 commits, ~35.6MB, no leaks found - git diff --check -> clean (exit 0) - git merge-tree --write-tree origin/main HEAD -> clean tree SHA, no conflict markers - git diff against PR #415 head (4958a2c446733d83b35353acb4347d14ab6855c2) touches only: plugins/aws-startup-advisor/README.md, fixtures/README.md, scripts/artifact-contracts.json, scripts/contracts/aws-design-ai.schema.json, skills/azure-to-aws/references/shared/schema-design-aws-ai.md, skills/{azure,gcp}-to-aws/references/vendored/README.md, skills/{azure,gcp}-to-aws/references/vendored/ai/schema-design-aws-ai.md, skills/shared/ai/README.md, skills/shared/ai/schema-design-aws-ai.md, tests/test_aws_design_ai_contracts.py, and the two new fixtures/aws-design-ai/<case>/aws-design-ai.json files — contracts, fixtures, tests, docs, and vendored files only. No design-ai.md, estimate-ai.md, generate-artifacts-ai.md, or other phase/runtime file touched. * fix(aws-startup-advisor): address PR #416 Cursor findings on aws-design-ai contract Finding 1 (blocking): restore Azure's aws-design-ai.json emission contract. The manifest had dropped azure-to-aws's shape/root_heading entry outright instead of repointing it, so Azure's current scalar metadata.ai_source output silently failed the shared target-only schema. Restore skills/azure-to-aws/references/shared/schema-design-aws-ai.md to its PR #415 content and restore the azure-to-aws artifact-contracts.json entry so it shadows the shared schema until design-ai.md is repointed. Rewrite the canonical-status section in skills/shared/ai/schema-design-aws-ai.md to name the restored Azure file as the current emission contract and this file as the target. Vendored copies re-synced for lint:vendored-parity. Finding 2 (should-fix): add a regression test proving an observed source with empty model_observations is rejected by the reused #415 _assert_profile_semantics helper, locking the completeness gap the schema itself cannot express without $data. Finding 3 (should-fix): add _assert_model_traceability and tests proving every design_blocks[].model_id traces to a metadata.ai_sources[] entry or a gateway upstream, with a negative case for an untraceable model id. Verified: pytest tests/test_aws_design_ai_contracts.py (22 passed), mise run lint:artifacts (0 errors), mise run lint:vendored-parity, mise run build all green. * fix(aws-startup-advisor): close GCP shadow gap, model_observations gap, gateway traceability test gap - Add gcp-to-aws manifest shadow entry for aws-design-ai.json mirroring azure-to-aws's pattern, with a new schema-design-aws-ai.md documenting GCP's current scalar emission; add the matching accept/reject test pair. - Require model_observations minItems:1 on the aiSource observed branch in aws-design-ai.schema.json, closing the schema-level gap the semantic oracle alone used to cover; update the existing test to assert the schema now rejects the mutation directly. - Add two tests exercising the gateway-upstream-walk branch of _assert_model_traceability (previously unexercised by any positive fixture). * fix(aws-startup-advisor): require model_observations on observed ai-workload-profile sources * fix(aws-startup-advisor): sync vendored schema-design-aws-ai.md copies The Azure and GCP vendored copies of schema-design-aws-ai.md had drifted from the canonical skills/shared/ai/schema-design-aws-ai.md — missing the PR's own round-2 fixes (the model_observations minItems note and the GCP manifest-shadow paragraph), which caused lint:vendored-parity to fail in CI. Re-ran tools/sync-vendored.py to bring both copies back to byte-identical with the canonical file. Verified: mise run build passes clean (lint:md, lint:manifests, lint:skill-parity, lint:spec, lint:skill-refs, lint:vendored-parity, lint:artifacts, gitleaks); pytest on test_aws_design_ai_contracts.py and test_ai_workload_profile_contracts.py — 50 passed. * fix(aws-startup-advisor): add missing canonical source for telemetry vendoring PR #407 (merged to main) vendored identical telemetry/ files into azure-to-aws, gcp-to-aws, heroku-to-aws, and llm-to-bedrock but the lint:vendored-parity gate did not yet exist on main at that time, so no skills/shared/telemetry/ canonical source or README rows were ever added. Merging upstream/main into this branch (which carries the gate from PR #386) now surfaces that pre-existing gap. Add skills/shared/telemetry/ as the canonical source (identical across all four vendored copies, verified via md5sum), add README rows for the eleven telemetry files in the three skills with existing vendored READMEs, and create the vendored README for llm-to-bedrock (which vendors only the telemetry tree and had no README.md at all). Verified: python3 plugins/aws-startup-advisor/tools/sync-vendored.py --check passes (90 files), and mise run build passes clean. * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): keep telemetry under its bundle gate Co-authored-by: Cursor <cursoragent@cursor.com> * fix(aws-startup-advisor): match the shared telemetry client to the bundle The parity check treats skills/shared as canonical. That copy was missing the plugin User-Agent the vendored bundles already send, so the check failed. * fix(aws-startup-advisor): address PR #410 Cursor review findings Three independently reproduced findings: 1. clarify-ai-only.md's "preferences.json (AI-only)" example contradicted its own prose: ai_framework.value was a bare string while Q1 is documented as multi-select with a ["direct"] default, and the example omitted the mandated metadata.report_caveats[] and never used the source/question_id provenance alternative the prose referenced. Fixed the example to match an array value and added report_caveats; clarified the prose that question_id is this flow's sole provenance field. 2. _assert_scenario_semantics deduplicated gateway upstreams on their raw (provider, source_service-or-None) key BEFORE resolving a service-less upstream to its actual source, so an explicit-service upstream and an overlapping service-less upstream for the same single retained source had different raw keys and both passed. Now dedupes on the resolved source's own (provider, source_service) key. Added a parametrized regression test covering both insertion orderings. 3. strip_jsonc attributed a trailing // comment to the textually-last "key": match on a line, which broke for an inline array/object value that closes before the comment ("call_sites": [{ "file": ..., "line": 1 }], // REQUIRED resolved to "line", not "call_sites"), so omitting the entire required field passed silently. Now tracks bracket depth and binds the comment to the key whose own value's depth matches (or, for a value still open at end of line, is one level shallower than) the depth at the comment. Added unit tests on strip_jsonc directly plus an end-to-end MISSING_REQUIRED regression against the real contract. All three reproduced against the live PR head before fixing. 104 tests pass in the two affected files; full mise run build, lint:artifacts (74 checked, 0 errors), lint:vendored-parity, diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR #416 Cursor review findings Five independently reproduced findings: 1. The GCP shadow contract (schema-design-aws-ai.md) put phase/focus/timestamp at the top level, but design-ai.md's own "top-level fields" table nests all three (plus ai_source, bedrock_models_selected) inside metadata -- GCP does not split them the way Azure's parallel shape does. The example also omitted design_blocks[].honest_assessment, required ("not_applicable") on the three traditional-AI capability rows per design-ai.md's Validation Checklist. Fixed the example's nesting and added the conditional honest_assessment field; corrected test_aws_design_ai_contracts.py's _gcp_scalar_design() fixture, which built from the canonical target shape with only metadata swapped and so never exercised the actual producer envelope. 2. test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count assumed only ai-migration-matrix contributes unmapped "shared" fixtures, but ai-workload-profiles/ (#415) and aws-design-ai/ (#416) are unmapped too. Generalized the assertion to count every unmapped fixture family. 3. clarify-ai-only.md's AI-only example contradicted its own prose (ai_framework as a bare string vs. the documented ["direct"] multi-select default, missing metadata.report_caveats); clarify-ai.md's mixed-flow example omitted its own documented Q21 ai_latency row. Fixed both examples; clarified clarify-ai-only .md's prose that question_id is this flow's sole provenance field. 4. test_positive_fixtures_project_exactly_into_pr415_semantics only ran the weaker profile-level oracle (_assert_profile_semantics), not the stronger matrix-level gateway/source correspondence oracle (_assert_scenario_semantics) that the dedicated negative mutation test already proves is strict. A model dropped from only a gateway upstream in a copied positive fixture passed all existing positive-fixture checks. Now runs both oracles on every positive fixture. 5. artifact-contracts.json's extra_shapes entries point preferences.json's AI fields at clarify-ai.md's "Step 3: Rows returned" example, which showed bare per-fragment values ("ai_priority": "balanced") instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's own rules require and every sibling fragment (e.g. clarify-global.md) already documents in that wrapped form. A properly assembled mixed-Azure artifact failed TYPE_MISMATCH/UNKNOWN_KEY on every AI row. Rewrote the example to show the wrapped envelope, matching the established sibling-fragment convention. All five reproduced against the live PR head before fixing. 181 tests pass in the plugin's full suite; mise run build, lint:artifacts (80 checked, 0 errors), lint:vendored-parity (90 files), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR #411 Cursor review findings Four independently reproduced findings: 1. The pre-split Azure cache had its own "### Embeddings — Bedrock" target table (Titan Text Embeddings v1/v2, Cohere Embed v4/English v3/Multilingual v3) nested under GCP's "## Bedrock Models (On-Demand)" heading, but it is a different subsection than the one the parity script's byte-equality check covers, so the split dropped it entirely. estimate-ai.md's design flow still references this table and still recommends Titan v2 as the default embedding target. Moved the table into the canonical cache (scoped out of the main section's byte-equality check, matching the existing Claude Fable exception pattern) and the model_rows() quick-reference scanner (its Dimensions column coincidentally parses as a numeric output price for three rows), synced all three vendors, and added a dedicated row-level parity check. 2. clarify-ai-only.md's AI-only example contradicted its own prose (bare-string ai_framework vs. the documented ["direct"] array default, missing metadata.report_caveats) and route_shapes does not inherit the parent preferences.json entry's allowed_anywhere -- so (REQUIRED on every DETECTED-equivalent row per schema-preferences.md's own fundamental rule) was rejected on this route even though the general contract requires it. Fixed the example and added allowed_anywhere to the AI-only route_shapes entry; added source to this route's extracted rows and corrected the prose accordingly. Added three documented-shape acceptance cases to the existing producer-route test. 3. artifact-contracts.json's extra_shapes pointed preferences.json's mixed-flow AI fields at clarify-ai.md's raw per-fragment "Step 3: Rows returned" values instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's rules require and every sibling fragment (e.g. clarify-global.md) already documents in that wrapped form; the example also omitted the documented Q21 ai_latency row. Rewrote the example to match the established convention, with source on DETECTED rows per finding 2's rule. 4. strip_jsonc attributed a trailing comment to the textually-last "key": match on a line, which broke for the committed workloads shape "call_sites": [{ "file": ..., "line": 1 }], // REQUIRED (resolved to "line", not "call_sites"), so omitting the entire required field passed silently in both Azure preferences routes. Now tracks bracket depth and binds the comment to the key whose own depth matches the end-of-line depth. Added strip_jsonc unit tests plus end-to-end MISSING_REQUIRED regressions on the real contract. All four reproduced against the live PR head before fixing. 73 tests pass in the plugin's full suite plus 23 in llm-to-bedrock's pricing suite; mise run build, lint:artifacts (53 checked, 0 errors), lint:vendored-parity (47 files), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows Correction to an earlier fix on this PR. While fixing the identical finding on PR #411, confirmed via schema-preferences.md's own fundamental rule ("the justification key is source, and it is REQUIRED on DETECTED") that source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's chosen_by: "extracted" rows -- the AI-only route's naming for the same DETECTED concept. The earlier fix on this PR incorrectly concluded source was "not part of this AI-only shape" and only clarified the prose to say so, rather than fixing the real root cause: route_shapes builds a shape contract from only the route object, not merged with the parent preferences.json entry, so it never inherited that entry's allowed_anywhere vocabulary (source, reason, context, etc.). Added allowed_anywhere to the AI-only route_shapes manifest entry (mirroring the parent preferences.json entry's), added source to the example's chosen_by: "extracted" rows (ai_framework, ai_monthly_spend, ai_model_baseline, ai_vision), and corrected the prose to say source is required on those rows. Added three acceptance cases (multi-value framework, report_caveats present, source present) to the existing producer-route test, replacing the test case that had asserted source must still fail. 136 tests pass in the plugin's full suite; mise run build, lint:artifacts (74 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): require source on AI-only DETECTED-equivalent rows Correction to an earlier fix on this PR. While fixing the identical finding on PR #411, confirmed via schema-preferences.md's own fundamental rule ("the justification key is source, and it is REQUIRED on DETECTED") that source should be ACCEPTED (and is in fact required) on clarify-ai-only.md's chosen_by: "extracted" rows -- the AI-only route's naming for the same DETECTED concept. The earlier fix on this PR incorrectly concluded source was "not part of this AI-only shape" and only clarified the prose to say so, rather than fixing the real root cause: route_shapes builds a shape contract from only the route object, not merged with the parent preferences.json entry, so it never inherited that entry's allowed_anywhere vocabulary (source, reason, context, etc.). Added allowed_anywhere to the AI-only route_shapes manifest entry (mirroring the parent preferences.json entry's), added source to the example's chosen_by: "extracted" rows, and corrected the prose. Added three acceptance cases (multi-value framework, report_caveats present, source present) to the existing producer-route test. 181 tests pass in the plugin's full suite; mise run build, lint:artifacts (80 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * fix(aws-startup-advisor): address PR #410 follow-up Cursor review findings Three independently reproduced findings: 1. Regression in the strip_jsonc depth-tracking fix: the unconditional 'if ch not in "\"\": key_start = None' reset the pending key on ANY non-quote character, including whitespace between a closed key string and its ':'. 'priority' : "balanced", // REQUIRED (space or tab before the colon) lost the key entirely, so omitting workloads[0].priority stopped producing MISSING_REQUIRED. Fixed by exempting whitespace from the reset; only a genuinely different character invalidates the pending key. Added parametrized unit tests for space/tab/double-space before the colon plus a control proving a real intervening token still invalidates. 2. artifact-contracts.json's extra_shapes still pointed preferences.json's mixed-flow AI fields at clarify-ai.md's raw per-fragment 'Step 3: Rows returned' values instead of the final assembled {disposition, value, default} envelope clarify-assemble.md's rules require (the same finding already fixed on #411/#416, not yet addressed here); the example also omitted the documented Q21 ai_latency row. Rewrote the example to match the established sibling-fragment convention, with source on DETECTED rows per the #411 correction. 3. Both clarify-ai.md and clarify-ai-only.md persisted workloads[].capability: "chat", but schema-discover-ai.md's capability enum is text_generation/structured_output/image_generation/embedding/… and design-ai.md has no routing case for "chat". Changed both examples to text_generation and added a test asserting both producer examples use a value from the discovery/design vocabulary. All three reproduced against the live PR head before fixing. 141 tests pass in the plugin's full suite; mise run build, lint:artifacts (74 checked, 0 errors), diff --check, and merge-tree against upstream/main are all clean. * refactor(aws-startup-advisor): move vendored-parity check into plugin dispatcher * fix(aws-startup-advisor): drop stale telemetry rows from vendored READMEs The merged-in sync-vendored.py now excludes telemetry/ from its managed file set (that subtree has its own parity gate, generated from scripts/telemetry/ via sync_bundles.py, not copied from skills/shared/). This branch's vendored READMEs still carried telemetry rows from before that exclusion landed, so every telemetry row failed as 'listed but missing on disk' once rglob stopped counting those files as managed. Removed the telemetry rows from azure-to-aws, gcp-to-aws, and heroku-to-aws's vendored READMEs (each still vendors other, real skills/shared/ files). Deleted llm-to-bedrock's vendored README entirely, since telemetry was the only thing it ever listed and upstream/main carries no README for that skill's vendored dir either. Verified: mise -C plugins/aws-startup-advisor run ci passes (vendored- parity: 46 files OK; artifacts, model-id, skill-refs, and 272 tests all green). mise run build from repo root clean. * fix(aws-startup-advisor): account for every neutral fixture directory, not just ai-migration-matrix test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count hardcoded the assumption that ai-migration-matrix was the only fixture directory with no skill prefix (gcp-/heroku-/azure-), so --fixtures' checked_artifacts total only needed one 'shared artifacts' bucket subtracted out. This PR's own fixtures/ai-workload-profiles/ directory is a second such neutral directory (its 4 ai-workload-profile.json files are themselves registered as a 'shared' contract in artifact-contracts.json), so --fixtures counts them in checked_artifacts, but the test's formula never added them to the subtracted total -- failing with 'assert (78 - 53) == 21' instead of the correct 25. Generalized the test to sum shared-artifact counts and validate_run_dir checks across every fixture directory that skill_for_fixture maps to None, not just the matrix by name. This also guards against the same accounting gap recurring if a third neutral fixture directory is added later. Verification: the fixed test passes; mise -C plugins/aws-startup-advisor run ci passes in full (vendored-parity 44 files, artifacts 78 checked/0 errors, model-id, skill-refs, 246 tests); mise run build from repo root clean; git diff --check clean. * test(aws-startup-advisor): pin Estimate and Generate AI contracts Target-only, additive JSON Schemas for estimation-ai.json and generation-ai.json, stacked on merged #415 (ai-workload-profile) and #416 (aws-design-ai). cost_by_source[] / artifacts_by_source[] carry the ai_sources[]/gateways[] traceability chain forward per-source, per-model; model_observations[] intentionally not carried forward (identity already resolved by Design). No runtime wiring, no producer change — estimate-ai.md / generate-ai.md / generate-artifacts-ai.md are untouched in both skills. Added: - scripts/contracts/estimation-ai.schema.json, generation-ai.schema.json - skills/shared/ai/schema-estimate-ai.md, schema-generate-ai.md (vendored byte-identically into gcp-to-aws and azure-to-aws only) - fixtures/estimation-ai/{multi-provider-direct,openrouter-gateway-multi-upstream}/estimation-ai.json - fixtures/generation-ai/{multi-provider-direct,openrouter-gateway-multi-upstream}/generation-ai.json - tests/test_estimation_ai_contracts.py, test_generation_ai_contracts.py (positive validation, malformed/duplicate/overlapping rejection, untraceable/dropped-model negatives, cross-phase projection into aws-design-ai.json and estimation-ai.json, vendored byte-identity) Registered two 'shared' entries in artifact-contracts.json, two fixture bullets in fixtures/README.md, two rows in skills/shared/ai/README.md, and a scripts/ clause in README.md. Verification run from plugins/aws-startup-advisor (all commands executed for real, output below): uv run --with pytest python -m pytest tests/ -q -> 429 passed (from the worktree root) mise run build -> lint:manifests, lint:spec, lint:skill-parity, lint:md, security all PASS mise run ci -> lint:artifacts: PASS — 84 artifact(s) checked, 0 error(s) lint:vendored-parity: PASS — 53 vendored file(s) byte-identical lint:skill-refs: PASS — 1339 reference(s) checked, 0 missing lint:md: 0 error(s) lint:model-id: OK (18 retired IDs enforced) test: 429 passed test:heroku-source-review: 44 passed * fix: address AI artifact contract review Keep target-only estimate and generate schemas out of named skill run validation while retaining neutral fixture and self-check coverage. Preserve unavailable estimate costs and dependent deltas as null, with synced documentation and regression coverage. Verification: - affected pytest modules: 87 passed - startup-advisor artifact validation: 84 checked, 0 errors - vendored parity: 53 files byte-identical - startup-advisor CI: 435 Python tests passed; 57 Node tests passed; all lint and validation tasks passed --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Gen Li <i.ligen@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The skills are prose state machines. A phase file says "Load
references/phases/design/design-ai.md" or "runpython3 \"<SKILL_BASE>/scripts/x.py\"" and the agent follows the string. Nothing checked that the target exists. A renamed or never-written file is a dead end the user hits mid-migration — and any file move during consolidation (the AI-engine extraction proposed in the companion issue) would break references silently.This is the third gate in the family:
validate-artifacts.py(#385, artifact contracts),sync-vendored.py(#386, vendored parity), and nowcheck-skill-refs.py(reference paths). Together they make "move a phase file to another skill" provable rather than hopeful.What it found on the real repo
1118 references checked; 7 dead, each a real defect, baselined with the fix named:
discover-iac.md:52-53references/shared/extract-bicep.md,extract-arm.md:55says HALT if a dialect's ref is not on disk → every Bicep/ARM workspace halts against a path that will never existdesign.md:67,schema-design-aws.md:90design-refs/patterns.mdpattern_status: catalog_absent); retire when the catalog landsSKILL.md:3,149references/offers.md../knowledge-base-for-startups/references/offers.md(line 29 has it right)design-ai.md:424shared/ai-model-lifecycle.mdreferences/vendored/ai/ai-model-lifecycle.mdChange
tools/check-skill-refs.py(stdlib). Scansskills/**/*.mdandagents/*.mdfor path tokens into the skill tree, with optional$PLUGIN_ROOT/${CLAUDE_PLUGIN_ROOT}/$SKILL_BASE/<SKILL_BASE>/$GCP_BASE/.//../prefixes.Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form>(agent-advisor'sphases/,shared/,design-refs/) → plugin/repo-relative. Library-origin files (skills/shared/*,*/references/vendored/*,*/references/shared/*) are written from the consuming skill's point of view and resolve againstskills/sharedand every skill.agent-advisorandllm-to-bedrockmay namegcp-to-aws's tree (the documented inline-execution dependency).agents/resolve againstllm-to-bedrock.Skips: run-artifact paths (
$MIGRATION_DIR/…,.migration/…,<PLAN_DIR>/…), templated tokens (<slug>,NNN,*), URLs, and.//../tokens with no skill-tree root in them (generated outputs such as./deploy.sh).tools/skill-refs-baseline.json:ignorerules (permanent policy — the scripts Generate writes into$MIGRATION_DIR/scripts/, one example path in the user's app) andentries(the 7 dead references above, each with its fix; STALE when they stop matching, failing the gate so the list only shrinks).mise run lint:skill-refs, added tovalidate.Tested
tools/test_check_skill_refs.py— 8 tests on a throwaway plugin tree: each resolution rule (skill-relative, every variable prefix,../sibling-skill/…), each skip rule, library-origin resolution incl. flattenedshared/x.md→skills/shared/ai/x.md, gcp-dependent andagents/fallbacks, ignore vs entries vs stale, CLI exit codes; plus the real repository passes with no stale entries.mise run lint:skill-refs→PASS — 1118 reference(s) checked, 0 missing, 7 baselined, 0 stale baseline entries.lint:artifacts,lint:vendored-parity,tools/validate.py, full-repo markdownlint: pass.Not in this PR
The 7 dead references are baselined, not fixed; the fixes touch skill prose (two of them are design decisions — Bicep support, pattern catalog) and belong in their own small PRs. Section-anchor references ("see § Step 3b", llm-to-bedrock's dangling "B4a") are prose, not paths, and are out of scope here.