Skip to content

feat(aws-startup-advisor): measure Anthropic usage and show measured spend and volume in Q3 and Q7 in bedrock-migration-assessment - #605

Merged
herosjourney merged 3 commits into
mainfrom
feat/389-bma-anthropic-usage
Nov 11, 2026
Merged

herosjourney merged 3 commits into
mainfrom
feat/389-bma-anthropic-usage

Conversation

@vagmur-aws

@vagmur-aws vagmur-aws commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

bedrock-migration-assessment could measure only OpenAI usage (#592). An app that calls Anthropic directly got no measured spend or volume, so Clarify's Q3 (spend) and Q7 (volume) relied on the user's guess. This is stage 3a.1b-5a of #389.

Fix

Anthropic usage reader (discover-anthropic-api.md, scripts/anthropic-usage.py). Like openai-usage.py, a shipped stdlib script is the only code that reads the Admin key file. Checked against Anthropic's current Usage and Cost API, usage report, cost report, and Admin key references.

  • Calls: GET only, to usage_report/messages and cost_report on https://api.anthropic.com, with x-api-key and anthropic-version. Never grouped by account, API key, or service account. A Console Admin key has no narrower usage-only scope, and the consent says so.
  • Workspaces: the probe prints each workspace's spend and writes nothing. The default workspace has workspace_id: null, which workspace_ids[] cannot select, and cost_report has no workspace filter. So the script filters rows by workspace_id and drops other workspaces' rows before writing anything. The consent says the replies can include those rows.
  • Key file and exits: the script deletes the key file after every capture, however it ends (including a network error), and after any probe that does not succeed. A 404 on the probe means Claude Enterprise or Claude Platform on AWS, and exits ORG_NOT_SUPPORTED. capture deletes earlier files first and records result. summarize reads only rows the manifest marks ok.
  • Cost: amounts are cents, converted to USD. cost_status uses the existing complete/partial_window/cost_unavailable values. cost_report covers only the standard and batch tiers (Priority Tier is billed separately), so usage in any other tier makes it partial_window, metadata.uncosted_service_tiers names those tiers, and capture_warnings says the total excludes their spend, so llm-to-bedrock's comparison is qualified.
  • Tokens: uncached input, cache reads, and 5-minute and 1-hour cache writes stay separate, under the field names llm-to-bedrock reads. The per-minute peak, the busiest day, and day_input_tokens count uncached input and cache writes, not cache reads. The API reports no request counts, so the profile leaves them out rather than writing 0.
  • Merge: adds anthropic/anthropic_api models with provenance: "usage_api", keeps raw_model verbatim, and refuses when Anthropic is a gateway upstream.

Discover, preview, and Clarify.

  • Discover Step 2c runs each reader whose source the profile has, OpenAI first and then Anthropic. Each has its own consent; declining one does not skip the other. Resume follows each reader's manifest.
  • The preview (preview_version 2) takes one --usage per profile. measured_usage lists each source with its own active period and adds total spend and tokens; total spend is null when any source's spend is unknown. The Discover display shows spend per provider and the total token volume.
  • Resuming at Clarify with a preview written by the version on main (preview_version 1) first re-runs Discover Step 2d from the files already in the run directory. It scans nothing and calls no API.
  • Q3 and Q7 show the measured totals from migration-preview.json, give each source its own period, and mark the matching option "(measured)". A source with a short window is labeled as not scaled to a month. The user still answers, recorded with chosen_by: "user", so Q7 stays never-skipped. Q3 now asks about "your current AI providers" instead of "OpenAI or Gemini", since it can show Anthropic spend.

Review fix that also applies to the OpenAI reader: openai-usage.py had the same gap, where a network error left the key file on disk. It gets the same fix.

Scope

  • Skill: bedrock-migration-assessment only. New: references/phases/discover/discover-anthropic-api.md, scripts/anthropic-usage.py, fixtures/bma-anthropic-usage/ (a capture with known totals, peaks, and costs), and fixtures/bma-preview-v1/ (a preview generated by main). Changed: discover.md, discover-openai-api.md, clarify-ai-only.md, SKILL.md, shared/schema-preferences.md, scripts/migration-preview.py, scripts/openai-usage.py, the plugin README, and the bma-ai-only-clarify fixture's Q3 prompt.
  • Tests: new test_bedrock_migration_assessment_anthropic_usage.py; test_bedrock_migration_assessment_usage_preview.py and test_consent_disclosure_audit.py (now 8 discovery sub-files) updated.
  • Size: docs 204, scripts 474, tests 406, fixture 423 lines.
  • No changes to gcp-to-aws, azure-to-aws, or any skills/shared/ file.

Type of Change

Select at least one.

  • New skill
  • Bug fix
  • Documentation update
  • CI/CD change
  • Other

Startup Advisor review checklist

By submitting this PR, I confirm that:

  • I have reviewed the full diff and understand all changes I am submitting.
  • I have tested my changes and confirmed they work as intended.
  • I have run mise -C plugins/aws-startup-advisor run ci and all checks pass.
  • I have disclosed any known issues, limitations, or untested scenarios.

Known limitations and what is not in this PR: the OpenRouter reader (3a.1b-5b); the usage-derived Mantle signals #585 adds to the gcp-to-aws and azure-to-aws readers, which belong in a follow-up once #585 merges; and the newer Usage and Cost Analytics API (the legacy endpoints used here remain supported, and the other skills' readers use them). The script sends anthropic-version: 2023-06-01, as the other Anthropic readers do; the header is optional, and the current reference examples use 2026-10-21. No real Anthropic API call was made: the script tests use a fake API.

Validation: mise -C plugins/aws-startup-advisor run ci passes (1,504 plugin tests and 225 subtests; 311 telemetry tests). Root mise run validate, python3 tools/validate.py, gitleaks, and the consent-disclosure audit pass. The new tests fail on main, and the review-fix tests fail on a3937bb (the head before these fixes). Script tests run the capture against a fake API: GET only, both headers, no forbidden grouping, pagination, a 429 retry, workspace_ids[] only for named workspaces, unselected rows dropped (including with the default workspace selected), a revoked key, a network error during probe and capture, a 403 that fails one row, a failed usage row that ends the capture, probe 401/403/404, cost failed versus empty, an uncosted tier on a full window, a full-window plus partial-window pair in the preview, a cache-read-heavy day, and no key in any output, file, or URL. Pre-review probes (schema boundaries, google-genai and framework matrices) report 0 problems. Live headless runs with Claude Code and this branch (no real API calls): declining both consents on an OpenAI-plus-Anthropic repo 3/3; a resumed run where Q3 and Q7 are asked with the measured values and the user's answers are recorded 3/3; a resumed run with a version-1 preview, which is refreshed without a scan or API call before Q7 shows the measured volume 3/3; existing scenarios (OpenAI usage decline, fast path, Specialized fast path, resume after Discover, mixed confidence, foreign run directory, two routing scenarios) 3/3 each. No run opened .env.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Refs #389

…spend and volume in Q3 and Q7 in bedrock-migration-assessment

Stage 3a.1b-5a of #389.

Anthropic usage reader (discover-anthropic-api.md, scripts/anthropic-usage.py),
a stdlib script like openai-usage.py so the agent never opens the Admin key
file. Checked against Anthropic's current Usage and Cost API reference:
- GET only, to usage_report/messages and cost_report on api.anthropic.com,
  with x-api-key and anthropic-version. Never grouped by account, API key,
  or service account. The consent says the Console Admin key has no
  narrower usage-only scope.
- The probe prints each workspace's spend and writes nothing. The default
  workspace has a null workspace_id, which workspace_ids[] cannot select,
  so the script filters rows by workspace_id; cost_report has no workspace
  filter at all. Rows of unselected workspaces are dropped before writing,
  and the consent says so.
- Capture deletes earlier files first, records result, and deletes the key
  file on every exit; a 404 probe (Claude Enterprise or Claude Platform on
  AWS) exits ORG_NOT_SUPPORTED and deletes it too. summarize reads only
  rows the manifest marks ok.
- Cost amounts are cents. cost_status follows the existing convention;
  cost_report reports only the standard and batch tiers (Priority Tier is
  billed separately), so other tiers make it partial_window and are named
  in uncosted_service_tiers. The API reports no request counts, so the
  profile omits them rather than writing 0. Peaks count uncached input and
  cache writes, not cache reads.
- merge adds anthropic/anthropic_api models with provenance usage_api,
  keeps raw_model verbatim, and refuses when Anthropic is a gateway
  upstream.

Discover Step 2c runs each reader whose source the profile has, OpenAI
then Anthropic, each with its own consent. The preview takes one --usage
per profile and reports each source plus totals (preview_version 2); the
total spend is null when any source's spend is unknown.

Clarify: Q3 and Q7 show the measured spend and volume from
migration-preview.json and mark the matching option, but the user still
answers (chosen_by user), so Q7 stays never-skipped. Q3 now asks about
"your current AI providers" instead of "OpenAI or Gemini".

The usage-derived Mantle signals in #585 are not added here; they belong
after #585 merges.

@herosjourney herosjourney left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: head b9209bbe (base 938d20db), 17 files (+1361/−30) — bedrock-migration-assessment, single plugin tree. Draft.
Ran: git merge-tree --write-tree upstream/main (clean; upstream is 28000d61, so the branch is behind and not conflicting); JSON parse of the five new capture fixtures (all valid); 42 passed in test_bedrock_migration_assessment_anthropic_usage.py and test_bedrock_migration_assessment_usage_preview.py; two mutation probes below. Primary sources read 2026-11-10: Usage and Cost API, Get Messages Usage Report, Get Cost Report. GitHub build, startup-advisor, CodeQL, and Python analysis passed on this head.
Did not run: root mise run build; the full plugin ci (1,478 tests); no live Anthropic call.

Cross-plugin parity: Not applicable. One plugin tree. No vendored skills/shared file changed.
Outside skills/ mirrored by hand: tests, fixtures, and README.md exist only in this tree.

Change narrative: After Discover, measure the selected Anthropic workspaces and show that spend and volume in Q3 and Q7, with the user still answering. Mechanism: a stdlib GET-only reader beside the OpenAI reader, plus preview totals. Surfaces: discover-anthropic-api.md, anthropic-usage.py, Discover step 2c, preview, Clarify Q3/Q7. Body matches the diff. The OpenRouter reader, the newer Analytics API, and usage-derived Mantle signals are named as out of scope. The legacy endpoints are still documented as supported.

Axis Stance One line
Parsimony clear Same shape as openai-usage.py: one script is the only reader of the key file.
Completeness finding day_input_tokens counts cache reads; the per-minute peak does not.
Security finding A network error during capture leaves the Admin key file on disk.
Alternatives clear The newer Analytics API is already named, and the docs still support these endpoints.
Contradictions finding The discovery doc and day_input_tokens disagree.
Overlap clear Cost-status values match the OpenAI reader. No second Anthropic reader.
Cross-plugin parity clear Single tree.
Propagation & data clear Cents, tier set, limits, and null default workspace match the API reference. cost_status stays in complete / partial_window / cost_unavailable.

Verified against the cost reference: amount is cents ("123.45" means $1.23); group_by is only description and workspace_id; there is no workspace_ids parameter; service_tier on a cost row is batch or standard. The usage reference: default workspace is workspace_id: null; limits are 31 / 168 / 1440 for 1d / 1h / 1m; grouping by account_id, api_key_id, and service_account_id exists and the script rejects those groups (ValueError on api_key_id, probed). A constructed bucket with 10 uncached, 1000 cache-read, and 5 cache-write tokens produced a per-bucket peak of [15, 3] and a day total of [1015, 3].

Blocking

None.

Should-fix

  1. [Contradictions] The busiest-day input counts cache reads. — scripts/anthropic-usage.py:254 and discover-anthropic-api.md:136

    d[0] += uncached + read + c5 + c1
    The peak's input counts uncached input and cache writes, not cache reads.
    The per-bucket peak used for the per-minute rate excludes read. The day total used for day_input_tokens and for choosing the busiest day includes it. Probe: 1000 cache-read tokens became part of the day total and not the per-minute peak.
    Consequence: a cache-heavy day is reported as the peak input, and Q7's volume story can cite that day.
    Fix: d[0] += uncached + c5 + c1. Regression: extend test_peak_input_counts_cache_writes_but_not_cache_reads to assert day_input_tokens equals uncached plus both cache writes.

  2. [Security] Capture does not delete the Admin key when the request raises URLError. — anthropic-usage.py:222 and discover-anthropic-api.md:13

    delete_key(args.key_file) sits after the request loop, and the except clauses cover KeyRejected, RuntimeError, Forbidden, and OrgNotSupported.
    The script deletes it after every capture, successful or not
    Probe: opener raising URLError left the key file in place. The key was not printed. The phase also tells the agent to rm -f the file when a step ends, so a run that continues still removes it. A crashed run does not.
    Consequence: the Admin key file remains in $MIGRATION_DIR after a timeout or DNS failure.
    Fix: wrap cmd_capture in try / finally: delete_key(args.key_file). On cmd_probe, delete the key in an except for URLError and then re-raise; a successful probe must still leave the file for capture. Regression: a fake URLError asserts the key file is gone and nothing in stdout contains sk-ant-admin01-.

Heads-up

The reference examples send anthropic-version: 2026-10-21. This script and test_capture_is_get_only_scoped_and_never_leaks_the_key pin 2023-06-01. The header is documented as optional. I could not show that 2023-06-01 drops inference_geo, speed, or the cache split.

The minute-window day picker (_busiest_on_demand_day) still sums cache reads. The discovery doc does not define that picker as peak input.

Alternatives

None worth raising. The body already keeps the legacy endpoints because they remain supported and the OpenAI reader uses the same family.

Verdict

Comment — consent, the GET allowlist, workspace filtering, and the cents conversion match the API reference. The busiest-day total counts cache reads, and a network error leaves the Admin key file. Code owners on plugins/aws-startup-advisor decide.

@vagmur-aws
vagmur-aws marked this pull request as ready for review November 11, 2026 00:30
@vagmur-aws
vagmur-aws requested review from a team as code owners November 11, 2026 00:30
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Problem, Fix, and Scope are filled in. A change type is selected and all four Startup Advisor author confirmations are complete.

The author checklist check now passes. CI and reviewers still need to verify the change.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Reviewed a3937bb5be674f36b5d976079d1bf172d0d49fe4 with OCR delegation, a repository review, and one independent general review.

Please address the three additional issues inline:

  1. Preserve each provider's activity period when displaying combined spend. A 30-day source plus a 2-day source must not be described as a combined 2-day measurement.
  2. Carry excluded-tier cost warnings through the existing usage-baseline/report contract. A full-window Priority Tier capture must still disclose the missing dollars in the downstream comparison.
  3. Handle saved version-1 previews on resume. They have neither sources nor total_tokens; Q3/Q7 need a compatible fallback or a refreshed preview before using those fields.

I also reproduced the existing key-cleanup concern and daily-peak input concern at this head. Those remain in their original threads: verify cleanup after transport failures and assert both bucket and daily peak token pools when fixing them.

Validation: 82 focused tests passed on the original source, git diff --check passed, and offline probes exercised capture errors, aggregate windows, excluded-tier metadata, and an actual base-generated version-1 preview. No live API or deployed workflow was run. The current author-checklist check is also failing; its existing bot comment lists the required description updates.

…ache reads out of the busiest day, and keep measured usage honest across sources and old previews

Review findings on #605, each reproduced first.

herosjourney:
- A URLError during capture or probe skipped the key-file deletion, so a
  timeout or DNS failure left the Admin key on disk. capture now deletes
  it in a finally block, and probe deletes it on any failure before
  re-raising; a successful probe still keeps it for capture, and a
  capture call missing --workspace keeps it so the corrected call works.
  openai-usage.py had the same gap and gets the same fix. Both readers
  document the network-error exit.
- The busiest day and day_input_tokens counted cache reads while the
  per-minute peak did not. Day totals now count uncached input and cache
  writes, the minute-sample day uses the same basis, and
  day_to_average_ratio divides by that same total.

leon1418:
- measured_usage reported the shortest source's active_days for the
  combined spend, so $3,000 over 30 days plus $20 over 2 days read as
  "$3,020 over 2 days". The aggregate no longer has active_days or
  partial_window; Q3 and Q7 give each source its own period.
- A full-window capture with Priority Tier usage had cost_status
  partial_window but no capture_warnings entry, and llm-to-bedrock's
  baseline and report read capture_warnings, not uncosted_service_tiers.
  The profile now adds "cost_report does not include priority tier
  spend; this total excludes it", as the gcp-to-aws Anthropic reader
  does, so the downstream comparison is qualified.
- A migration-preview.json written by the version on main (preview_version
  1) has no sources or total_tokens, and resuming at Clarify does not
  rewrite it. Step 0 now re-runs Discover Step 2d first when the preview
  is not version 2; that reads only the files already in the run
  directory, scans nothing, and calls no API. fixtures/bma-preview-v1 is
  a preview generated by main.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Re-reviewed 1255fd3d0a843e00297f2605adcfc9d56a1c8d9d with OCR delegation, a repository review, and one fresh independent general reviewer. All five previous concerns are verified fixed, and this review found no additional substantiated issues.

The fixes cover both readers' transport-error key cleanup, consistent daily/minute peak token pools, per-provider activity periods, excluded-tier cost warnings, and refreshing version-1 previews before Clarify. Validation: 91 focused tests passed on the original source, git diff --check passed, and offline probes checked failure/retry behavior and actual preview regeneration with the run ID and saved answers preserved. Consumer/report behavior was checked in source; no live API or deployed LLM workflow was run.

I replied with evidence and resolved only the three conversations I originally opened. The other reviewer's two conversations remain open for that reviewer. Approval is recommended for the user to submit; this is a COMMENT review, and the earlier CHANGES_REQUESTED review has not been dismissed.

@herosjourney
herosjourney dismissed leon1418’s stale review November 11, 2026 16:14

Both findings were fixed in 1255fd3 (cache reads excluded from the busiest-day input; Admin key deleted on any probe/capture failure in both usage readers), verified by re-running the 69 usage tests on the head; the two threads are resolved. Dismissing the request from the pre-fix head a3937bb.

@herosjourney herosjourney left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 1255fd3: busiest-day input now counts uncached plus both cache writes (same basis as the per-minute peak), and the Admin key file is removed on any failure in probe and capture for both the Anthropic and OpenAI usage readers, with URLError regression tests. 69 usage tests pass on the head. Approving.

@herosjourney
herosjourney added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 11, 2026
@herosjourney
herosjourney added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Nov 11, 2026
@vagmur-aws
vagmur-aws added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Nov 11, 2026
@herosjourney
herosjourney added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit ac89c65 Nov 11, 2026
22 of 24 checks passed
@herosjourney
herosjourney deleted the feat/389-bma-anthropic-usage branch November 11, 2026 20:04
herosjourney pushed a commit to herosjourney/agent-toolkit-for-aws that referenced this pull request Nov 12, 2026
… bedrock-migration-assessment (aws#619)

* feat(aws-startup-advisor): measure Anthropic usage and show measured spend and volume in Q3 and Q7 in bedrock-migration-assessment

Stage 3a.1b-5a of aws#389.

Anthropic usage reader (discover-anthropic-api.md, scripts/anthropic-usage.py),
a stdlib script like openai-usage.py so the agent never opens the Admin key
file. Checked against Anthropic's current Usage and Cost API reference:
- GET only, to usage_report/messages and cost_report on api.anthropic.com,
  with x-api-key and anthropic-version. Never grouped by account, API key,
  or service account. The consent says the Console Admin key has no
  narrower usage-only scope.
- The probe prints each workspace's spend and writes nothing. The default
  workspace has a null workspace_id, which workspace_ids[] cannot select,
  so the script filters rows by workspace_id; cost_report has no workspace
  filter at all. Rows of unselected workspaces are dropped before writing,
  and the consent says so.
- Capture deletes earlier files first, records result, and deletes the key
  file on every exit; a 404 probe (Claude Enterprise or Claude Platform on
  AWS) exits ORG_NOT_SUPPORTED and deletes it too. summarize reads only
  rows the manifest marks ok.
- Cost amounts are cents. cost_status follows the existing convention;
  cost_report reports only the standard and batch tiers (Priority Tier is
  billed separately), so other tiers make it partial_window and are named
  in uncosted_service_tiers. The API reports no request counts, so the
  profile omits them rather than writing 0. Peaks count uncached input and
  cache writes, not cache reads.
- merge adds anthropic/anthropic_api models with provenance usage_api,
  keeps raw_model verbatim, and refuses when Anthropic is a gateway
  upstream.

Discover Step 2c runs each reader whose source the profile has, OpenAI
then Anthropic, each with its own consent. The preview takes one --usage
per profile and reports each source plus totals (preview_version 2); the
total spend is null when any source's spend is unknown.

Clarify: Q3 and Q7 show the measured spend and volume from
migration-preview.json and mark the matching option, but the user still
answers (chosen_by user), so Q7 stays never-skipped. Q3 now asks about
"your current AI providers" instead of "OpenAI or Gemini".

The usage-derived Mantle signals in aws#585 are not added here; they belong
after aws#585 merges.

* fix(aws-startup-advisor): delete the Admin key on any failure, keep cache reads out of the busiest day, and keep measured usage honest across sources and old previews

Review findings on aws#605, each reproduced first.

herosjourney:
- A URLError during capture or probe skipped the key-file deletion, so a
  timeout or DNS failure left the Admin key on disk. capture now deletes
  it in a finally block, and probe deletes it on any failure before
  re-raising; a successful probe still keeps it for capture, and a
  capture call missing --workspace keeps it so the corrected call works.
  openai-usage.py had the same gap and gets the same fix. Both readers
  document the network-error exit.
- The busiest day and day_input_tokens counted cache reads while the
  per-minute peak did not. Day totals now count uncached input and cache
  writes, the minute-sample day uses the same basis, and
  day_to_average_ratio divides by that same total.

leon1418:
- measured_usage reported the shortest source's active_days for the
  combined spend, so $3,000 over 30 days plus $20 over 2 days read as
  "$3,020 over 2 days". The aggregate no longer has active_days or
  partial_window; Q3 and Q7 give each source its own period.
- A full-window capture with Priority Tier usage had cost_status
  partial_window but no capture_warnings entry, and llm-to-bedrock's
  baseline and report read capture_warnings, not uncosted_service_tiers.
  The profile now adds "cost_report does not include priority tier
  spend; this total excludes it", as the gcp-to-aws Anthropic reader
  does, so the downstream comparison is qualified.
- A migration-preview.json written by the version on main (preview_version
  1) has no sources or total_tokens, and resuming at Clarify does not
  rewrite it. Step 0 now re-runs Discover Step 2d first when the preview
  is not version 2; that reads only the files already in the run
  directory, scans nothing, and calls no API. fixtures/bma-preview-v1 is
  a preview generated by main.

* feat(aws-startup-advisor): measure OpenRouter usage and BYOK spend in bedrock-migration-assessment

Stage 3a.1b-5b of aws#389. Stacked on aws#605.

OpenRouter usage reader (discover-openrouter-api.md,
scripts/openrouter-usage.py), a stdlib script like the OpenAI and
Anthropic readers so the agent never opens the management key file. It
keeps every field the gcp-to-aws OpenRouter reader produces, checked
against OpenRouter's current API reference:
- GET /credits (the probe), /workspaces, /activity, /key, and
  /analytics/meta, plus one read-only POST /analytics/query (three
  queries: BYOK classification, volume and fees, hourly peak). The script
  refuses any other endpoint, per-key or per-user parameter, grouping, or
  query body other than metrics by model, hourly at most, and a workspace
  filter. The analytics queries ask only for metrics /analytics/meta
  lists, so a renamed metric makes that field unknown instead of failing.
  The consent names the POST and says a management key has no read-only
  scope.
- One workspace or the whole account, filtered (never grouped) on both
  /activity and the analytics queries.
- Capture deletes earlier files, records result, and deletes the key on
  every exit, including a network error; a revoked key on /credits or
  /activity is key_invalid_mid_run and a failed /activity is
  all_usage_failed. A failed analytics row does not stop the capture.
- summarize joins analytics on model_permaslug and rolls up to the model
  id: byok_passthrough, mixed, no_byok, uncertain, or unknown, with the
  gcp-to-aws thresholds and profile-level rule. It adds byok_fees_usd,
  request counts, p95 latency and throughput, web-search cost, guardrail
  counts, and an hourly peak (day peak when the hourly file is missing or
  truncated), and warns when /activity is empty but lifetime usage is not.
- New: /activity now reports byok_usage_inference, so the profile records
  byok_inference_cost_usd (billed by the user's own provider keys, not in
  monthly_cost_usd) and warns when it is above zero.
- merge applies the scan's OpenRouter prefix rules (openai/, anthropic/,
  google/ by serving provider) and adds models to the source and the
  gateway upstream only when the gateway's upstreams are observed;
  otherwise the profile is unchanged and the models are named.

Discover Step 2c runs the reader when the profile has an OpenRouter
gateway. The preview adds OpenRouter tokens, adds BYOK inference to the
total, and leaves out OpenRouter rows a direct profile also counts unless
they are no_byok (the llm-to-bedrock name rule), listing them in
overlap_excluded; Q3 and Q7 say so.

* fix(aws-startup-advisor): join OpenRouter analytics by permaslug, keep unmeasured values unknown, keep router fees, and merge across every gateway

Review findings on aws#619 (leon1418), each reproduced first.

- Analytics replies label the model dimension for display ("OpenAI:
  GPT-4.1"), and only filters take the permaslug, so joining a grouped
  reply to /activity's model_permaslug matched nothing. Each analytics
  row now sends one query per permaslug in /activity, filtered by it and
  never grouped, and stores the replies keyed by that permaslug (at most
  50 models; the manifest names the rest). A truncated reply is left out
  and named.
- An unoffered or unreported metric became 0, so a model with BYOK spend
  could be certified no_byok and its provider bill counted twice. A
  permaslug is classified only when both byok_usage and openrouter_usage
  were offered and reported; volume fields that were not measured are
  null with a warning, never 0.
- Leaving out an overlapping OpenRouter row also dropped OpenRouter's own
  charge with no warning. The preview keeps a measured router fee on a
  byok_passthrough row, reports every dropped charge in overlap_excluded
  and spend_excludes_usd, says when a mixed row drops independent usage,
  and reports only the BYOK inference it actually includes
  (byok_inference_included_usd). Q3 reads those fields.
- The merge refreshed only the OpenRouter upstream after changing a
  shared source, so a valid OpenRouter plus LiteLLM profile failed
  validation. Every gateway upstream that references a changed source is
  rebuilt before validation.
- The merge only lowercased Gemini names. It now applies the scanner's
  Step 4 Gemini rule (drop version suffixes, keep -lite, -thinking,
  -image, -live, -tts), records the OpenRouter id in the evidence, and
  keeps it as raw_model for a new model.

* fix(aws-startup-advisor): merge the OpenRouter models that can be placed, read the date__hour field, and describe left-out rows accurately

Review findings on aws#619 (herosjourney), each reproduced first.

- One model with an unplaceable prefix (meta-llama/, mistralai/, ...)
  blocked the whole merge. The merge now adds every model it can place and
  lists the rest under not_merged; it exits 5 only when none can be placed.
- The analytics reference's example time field is date__day, so the hourly
  field is date__hour, which the hour lookup missed; two permaslugs in the
  same hour were then not added up. The lookup now reads hour or any
  date__ field.
- The Q3 and preview wording said a direct provider also counts every
  left-out row, which is false for a mixed row's OpenRouter-funded usage.
  Both now say the row is left out so its BYOK traffic is not counted
  twice, and a mixed row's note says its OpenRouter-funded usage is not in
  the total.
- An error body whose error is a string raised AttributeError and aborted
  the capture. All three readers now report it as the row's failure.
- A rejected key on the optional /key row is recorded skipped, as the
  reference says.
- SKILL.md's Discover output now names all three usage profiles.

The preview's router-fee and mixed-row handling (Should-fix 1) was already
fixed in 6802fd4; a test now runs the reviewer's probe (direct $100, a
byok_passthrough row with a $4 fee, a mixed row) and asserts $104 and the
mixed-row warning.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants