[kapurshr]

Problem

Startup Advisor feedback F2 (tracking issue #566, Track B): agent-advisor's runtime scoring has no volume dimension. A prototype handling a few dozen sessions a day scores the same as a service handling tens of thousands. Fixed-cost container runtimes can win small workloads where a scale-to-zero runtime costs nothing while idle.

Fix

• New scoring dimension expected_scale: expected sessions per day over the next few months.

  • prototype means under 100, growing 100 to 10,000, and high over 10,000.
  • unknown scores neutral (2) on every runtime.

• Per-runtime scores, based on each runtime's cost model as the runtime cards describe it:

  expected_scale	AgentCore	ECS	EKS	Lambda	Lambda MicroVMs
  prototype	4	1	0	4	3
  growing	3	2	1	3	3
  high	3	4	4	2	2
  unknown	2	2	2	2	2

  • Small scale: AgentCore and Lambda scale to zero, so they have no idle cost.
  • Fixed costs: ECS carries an always-on baseline (ecs.md), and EKS adds a cluster control plane.
  • High scale: at steady high volume, containers are the cost-efficient choice (ecs.md: "cost-optimized at steady scale"). AgentCore stays at 3 because it scales without operator work.

• Clarify:

  • Asks it in business and technical wording.
  • Adds it to the primary unit's question set, and "volume" to the delta question for other units.
  • Documents the legal values.
  • It is a per-unit dimension (seed unitDims), because volume belongs to a workload.

• No derived default: when the question is not asked or answered, expected_scale stays unknown. It is never filled from Intake's target_maturity. Every run carries a maturity, so deriving scale from it would change verdicts for people who never answered a scale question.

Scope

• Changed: agent-advisor only. The five references/runtimes/*.json, scripts/scoring.py, scripts/schemas/seed.json, the three Clarify docs, and scripts/test_scoring.py.

• Existing runs and seeds: they carry no expected_scale, so they score exactly as before. assumptions_used gains the line "expected_scale defaulted to unknown", the same line every unasked dimension already produces.

• Verdict changes when the question is answered. In a uniform sample of 100,000 answer sets (seeded, all other dimensions random over their legal values), each with expected_scale set versus unknown:

  expected_scale	Verdicts changed	Runtimes entering results	Runtimes leaving results
  prototype	25.5%	AgentCore 7.8%, MicroVMs 1.2%, Lambda 0.5%, ECS 0.3%	EKS 8.1%, ECS 7.5%, MicroVMs 4.9%
  growing	10.8%	AgentCore 2.3%, MicroVMs 1.0%, ECS 0.5%, Lambda 0.1%	EKS 5.1%, ECS 3.0%
  high	18.6%	ECS 4.5%, EKS 3.5%, AgentCore 3.2%	MicroVMs 6.3%, AgentCore 2.0%, Lambda 0.8%

  A uniform sample is not a usage distribution; it bounds how much of the answer space moves.

• Known overlap: traffic_pattern (shape) and expected_scale (volume) can point the same way. For example, an idle prototype adds to AgentCore and Lambda on both. The Clarify text separates them (volume, shape, and peak start rate are three questions), and the product owner accepted the stacking.

• Not covered:

  • The Temporal adapter (decision-refs/temporal.md) does not derive expected_scale, so Temporal agent-session units leave it unknown unless asked. Workflow volume does not map cleanly to sessions per day.
  • Pre-existing and unchanged here: seed unitDims lacks instance_type_requirement, so a seed that supplies it fails validation.
  • Related: fix(aws-startup-advisor): AgentCore single-agent scoring parity and when-to-revisit triggers in agent-advisor #622 adds AgentCore "revisit" triggers. Scale is deliberately not one of them, because AgentCore does not gain at high scale relative to containers.

Tests

• unknown is neutral on every runtime, and omitting the dimension scores identically to unknown.
• Score gains by scale:
  • At prototype: AgentCore and Lambda gain more than MicroVMs, which gains more than ECS, which gains more than EKS.
  • At high: ECS and EKS gain more than AgentCore, which gains more than Lambda and MicroVMs.
• prototype brings AgentCore into a MicroVMs-only result, and high brings ECS into one.
• The seed schema accepts expected_scale and rejects an illegal value.
• clarify.md documents exactly the engine's legal values, and both audience docs ask the question.
• Reverts: flattening the scale scores to neutral fails 4 tests, and removing the seed enum fails the schema test.

Type of Change

Select at least one.

• New skill
• Bug fix
• Documentation update
• CI/CD change
• Other

Startup Advisor review checklist

By submitting this PR, I confirm that:

• I have reviewed the full diff and understand all changes I am submitting.
• I have tested my changes and confirmed they work as intended.
• I have run mise -C plugins/aws-startup-advisor run ci and all checks pass.
• I have disclosed any known issues, limitations, or untested scenarios.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[kapurshr]

Pre-PR review

An independent read-only reviewer (a Sonnet subagent) checked this diff before the PR was opened. It found no high or medium issues. Its findings and my responses:

1. The delta question for other units omitted volume (low). Agreed. The batched "How does <id> differ from <primary>?" question in clarify.md now names volume, so a unit with very different volume can say so instead of silently inheriting the primary's value.
2. The Temporal adapter has no expected_scale row (low). Agreed in part. It is a real gap, but I left it out. Temporal workflow volume does not map cleanly to agent sessions per day, and a guessed mapping would move verdicts for users who never answered. These units stay unknown (neutral) unless asked, and the PR description says so.
3. traffic_pattern and expected_scale can stack (low). Acknowledged, no change. They measure shape and volume, and the Clarify text keeps them as separate questions. The product owner chose these scores knowing an idle prototype counts on both. The PR description discloses it.
4. Business and technical wording differed slightly (nit). Agreed. The business question now says "sessions (conversations or tasks) per day", matching the technical "sessions/day".

The reviewer also confirmed these needed no change:

• assumptions_used gains "expected_scale defaulted to unknown" like every unasked dimension, and nothing downstream asserts on that list.
• expected_scale belongs in seed unitDims, not systemDims.
• score_units.py and the Confirm, Design and Generate docs have no per-dimension lists that needed updating.
• An unasked value is the legal unknown, not an assumed entry.
• No em dashes in added text.

Pre-existing, noted in the description and not fixed here: seed unitDims lacks instance_type_requirement.

[leon1418]

[🤖 AI review 🤖] One P2 scoring issue remains: the new small-scale cost bonus also applies to AgentCore Instances.

Required correction: make the AgentCore scale affinity depend on its applicable compute/cost model in scripts/scoring.py and references/runtimes/agentcore.json, with regression coverage in scripts/test_scoring.py. Prototype/growing workloads requiring Instances through duration, heavy/GPU compute, or instance-type choice should not receive the microVM scale-to-zero bonus. Preserve the intended microVM cases, unknown-scale verdict compatibility, and per-unit scoring.

Reviewed all 11 changed files at 00c95a758118e4ccd69d594d3b65b2aae95b2493 with OCR-assisted coverage, an independent general review, and repository contract verification. Local validation: 647 agent-advisor tests; 1,717 plugin tests and 225 subtests; artifact, parity, reference, Markdown and diff checks passed. The focused reproduction below also passed. GitHub CI was still pending when captured.

[leon1418]

[🤖 AI review 🤖] [P2] Condition the small-scale bonus on AgentCore's compute type

This affinity rewards AgentCore at small scale even when the workload requires Instances. _compute_scores and _determine_verdict run before _select_agentcore_compute_type, so the microVM scale-to-zero cost rationale is applied to the EC2-backed branch too. The existing AgentCore card and Estimate instructions say idle Instances incur EC2 cost plus a management fee.

For session_duration=over_8hr, traffic_pattern=steady, and compute_tier=gpu, unknown scale gives ECS alone (ECS 40, AgentCore 37). Setting expected_scale=prototype produces an ECS/AgentCore tie (39 each), and AgentCore is routed to instances. The later billing warning doesn't correct that cost-based ranking signal.

Resolve the applicable AgentCore compute/cost model before applying this affinity, and add prototype/growing regression cases for duration, heavy/GPU compute, and instance-type requirements. Keep the scale-to-zero bonus for the microVM branch.