Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ clarify.md Step 3 (do not invent new keys/values).
it paused?" → process_level / filesystem / none.
- **launch_concurrency**: "At peak, roughly how many new sessions start per second?" → high
(many) / moderate / low.
- **expected_scale**: "Over the next few months, roughly how many sessions (conversations or tasks)
will the agent handle per day: under 100, 100 to 10,000, or more than 10,000?" → prototype /
growing / high; unknown if unsure.
- **multi_agent**: "One agent, or several working together?" → no / yes.
- **deployment_preference**: "Would you rather AWS fully manage the agent for you (you just
describe it — no code to maintain), bring your own agent code, or let me recommend?" →
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ Use direct technical terms. Map each answer onto the keys in clarify.md Step 3.
- **idle_resume**: "On idle-then-resume, do running processes need to continue exactly
(process_level), is filesystem persistence enough (filesystem), or not needed (none)?"
- **launch_concurrency**: "Peak new-session launch rate? high (>5/sec) / moderate / low."
- **expected_scale**: "Expected volume over the next few months? <100 sessions/day (prototype) /
100-10K (growing) / >10K (high)." Volume only; shape is `traffic_pattern`, peak rate is
`launch_concurrency`.
- **multi_agent / framework / existing_cluster / multi_cloud / platform_fit**: ask directly.
- **deployment_preference**: "Do you want a no-code managed agent runtime (AgentCore Harness),
bring your own framework code (Strands/LangGraph/CrewAI/custom), or either?" Ask early. Only
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,9 @@ single workload), treat it as `agent_session` and proceed.
question set
(`session_duration`, `traffic_pattern`, `session_state`, `isolation`,
`memory_needs`, `multi_agent`, `framework`, `idle_resume`, `compute_tier`,
`launch_concurrency`, `instance_type_requirement`).
`launch_concurrency`, `instance_type_requirement`, `expected_scale`).
- **Every other unit: ONE batched delta question** — "How does `<id>` differ from
`<primary>`? (session duration / traffic / compute / state / memory / isolation — name
`<primary>`? (session duration / traffic / volume / compute / state / memory / isolation — name
only what differs)". Parse the reply into per-dimension overrides; dimensions the
user does not mention inherit the primary unit's answers. Non-agent units only need
the dimensions their workload-classes rules read (traffic, duration, compute) —
Expand Down Expand Up @@ -241,6 +241,10 @@ Collect answers for these keys. Legal values are fixed (Plan 1 Data Model):
wins, routes it to the Instances compute type (capacity provider).
- `idle_resume`: process_level | filesystem | none | unknown
- `launch_concurrency`: high | moderate | low | unknown
- `expected_scale`: prototype | growing | high | unknown. Expected sessions per day over the next
few months: prototype under 100, growing 100 to 10,000, high over 10,000. This is volume, not the
traffic shape (`traffic_pattern`) or the peak start rate (`launch_concurrency`). Never derive it
from `target_maturity`: when it is not asked or answered it stays `unknown`, which scores neutral.
- `multi_agent`: yes | no | unknown
- `deployment_preference`: harness | framework | either | unknown — do you want a no-code
**managed** agent runtime (AgentCore Harness — declare the agent as config, AWS runs the loop),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"idle_resume": { "process_level": 1, "filesystem": 4, "none": 2, "unknown": 2 },
"compute_tier": { "light": 4, "heavy_non_gpu": 3, "gpu": 3, "unknown": 2 },
"launch_concurrency": { "high": 5, "moderate": 3, "low": 2, "unknown": 2 },
"instance_type_requirement": { "yes": 3, "no": 3, "unknown": 2 }
"instance_type_requirement": { "yes": 3, "no": 3, "unknown": 2 },
"expected_scale": { "prototype": 4, "growing": 3, "high": 3, "unknown": 2 }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖] [P2] Condition the small-scale bonus on AgentCore's compute type

This affinity rewards AgentCore at small scale even when the workload requires Instances. _compute_scores and _determine_verdict run before _select_agentcore_compute_type, so the microVM scale-to-zero cost rationale is applied to the EC2-backed branch too. The existing AgentCore card and Estimate instructions say idle Instances incur EC2 cost plus a management fee.

For session_duration=over_8hr, traffic_pattern=steady, and compute_tier=gpu, unknown scale gives ECS alone (ECS 40, AgentCore 37). Setting expected_scale=prototype produces an ECS/AgentCore tie (39 each), and AgentCore is routed to instances. The later billing warning doesn't correct that cost-based ranking signal.

Resolve the applicable AgentCore compute/cost model before applying this affinity, and add prototype/growing regression cases for duration, heavy/GPU compute, and instance-type requirements. Keep the scale-to-zero bonus for the microVM branch.

},
"deployment_models": ["harness", "framework_on_runtime"],
"compute_types": ["microvms", "instances"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"idle_resume": { "process_level": 1, "filesystem": 1, "none": 2, "unknown": 2 },
"compute_tier": { "light": 1, "heavy_non_gpu": 4, "gpu": 5, "unknown": 2 },
"launch_concurrency": { "high": 3, "moderate": 2, "low": 2, "unknown": 2 },
"instance_type_requirement": { "yes": 4, "no": 2, "unknown": 2 }
"instance_type_requirement": { "yes": 4, "no": 2, "unknown": 2 },
"expected_scale": { "prototype": 1, "growing": 2, "high": 4, "unknown": 2 }
},
"deployment_models": [],
"volatile_facts": [{ "key": "regions", "value": "all", "verify_via_mcp": false }]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"idle_resume": { "process_level": 1, "filesystem": 1, "none": 2, "unknown": 2 },
"compute_tier": { "light": 1, "heavy_non_gpu": 4, "gpu": 5, "unknown": 2 },
"launch_concurrency": { "high": 3, "moderate": 2, "low": 2, "unknown": 2 },
"instance_type_requirement": { "yes": 5, "no": 2, "unknown": 2 }
"instance_type_requirement": { "yes": 5, "no": 2, "unknown": 2 },
"expected_scale": { "prototype": 0, "growing": 1, "high": 4, "unknown": 2 }
},
"deployment_models": [],
"volatile_facts": [{ "key": "regions", "value": "all", "verify_via_mcp": false }]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@
"idle_resume": { "process_level": 0, "filesystem": 1, "none": 3, "unknown": 2 },
"compute_tier": { "light": 3, "heavy_non_gpu": 1, "gpu": 0, "unknown": 2 },
"launch_concurrency": { "high": 4, "moderate": 3, "low": 2, "unknown": 2 },
"instance_type_requirement": { "yes": 0, "no": 3, "unknown": 2 }
"instance_type_requirement": { "yes": 0, "no": 3, "unknown": 2 },
"expected_scale": { "prototype": 4, "growing": 3, "high": 2, "unknown": 2 }
},
"deployment_models": [],
"volatile_facts": [{ "key": "timeout", "verification_key": "lambda.timeout", "value": "15m", "verify_via_mcp": true }]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@
"idle_resume": { "process_level": 5, "filesystem": 3, "none": 1, "unknown": 2 },
"compute_tier": { "light": 2, "heavy_non_gpu": 7, "gpu": 0, "unknown": 2 },
"launch_concurrency": { "high": 0, "moderate": 2, "low": 3, "unknown": 2 },
"instance_type_requirement": { "yes": 0, "no": 3, "unknown": 2 }
"instance_type_requirement": { "yes": 0, "no": 3, "unknown": 2 },
"expected_scale": { "prototype": 3, "growing": 3, "high": 2, "unknown": 2 }
},
"deployment_models": [],
"volatile_facts": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@
"compute_tier": { "enum": ["light", "heavy_non_gpu", "gpu", "unknown"] },
"idle_resume": { "enum": ["process_level", "filesystem", "none", "unknown"] },
"launch_concurrency": { "enum": ["high", "moderate", "low", "unknown"] },
"expected_scale": { "enum": ["prototype", "growing", "high", "unknown"] },
"multi_agent": { "enum": ["yes", "no", "unknown"] },
"deployment_preference": { "enum": ["harness", "framework", "either", "unknown"] },
"framework": { "enum": ["strands", "langgraph", "crewai", "custom", "none", "unknown"] },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"session_duration", "traffic_pattern", "platform_fit", "session_state",
"ops_preference", "isolation", "memory_needs", "multi_agent", "framework",
"existing_cluster", "multi_cloud", "idle_resume", "compute_tier",
"launch_concurrency", "instance_type_requirement",
"launch_concurrency", "instance_type_requirement", "expected_scale",
]

# Legal answer values per scoring dimension (the closed set the engine reasons about).
Expand All @@ -36,6 +36,7 @@
"compute_tier": ["light", "heavy_non_gpu", "gpu", "unknown"],
"launch_concurrency": ["high", "moderate", "low", "unknown"],
"instance_type_requirement": ["yes", "no", "unknown"],
"expected_scale": ["prototype", "growing", "high", "unknown"],
}

DEFAULTS = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -834,3 +834,67 @@ def test_runtimes_dir_points_at_skill_references():
parts = RUNTIMES_DIR.parts
assert parts[-2:] == ("references", "runtimes"), RUNTIMES_DIR
assert parts[-3] == "agent-advisor", RUNTIMES_DIR


# --- B4 (#566 feedback F2): expected_scale -----------------------------------------------------

def test_unknown_scale_scores_neutral_on_every_runtime():
# Existing runs and seeds carry no expected_scale; they must score exactly as before.
for profile in scoring.load_profiles():
assert profile["affinities"]["expected_scale"]["unknown"] == scoring.NEUTRAL_SCORE
answers = {"session_duration": "15min_to_8hr", "traffic_pattern": "steady"}
assert scoring.score({"answers": answers})["scores"] == scoring.score(
{"answers": {**answers, "expected_scale": "unknown"}})["scores"]


def _scale_gain(scale):
answers = {"session_duration": "15min_to_8hr"}
before = scoring.score({"answers": answers})["scores"]
after = scoring.score({"answers": {**answers, "expected_scale": scale}})["scores"]
return {runtime: after[runtime] - before[runtime] for runtime in before}


def test_prototype_scale_favors_scale_to_zero_runtimes():
gain = _scale_gain("prototype")
assert min(gain["agentcore"], gain["lambda"]) > gain["lambda_microvms"] > gain["ecs"] > gain["eks"]


def test_high_scale_favors_container_runtimes():
gain = _scale_gain("high")
assert min(gain["ecs"], gain["eks"]) > gain["agentcore"] > max(gain["lambda"], gain["lambda_microvms"])


_STEADY_AGENT = {"session_duration": "15min_to_8hr", "traffic_pattern": "steady",
"ops_preference": "moderate", "existing_cluster": "none"}


def test_prototype_scale_brings_agentcore_into_the_recommendation():
answers = {**_STEADY_AGENT, "framework": "strands", "multi_agent": "no", "platform_fit": "none"}
assert scoring.score({"answers": answers})["verdict"] == "lambda_microvms"
result = scoring.score({"answers": {**answers, "expected_scale": "prototype"}})
assert result["co_recommend"] == ["agentcore", "lambda_microvms"]


def test_high_scale_brings_ecs_into_the_recommendation():
answers = {**_STEADY_AGENT, "framework": "custom", "session_state": "stateful", "idle_resume": "none"}
assert scoring.score({"answers": answers})["verdict"] == "lambda_microvms"
result = scoring.score({"answers": {**answers, "expected_scale": "high"}})
assert result["co_recommend"] == ["ecs", "lambda_microvms"]


def test_seed_schema_accepts_expected_scale_and_rejects_other_values():
import jsonschema
schema = json.loads(
(pathlib.Path(scoring.__file__).parent / "schemas" / "seed.json").read_text())
jsonschema.validate({"units": {"agent": {"expected_scale": "growing"}}}, schema)
with pytest.raises(jsonschema.ValidationError):
jsonschema.validate({"units": {"agent": {"expected_scale": "huge"}}}, schema)


def test_clarify_documents_every_expected_scale_value():
clarify = (pathlib.Path(scoring.__file__).parent.parent / "references" / "phases" / "clarify")
legal_line = next(line for line in (clarify / "clarify.md").read_text().splitlines()
if line.startswith("- `expected_scale`:"))
assert legal_line.split(":", 1)[1].split(".")[0].strip().split(" | ") == scoring.LEGAL_VALUES["expected_scale"]
for audience in ("clarify-business.md", "clarify-technical.md"):
assert "**expected_scale**" in (clarify / audience).read_text()
Loading