Written so the next person reads code instead of re-deriving. Every finding below names the file and function that establishes it.
Measured vs asserted. Findings 1, 2 and 6 are read directly off the code and are exact. Findings 3, 4, 5, 7 and 8 are measurements from a small number of runs and carry their n and their spread — none of them is a claim beyond that. Nothing in this document is an estimate presented as a measurement; where a number could not be obtained, it says so.
Start here: README.md for setup and commands, this file for why the system
is shaped the way it is and what is already known not to work.
| Commit | What |
|---|---|
b75e4f8 |
Deleted _REQUIREMENT_OPEN, a branch that could never decide. Proved behaviour-neutral over 1,032 cases before removal. |
a813c55 |
_SYMPTOM_CUE's still\s+\w+ing could not cross a hyphen, so "still hand-rolling" was rejected. Now still\s+[\w-]+ing. |
b2a9901 |
Three prose edits: a pivot form for Multi-Turn, trigger events for Command, toil-as-symptom for Implicit. |
3a05e67 |
Pinned the prefix budget (superseded by f607a9a). |
3acfcef |
Pre-warm the prefix cache before topic fan-out. Prefix billed-equivalents per run 43,250 → 12,110 (−72%). Cost, not latency. |
f607a9a |
Budget the prefix in measured tokens via count_tokens, not chars/4, which understated by ~41%. |
ffbcf02 |
Restored the axis parenthetical in the pivot block. |
77e5758 |
Multi-Turn candidates carry form (constraint | pivot), set at construction. |
51cfe43 |
Attributed every stage-4 and stage-5 drop. |
3c16a7e |
Prefill/decode split in --timings. |
44dbd78 |
The stage-4 reporting tail sat after return 0 and had never printed. Made reachable. |
6904e53 |
Split the stage-5 label: quota cut vs ICP floor backfill. |
083e155 |
Persist every validated candidate with the stage that decided its fate (migration 0004). |
84a50db |
Renamed drop_stage → stage_outcome; reserved its NULL (migration 0005). |
a24f10d |
Run integrity gate: retries, non-zero exit on failure, partial runs, export refusal (migration 0006). |
85a84c9 |
Scoped _generate's session to reads and writes; corrected the NIM cache claim. |
c4b7fe0 |
Stage precedence fix; surfaced the ceiling undoing ICP floor backfill. |
c26601a |
Capsule coverage report — sources per field, empty cells, gated archetypes (migration 0007). |
4120464 |
Cross-run stability measurement (scripts/cross_run_stability.py). |
bc5faf8 |
Answerability smoke test (scripts/answerability.py). |
Do not re-derive these. Each names where to read it.
apply_quotas — promptgen/validation/setlevel.py:136 —
recurses per topic and hard-caps each archetype inside it with while taken < quota. quota comes
from the directives. Everything past it is discarded before ranking is consulted. Evidence
weight only orders candidates within a persona bucket; it decides which candidates fill the slots,
never how many slots exist.
Consequence: the Multi-Turn pivot edit changes which Multi-Turn prompts ship, never how many. Vercel has 5 Multi-Turn slots per run regardless of the prompt spec.
topic_quotas and archetype_quotas — promptgen/capsule/directives.py:189
and :90. Per topic: size = max(MIN_PER_TOPIC, min(MAX_PER_TOPIC, round(total / n_topics))), then
size - 1 generated slots after the reserved sentiment-audit slot. For vercel that is 3 slots.
Deployments has all four archetypes live — it has the Multi-Turn evidence. Three slots cannot
cover four live archetypes, so one gets zero, and largest-remainder plus rotate=index decides
which. The rotation exists precisely so it is not the same archetype on every topic.
Measured for vercel:
| topic | live archetypes | E | C | I | M |
|---|---|---|---|---|---|
| AI integrations | E,C,M | 1 | 1 | 0 | 1 |
| Agent | E,C,M | 0 | 2 | 0 | 1 |
| CLI | E,C,M | 0 | 1 | 0 | 2 |
| Deployments | E,C,I,M | 1 | 1 | 1 | 0 |
| Functions | E,C,I,M | 0 | 1 | 1 | 1 |
MIN_PER_TOPIC (currently 4, directives.py:185) is the knob. See Open decisions.
Stage-1 rejection across four clean runs (vercel ×2, mastra, nike), 60 rejections total:
| reason | count | share | validator |
|---|---|---|---|
no_situation |
34 | 57% | carries_situation — schema.py:162 |
not_symptom_first |
23 | 38% | opens_with_symptom — schema.py:169 |
over_constrained |
3 | 5% | _CONSTRAINT_RE |
Per-topic fill ran 17–50% across six topics and three clients. Slots go unfilled, and the two validators above account for 95% of stage-1 loss — which is exactly where the Command trigger-event and Implicit toil edits are aimed.
Four vercel runs exist but only two of the four share the current spec: the
prefix moved by 13 tokens when ffbcf02 restored the axis parenthetical at
11:29, and the runs at 10:55 (21 rejections) and 11:27 (24) predate it. They are
not in any band.
Three runs are attributed (capsules 59, 60, 61), all on the frozen spec (prefix 19,654 chars / 6,938 tokens, no validator change):
| capsule | rows | rejected | validated | rate |
|---|---|---|---|---|
| 59 | 92 | 22 | 65 | 25% |
| 60 | 80 | 20 | 55 | 27% |
| 61 | 80 | 16 | 59 | 21% |
The raw count is a bad measure and it is what made the variance look unmanageable. Counts ran 16-22 here, and an earlier post-C3 run landed at 9 -- but emission volume also moved (92 vs 80 rows), so the count conflates how much the model produced with how much was rejected. On rate the three runs sit at 21-27%, a 6-point spread. The 9-rejection run predates persistence and has no denominator, so it cannot be rate-normalised and cannot be compared to these.
Where the variance actually sits:
| reason | 59 | 60 | 61 | archetype |
|---|---|---|---|---|
no_situation |
14 | 12 | 12 | Command |
not_symptom_first |
4 | 7 | 4 | Implicit |
tail (evidence_not_verbatim, over_constrained, too_long) |
4 | 1 | 0 | mixed |
| topic | 59 | 60 | 61 | range |
|---|---|---|---|---|
| CLI | 4 | 4 | 4 | 0 |
| Agent | 2 | 3 | 2 | 1 |
| AI integrations | 5 | 2 | 4 | 3 |
| Deployments | 6 | 2 | 2 | 4 |
| Functions | 5 | 9 | 4 | 5 |
It does not spread evenly. ~60% of every run is Command/no_situation,
stable at 12-14 and spread across all five topics -- that is a structural rate,
not noise. The volatility is concentrated in Functions (range 5) and
Deployments (range 4), while CLI is perfectly stable at 4 every run. The
one-off tail reasons appear in a single run each.
Practical consequence: a prose edit aimed at carries_situation is measurable
against the stable Command/no_situation core; one aimed at Implicit on the
Functions topic is not, at n=3. Use the rate, hold emission volume in view, and
report per-topic rather than per-run totals.
Validated candidates per archetype against that archetype's quota, three
frozen-spec vercel runs (capsules 59/60/61), now recorded on every run row as
capsules.archetype_supply:
| archetype | quota | 59 | 60 | 61 | ratio |
|---|---|---|---|---|---|
| Explicit | 2 | 20 | 17 | 17 | 9.0x |
| Multi-Turn | 5 | 27 | 27 | 26 | 5.3x |
| Implicit | 2 | 8 | 2 | 6 | 2.7x |
| Command | 6 | 15 | 14 | 15 | 2.4x |
apply_quotas discards everything past quota before ranking is consulted
(setlevel.py:136), so no validator
edit and no prompt-spec edit anywhere changes shipped output. Command is the
worked example: it rejects ~47% of its candidates (carries_situation,
schema.py:162), and halving that rate
would add ~13 more validated candidates competing for the same 6 slots — zero
additional shipped prompts.
This caps the prompt-spec work directed across several sessions. It is the single most decision-relevant number in the system, which is why it is now a column rather than an investigation.
directives_for_topic (generate.py:260)
asks for max(6, 2 * max per-archetype quota in the topic) per archetype. The
client-level CANDIDATES_PER_ARCHETYPE = 16 is deliberately overridden — 16
inside a per-topic call asked for ~48 records and overran max_tokens, losing a
whole topic as "not valid JSON".
For vercel the max(6, ...) floor binds on every topic and the 2x term never
does: 78 candidates requested against 15 generated slots. A 21-27% rejection
rate therefore does not arithmetically force under-fill. CLI's 1-of-3 is a
per-cell effect (Command 0 valid / 4 rejected on that topic), not a volume one,
and doubling the request cannot fix a cell that rejects everything it emits.
Raising over-generation is a config change with a token cost and belongs with the other open decisions, not as cleanup.
Six clean frozen-spec vercel runs (capsules 59/60/61/64/65/66, 20 shipped each),
measured by scripts/cross_run_stability.py. Similarity is lexical, reusing
stage 4's own term-overlap cosine at its 0.62 threshold — there is no embeddings
provider configured, so paraphrase is under-counted. A lexical number called a
semantic one would be this repo's recurring defect.
Prompt strings are not stable. Pairwise overlap is 40–50%; 91% of prompt clusters appear in only 1 of the 6 runs; 8 of 80 distinct strings repeat at all.
Coverage is stable. Spread below is max−min across the six runs, in percentage points of each run's 20:
| axis | worst spread | present every run |
|---|---|---|
| topic | 5pp (three of five topics identical every run) | 5 of 5 |
| archetype | 10pp (Implicit 0pp, Explicit 5pp) | 4 of 4 |
| persona | 15pp (platform-lead 2–5, startup-cto 2–5) | 3 of 3 |
Read it this way. Low string stability is expected and not a defect: 20 slots are sampled from a pool running 2.4x–9.0x over quota (finding 5), so which strings win varies by design. The instrument-level claim is coverage, and it holds — every topic, archetype and persona is represented in every run.
Two caveats, or the number overstates itself. Topic and archetype shape is partly structural — quotas fix it in advance, so this measures that selection realises the configured shape run to run, not that the model is consistent. And persona is the least stable axis at 15pp, the one place the mix moves enough to matter; it is also the axis with no quota of its own, only an ICP floor.
scripts/answerability.py, one engine (claude-opus-5), no search tool, on
capsule 66's 20 shipped vercel prompts. Rates are over the 19 responses that
measured anything; one hit max_tokens before emitting text and is excluded.
| of 19 measured | |
|---|---|
| named any product | 19 (100%) |
| declined to recommend | 0 (0%) |
| named the client | 13 (68%) |
Zero prompts declined to recommend. A prompt that answers "I'd need more context" is a wasted tracking slot however cleanly it validates, and nobody had checked. This is the narrow result that it never happens here.
What it is not. It says nothing about ranking accuracy, engine coverage, or
whether these prompts resemble real search behaviour. With no search tool the
named the client figure is the model's prior, not share-of-voice — do not
quote it as one.
One defect this found in itself. The first version reported "named any product 20/20" while every response was EMPTY: Opus 5 runs thinking by default and it consumed a 300-token budget, and the script inferred "named a product" from the absence of a hedge phrase rather than from a product name. Two numbers labelled as things they were not, in a script written to measure something else. The rerun disables thinking and counts product names directly.
These go to Naman with the arithmetic attached. None is a diff anyone should write first.
- Evidence: quota decides how many ship (finding 1); within a slot, ordering is by evidence
weight only —
apply_quotas, theweights.get(c.source_evidence, 0)sort. Stage 3 scoring is unbuilt. So which pivot ships, when more than one competes, is decided by the weight of the capsule row behind it and nothing about the prompt. - Tradeoff: a real score could prefer forms that measure better, but there is no measurement behind any candidate scoring function yet, and finding 4 says none is admissible until a variance band exists.
- This is now the highest-value open decision, not a nice-to-have. Finding 5 shows every archetype over-supplies its slots 2.4x–9.0x, so validation and the prompt spec are capped and cannot change shipped output at all. Ranking within a slot is the ONLY remaining lever on output quality. Lead with the over-supply number when making the case.
- Decides: Naman.
- Evidence: finding 2. At 4, each topic gets 3 generated slots; any topic with 4 live archetypes must zero one.
- Tradeoff: raising it to 5 gives 4 slots and removes structural zeros, at the cost of more prompts per topic against a fixed 20-record ceiling — so it trades per-topic completeness against topic coverage.
- It is waste, not just a coverage preference, and it recurs. Deployments'
Multi-Turn quota is 0, yet candidates keep landing in that cell because a call
for one topic may label prompts with any capsule topic (deliberate —
generate.py:208— it once saved 27 of 47 candidates). Measured across capsules 59/60/61: 10, 12 and 11 validated Multi-Turn candidates labelled Deployments, 11–15% of every run's rows, all discarded for a cell that was never eligible. That is decode spent every run on output that could never ship. - Decides: Naman. Needs no new measurement; the arithmetic above is sufficient.
- Evidence:
formandstage_outcomeare useful downstream butEXPORT_FIELDS(cli.py, checked inexport_records) is a contract with the eval service, validated per row on every export. - Tradeoff: adding a field is a product decision requiring the consumer to accept it, not cleanup. Both fields are available in Postgres today without touching the contract.
- Decides: Naman, with the downstream consumer.
- Run integrity (
a24f10d). A failed topic call can no longer look like a thin topic: calls retry with backoff, a run with any failure exits non-zero, is persisted aspartialwith its failure count, and refuses to export without--force-partial. This was the disqualifying defect — credit exhaustion once produced 6/24 and 15/24 deliveries that exited 0. _generatesession lifetime (85a84c9). Scoped to the reads and the writes; no session spans the model calls. Guarded structurally by an AST test, becauseasync with ... as sessionleaves the name BOUND after the block, so a closed-session bug raises no NameError and neither ruff nor pytest sees it.nim.pycache_control(85a84c9). The premise was wrong in both directions.cache_controlis an Anthropic request field with no OpenAI-compatible equivalent, so there is nothing to un-drop; but NIM does cache automatically and report it, so the old comment "there is nothing here to cache" was false. A pre-warm — the fix that worked on the Anthropic path — was written, measured over three runs (1/3, 1/3, 0/3 topic calls cached) and reverted: the warm-up reported 0 cached and never seeded anything the topic calls could read. Do not re-add it without new evidence.
trim_to_ceiling runs after enforce_persona_floors, so a candidate restored
specifically to meet an ICP floor can then be trimmed, defeating the coverage
the floor exists to enforce. Found by scripts/reconcile_run.py as a mismatch,
not by review. The run now prints it (-N of those restored were then cut by the 20-record ceiling) and the row reads stage_outcome='ceiling'. Whether the
ceiling should be floor-aware is a selection decision and was not changed.
The credit-blocked items are closed. All three were confirmed against live runs, not asserted:
-
Prefix token pin measured.
test_prefix_stays_inside_its_token_budgetun-skips and reads 6,938 against the 7,600 ceiling, matching its pin. It measures rather than skipping; confirmed with-rs. -
Persistence reconciles against live runs. Two runs, both exit 0, both fully reconciled by
scripts/reconcile_run.py <run.out> <capsule_id>:run rows shipped rejected dedupe quota (gross) floor ceiling mastra.ai 55 19 6 14 19 3 0 vercel.com 92 20 22 25 24 0 1 Row count reconciles against validated candidates, not shipped (55 vs 19, 92 vs 20). No row carries a NULL
stage_outcome. Every bucket is exercised across the pair: mastra coversfloor, vercel coversceiling. -
One bucket is still unexercised live.
brand_leakcame back 0 in both runs -- no Implicit prompt named the brand. It is covered by the offline assertions intests/test_contracts.pyonly. Do not read a green reconciliation as proving that path.
Confirmed in these runs: mastra's pre-warm created the prefix entry, and vercel's pre-warm read it. vercel showed 6 x 6,933 cached tokens for 5 topic calls plus its pre-warm -- the pre-warm itself cost nothing.
This is stronger than the per-run C1 number. The 72% saving is what one isolated client sees; run clients back to back inside the 5-minute TTL and only the first pays cache creation at all, so the marginal client pays reads only. That makes batching a cost lever in its own right, separate from the pre-warm.
Boundary, and it is silent when crossed. This holds only while the prefix is
client-independent -- static_prefix() is loaded verbatim from
promptgen/synthesis/master-prompt.md with everything per-client travelling in
<generation_directives> instead. Anything that ever interpolates a client value
into the prefix kills both the cross-client amortisation and the within-run
pre-warm, and the loss shows up as cost, not as an error. The guard is
test_prefix_is_byte_identical_across_clients in tests/test_synthesis.py.
Five vercel runs to establish the variance band from finding 4, then three each on mastra and nike only if vercel's band is tight. Freeze conditions: no prose edits (prefix must read 6,938 tokens), no validator changes, capsule read from cache and confirmed identical, same model/thinking/effort. Report the band, do not act on it.
Reconciliation no longer gates this -- it was closed by the two runs above. Three vercel stage-1 rejection counts are on record so far, all on the same spec: 21, 9, 22. That is the spread the band has to characterise, and it is wide enough that no prose-edit effect could be separated from it today.
The one measurement that would change how this product is understood. It needs Manicule's data, not credits, so it is scoped here rather than attempted.
State the question correctly or a good result reads as a failure. It is not "does the instant 20 reproduce the hours-long dataset" — 20 prompts against thousands is not a coverage claim and never could be. It is: does the instant 20 produce the same answer? If share-of-voice measured with 20 prompts tracks share-of-voice measured with the full set, the instant set is a sampling instrument rather than a smaller version of the slow one. That is a much lower bar and the right one.
Inputs needed (ask before building past the harness):
- The slow pipeline's full prompt set for one client promptgen also has a capsule for — vercel or mastra ideally.
- That client's tracking results from the full set: per prompt, which brands the engines returned, and the derived share-of-voice.
- One tracking run of the instant 20 through the same engines, on Manicule's infrastructure.
If (2) already exists, most of this is analysis rather than new measurement.
Tier 1 — free, no engine runs. Runnable the moment the slow prompt set arrives; report before asking for anything else. Cluster recall (what fraction of the slow set's clusters the instant 20 lands in, and which large clusters it misses — a long-tail miss is expected and fine). Shape comparison (classify the slow set with promptgen's own validators and compare archetype distributions). Vocabulary overlap (category nouns and named competitors; a competitor in the slow set and absent from the capsule is a capsule coverage gap with a name attached, useful regardless of how the rest turns out).
Note for whoever builds it: there is no embeddings provider configured, so
"cluster" means the lexical term-overlap cosine in setlevel.py, or a new
dependency. Say which.
Tier 2 — needs the tracking runs. Share-of-voice from both sets, per engine, with the gap. Stability across more than one promptgen run — an instant number that swings 15 points between runs is unusable however well it tracks on average, and finding 7 says coverage is stable while strings are not, so this is open. Per-archetype contribution: which archetypes actually retrieved the client. If one contributes nothing, that is a quota conversation with evidence, and it connects to the ranking decision above.
Reporting rules. Rates, not counts — the sets differ in size. No claim from a single run. Label every metric with what it is over. If the instant set does not track the slow set, report that plainly: a negative result here is worth more than a hedged positive, and it points at capsule coverage rather than at the generator.
A number that is correct about one thing and labelled as another. Five instances so far:
_REQUIREMENT_OPEN— a branch that could never change a verdict (b75e4f8).session_scopeundefined at six CLI call sites, with 236 tests green — no test exercises a CLI database path.- The stage-4 reporting tail sitting after
return 0, printing nothing while 254 tests passed (44dbd78). -30 cut by quota/ICP floorsnetting an add against a subtract, which let under-supply exceed the delivery gap (6904e53).model returned N valid candidatescomputed on the post-dedupe pool, attributing to the model what dedupe may have caused (083e155).
Plus two caught before landing: drop_stage = 'floor' on rows that ship (84a50db), and a NULL
meaning both "clean survivor" and "predates the column" (84a50db).
When something reconciles badly, report it. Do not adjust either side until they agree.