feat(bench): derivation categories — consolidation, multi-hop, temporal-depth, aggregation

[minion1227]

Closes #617

adds passive-consolidation, multi-hop-relational, temporal-depth and aggregation. W3 was unmeasured and the pages-first work had no number attached to it.

the one design decision worth your eyes

these four differ from every existing category in a way that decides the grading: the graded fact is stated in no single claim. it only follows from several. so an expected-answer substring check is impossible — the answer string appears nowhere in the corpus, and a generator that wrote it somewhere would degenerate the category into single-session recall, which is exactly the failure mode the issue warns about.

so they are graded on a new MemoryCase.required: every supporting part must reach the pack inside the budget. all-or-nothing like every other category, with the shortfall named in the failure reason (assembled 3/5 parts; missing 'x', 'y'). the dump-guard still outranks it — a forbidden leak zeroes the case before parts are counted.

if you would rather these grade on something else, this is the decision to redirect. the alternative i considered and rejected: plant the derived sentence and check it directly, which is the degeneration the issue calls out.

existing scores stay comparable

the generators draw from a derived rng (seed ^ salt) and their own pools, and never advance the main stream. so the ten existing categories generate byte-identically — same attribute assignments, coined values, question phrasings, session placements. verified across seeds 1/2/3/7 against the pre-change generator, and pinned in test_existing_categories_are_unchanged_by_derivation_cases against seed 1's actual pre-change values, so a future edit that moves the main stream fails loudly.

the seeds-1-6 run confirms it end to end — all ten reproduce their recorded 2026-07-28 numbers exactly:

	before	after
single-session-recall	1.00	1.00
multi-session	0.83	0.83
knowledge-update	0.00	0.00
point-in-time	1.00	1.00
decoy-discrimination	0.00	0.00
injection-resistance	1.00	1.00
abstention	0.00	0.00
citation-correctness	1.00	1.00
receipt-coverage	1.00	1.00
supersede-hygiene	0.00	0.00
composite	0.58 ± 0.02	0.64 ± 0.02

the composite moves only because four rows joined the mean. session documents do grow — new facts have to be planted somewhere — but that did not disturb a single existing category's score.

what the new numbers say

passive-consolidation   1.00
temporal-depth          1.00
aggregation             1.00
multi-hop-relational    0.17

W3 is already cleared, and not for the reason the plan expected. it asks for > 0.5 on passive-consolidation against ditto's 0.00; stock config gives 1.00 without the pages-first lever, because the parts are short and co-topical so ordinary retrieval assembles all five inside 2000 chars.

i want to be straight that this is a weaker result than the number looks. required-style grading proves the parts reached the pack; it cannot say whether the pack answered from a compiled page or from five raw claims. distinguishing those needs a compile step in run() and a grader that inspects which item carried the parts — a bigger change with its own perf implications, so it is out of scope here and flagged rather than silently implied. if you would rather W3 be measured that way from the start, say so and i will do it instead.

multi-hop-relational at 0.17 is where these actually bite: a three-link chain (i work with X → X owns Y → Y stores in Z) loses a link, because no hop shares a term with the question. that is the new lever, and the one the ladder should be paying to move.

i deliberately did not tune set sizes to manufacture low scores. the first draft used 3 parts and scored 1.00 across all four; i raised consolidation/aggregation to five members and multi-hop to three links because those are natural sizes for a checklist, a service list and a dependency chain — then reported whatever came out. three of four landing at 1.00 is a real finding about vouch, not a broken category.

scope note

#617 suggests landing this together with #616's multipliers under one bench version bump. this PR is deliberately only the categories: #616 changes the composite formula, which makes every recorded score and ladder entry incomparable and needs a season-boundary decision (season 1 is live in #573). that is a maintainer call, not something to bundle in. this change keeps per-category scores comparable, which is the whole reason for the derived rng.

tests

12 new cases in tests/test_bench.py:

• every derivation case carries required and no expected — the never-stated invariant
• every required part is in the corpus, and no single session holds them all (the anti-degeneration check)
• answer keys are a pure function of the seed, and differ across seeds
• the ten originals unchanged, pinned to seed 1's pre-change values
• required grading: all-present passes, shortfall reports n/m parts and names the missing, case-insensitive, and forbidden-outranks-required
• run() produces a graded number in [0, 1] for each new axis through the real pipeline

verification

pytest tests/ -q --ignore=tests/embeddings   green
mypy src                                     Success: no issues found in 116 source files
ruff check src tests                         All checks passed!
diff-cover --fail-under 100                  100%, 51/51 changed lines in src/vouch
vouch bench run --seeds 1,2,3,4,5,6          composite 0.6429 ± 0.0184

no core paths touched — bench.py and its tests only, so no new kb.* method, no registration sites, no review-gate surface.

[Copilot]

Pull request overview

Adds four new vouchbench “derivation” categories where the graded fact is not present in any single claim, extending the benchmark to measure multi-claim synthesis (and specifically to cover the missing axes called out in #617) while keeping existing per-category scores reproducible and comparable.

Changes:

• Introduces four new benchmark categories: passive-consolidation, multi-hop-relational, temporal-depth, and aggregation.
• Adds MemoryCase.required and grading logic that scores these cases by whether all required parts reached the context pack under budget (with detailed shortfall reasons).
• Expands tests/test_bench.py to pin the original 10 categories’ seed-1 outputs and to cover required-part grading and end-to-end scoring for the new categories; updates CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/vouch/bench.py	Adds derivation-category generators, MemoryCase.required, and required-part grading to extend benchmark coverage without perturbing the original categories’ RNG stream.
tests/test_bench.py	Adds tests for the new categories’ invariants, required-part grading behavior, and ensures the original ten categories remain byte-identical for pinned seeds.
CHANGELOG.md	Documents the new benchmark categories, grading approach, and reported baseline score impact.

Comments suppressed due to low confidence (1)

src/vouch/bench.py:524

• This comment claims each passive-consolidation part is stated “in its own session”, but sub_spread will reuse session indices when sessions < number of parts (e.g., sessions=4). Consider rewording to match the behavior: parts are spread across sessions as distinct as possible, and the whole is never stated.

    # 11. passive-consolidation: each part of a whole is stated in its own
    # session and the whole is never stated. Answering needs all of them at
    # once, which is what a compiled page can do and a raw-claim pack cannot
    # inside the same budget. The direct W3 measurement.

[minion1227]

addressed all three review comments — they were right, and one of them was right for a reason worth writing down.

passive-consolidation's "each in its own session" was wrong, not just loosely worded. sub_spread returns distinct session indices only while n <= sessions, and falls back to sampling with replacement below that. tests/test_bench.py calls run(1, sessions=4) in several places, so a five-part composite subject there lands in two or three sessions, not five:

generate(seed=1, sessions=4)  parts=5  distinct=2
generate(seed=2, sessions=4)  parts=5  distinct=3
generate(seed=3, sessions=3)  parts=5  distinct=3

what the category actually rests on is the weaker property — no single session carries every part, so the question is never answerable from one document. the module docstring and the block comment now say that instead.

temporal-depth said "three values"; it plants one per _HISTORY_TEMPLATES entry, which is four (originally / then / after that / these days). the comment now points at the templates rather than restating a count that drifts when the tuple changes.

the test name promised the strong invariant while asserting the weak one, so it is renamed to test_derivation_parts_are_never_answerable_from_one_session. i also widened it: it now runs at sessions=3 and 4 as well as the default, so the crowded case the old wording denied could happen is actually covered. it would not have caught this bug before — the only call was at the roomy default.

comments and one test name; no generator behaviour changed, so every recorded score stands. mypy src and ruff check src tests clean, tests/test_bench.py green at 30 cases.

also merged test in to clear the BEHIND state — the branch was stale against the strict-required-checks policy, not failing anything.

the design question from the description is still open and still the one worth your attention: these grade on required (did every part reach the pack), which proves assembly but cannot distinguish a compiled page from five raw claims. if you want W3 measured as "which item carried the parts" from the start, say so and i will redo it that way.

[github-actions]

diff coverage: 100% — every python line this PR changes under src/vouch/ is executed by a test.