What you're trying to do
.superpowers/BEAT-DITTO-PLAN.md names the categories ditto's engine scores
near zero on — multi-hop-relational, temporal-depth, passive-consolidation —
and stakes win condition W3 on beating them at consolidation:
W3 consolidation: > 0.5 on passive-consolidation-style questions (they
score 0.00) — answered from compiled topic pages, not raw pairs.
CATEGORIES in src/vouch/bench.py is:
single-session-recall, multi-session, knowledge-update, point-in-time,
decoy-discrimination, injection-resistance, abstention,
citation-correctness, receipt-coverage, supersede-hygiene
no passive-consolidation, no multi-hop-relational, no temporal-depth, no
aggregation. so W3 is currently unmeasured, and the pages-first work from P2
of that plan has no number attached to it — which is awkward, because
consolidation is the wedge the whole competitive story rests on.
What you've tried
multi-session → adjacent, but tests recall across sessions, not synthesis
across claims.- reading compiled pages by hand and being satisfied → not a measurement.
Suggested shape
add four generated categories, same seeded judge-free contract as the rest:
- passive-consolidation — a fact that appears nowhere in a single claim but
follows from several; answerable from a compiled page, not from any one raw
claim. the direct W3 measurement, and the one where pages-first packs should
visibly beat raw-claim retrieval.
- multi-hop-relational — answer requires traversing entity relations
(a works-with b, b owns c → who owns c). graph.py already walks these
edges; nothing scores whether retrieval uses them.
- temporal-depth — reasoning over a value's history rather than its current
value ("how many times did x change", "what was it before the change").
- aggregation — count / sum / list-all over claims matching a predicate.
each needs its generator to plant the supporting facts and its answer key to
stay a pure function of the seed. these are harder to generate than recall
cases: the generator has to guarantee the fact is only derivable, never
stated, or the category silently degenerates into single-session recall.
Compatibility considerations
adding categories shifts the overall mean, so the same versioning and
season-boundary question as the multipliers issue applies — worth landing both
under one bench version bump rather than two.
expect the new categories to score low at first. that is the point: they mark
where the engine actually is, and they are the ones the koth ladder should be
paying to move.
Alternatives
- measure consolidation by hand on a fixed corpus — not seeded, not
reproducible, not usable by the ladder.
- drop W3 from the plan — the honest alternative if this doesn't get built, and
a real loss, since it is the category where compiled pages beat every
retrieval-only competitor by construction.
part of the ditto-style track — see .superpowers/DITTO-STYLE-PLAN.md (T6.2)
and .superpowers/BEAT-DITTO-PLAN.md (W3, P2).
What you're trying to do
.superpowers/BEAT-DITTO-PLAN.mdnames the categories ditto's engine scoresnear zero on — multi-hop-relational, temporal-depth, passive-consolidation —
and stakes win condition W3 on beating them at consolidation:
CATEGORIESinsrc/vouch/bench.pyis:no
passive-consolidation, nomulti-hop-relational, notemporal-depth, noaggregation. so W3 is currently unmeasured, and the pages-first work from P2of that plan has no number attached to it — which is awkward, because
consolidation is the wedge the whole competitive story rests on.
What you've tried
multi-session→ adjacent, but tests recall across sessions, not synthesisacross claims.
Suggested shape
add four generated categories, same seeded judge-free contract as the rest:
follows from several; answerable from a compiled page, not from any one raw
claim. the direct W3 measurement, and the one where pages-first packs should
visibly beat raw-claim retrieval.
(a works-with b, b owns c → who owns c).
graph.pyalready walks theseedges; nothing scores whether retrieval uses them.
value ("how many times did x change", "what was it before the change").
each needs its generator to plant the supporting facts and its answer key to
stay a pure function of the seed. these are harder to generate than recall
cases: the generator has to guarantee the fact is only derivable, never
stated, or the category silently degenerates into single-session recall.
Compatibility considerations
adding categories shifts the overall mean, so the same versioning and
season-boundary question as the multipliers issue applies — worth landing both
under one bench version bump rather than two.
expect the new categories to score low at first. that is the point: they mark
where the engine actually is, and they are the ones the koth ladder should be
paying to move.
Alternatives
reproducible, not usable by the ladder.
a real loss, since it is the category where compiled pages beat every
retrieval-only competitor by construction.
part of the ditto-style track — see
.superpowers/DITTO-STYLE-PLAN.md(T6.2)and
.superpowers/BEAT-DITTO-PLAN.md(W3, P2).