Screen the opener vocabulary before folding it, and store what it settles

[ahernsean]

Summary

The leaderboard folded all 14,855 candidate openers on every build, constructing
a per-group state dict for each of ~1.4 million response groups in order to
discover that 13,983 of them are still being searched. Measured on the
production cache at 872 completed openers, one build took 25.3s against a
client that polls every 2s. That is why collect_leaderboard_once's stampede
lock exists.

Screen before folding. ScoreCache.report_reusable_branch_facts decides the
reusability gate in SQL and returns the qualifying keys with three columns,
instead of loading every row of the policy with six and filtering in Python
(4.5s to 0.9s on the live cache). _screen_and_fold_openers then reaches
_candidate_erd_summary only for the openers a fold can settle. The result is
3.0s, returning the same 872 rows byte-identical to the shipped collector.

The screen visits every group rather than stopping at the first unsettled one. A
candidate holding both an unsettled group and a proven loss is infeasible and
not pending, because _candidate_erd_summary decides infeasibility ahead of
pendency, and an early exit can return before reaching the loss that decides it.
Early exit measured 0.13s against 1.0s for the full scan; against a 0.9s load
the full scan is worth its cost. Groups of fewer than two answers hold no branch
result and never will, so the screen skips them instead of asking the cache.

opener_erd_by_policy stores each completed opener's fold. The invalidation
problem that retired candidate_erd_by_policy does not apply at this key: the
table is bounded at one row per candidate word rather than at every (branch,
candidate) pair, which is what makes it affordable to rescreen the whole set on
every build instead of trusting a stored row. _store_opener_folds writes the
folds the screen settled and deletes the rows whose openers it no longer
settles, so a repair or requeue that removes a branch result removes the folds
that read it. Nothing reads a stored fold in place of folding. A row is
rewritten only when its value changed, because the build runs on a poll against
the cache the swarm is writing into. The table is local to each machine and
travels in neither EXPORT_TABLES nor TABLES.

candidate_erd_by_policy is now dropped on every writable open rather than
once behind a migration flag. A process running code from before the table was
removed recreates it, and a migration already recorded as done never looks
again. That is not hypothetical: the production cache carried the table with its
original schema, empty, two weeks after drop_candidate_erd_memo completed. The
check is a sqlite_master lookup, so carrying it permanently costs one indexed
read per open.

AGENTS.md's "a candidate's own ERD is derived, never stored" section is scoped
rather than removed: the general rule still holds for a candidate's ERD at an
arbitrary branch, and the section now states what makes the opener fold
different and what keeps it honest.

Schema

opener_erd_by_policy is created by an idempotent CREATE TABLE IF NOT EXISTS
in _ensure_schema. It is additive, so the running swarm's pre-change workers
are unaffected, and it is outside the five tables that cross to the phone, so it
needs no deploy sequencing.

[ahernsean]

Review. @codex

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 403801b31d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".