Add draft SEC and WRENCH extraction queries

[shreyashankar]

Add nine draft queries: SEC-1 to SEC-7 over 100,000 8-K filings, and WRENCH-6 and WRENCH-7 over the Terminal Wrench runs. A draft query has a plan but no reference answers. It is not part of the default 85 queries, and you load it by ID with accuracy=False.

Query	Question	Operators
SEC-1	Which filings announce a CEO departure?	ai_filter
SEC-2	Who is the departing CEO in each filing?	ai_extract
SEC-3	When does the CEO leave, in each filing?	ai_extract
SEC-4	Who left, when, and why?	ai_filter, then three ai_extracts
SEC-5	Is the successor interim or permanent?	ai_filter, then 3 labels
SEC-6	How many departures per month and successor status?	SEC-5, then GROUP BY, COUNT
SEC-7	Which sentence states the event each filing reports?	ai_extract of a long sentence
WRENCH-6	Which command or edit exploits the verifier, in each exploiting run?	WRENCH-1, then ai_extract
WRENCH-7	Which file or check does each exploit step tamper with?	WRENCH-4's step filter, then ai_extract

SEC inputs

The SEC filings are not in the published corpus. tools/make_sec_corpus.py downloads them from TeraflopAI/SEC-EDGAR at a pinned revision and selects a fixed sample across all 217 source files. The loader checks the sample against its manifest. SEC queries run only on their own, at scale_factor=1.0. WRENCH-6 and WRENCH-7 use the published Terminal Wrench tables at any scale factor.

How drafts work

• A draft flag in the catalog keeps a query out of queries() but lets get_query and load_benchmark find it by ID.
• Each draft extraction has a predicate marked unpublished, so asking for its reference source raises an error.
• With accuracy=True and no labels, load_benchmark raises an error that names accuracy=False.

Checks

• Ruff, the long-string check, Vulture, and all 147 tests pass. The tests cover sampling, manifest checks, plan structure, aggregation, and scoring against synthetic labels.
• SEC-1 to SEC-6 loaded against the downloaded 100,000-filing sample, and its corpus identity matched. SEC-7 was added after that check.
• SEC-7 ran through Quail on the first 1,000 filings with an equivalent ad hoc prompt, not the published plan. 999 of them had an answer. The median answer was 24 tokens, and the 90th percentile 53.
• WRENCH-6 and WRENCH-7 have not run on a model yet.

Quail still needs to move its quail_b pin and add a runner path for unscored queries.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PtZ4CLzq12XSVfZos1Gc9R