Add draft SEC and WRENCH extraction queries

AGENTS.md

@@ -4,6 +4,9 @@ QUAIL-B is an academic benchmark of 85 AI SQL queries over document tables.
 It publishes Substrait query plans, input tables, reference labels, and a
 scoring harness. It does not include an execution engine.
 
+SEC-1 to SEC-7, WRENCH-6, and WRENCH-7 are drafts: they have no reference
+answers yet and are not part of the default 85 queries.
+
 Read `README.md` for the public benchmark contract and metric definitions.
 
 ## Repository map
@@ -13,6 +16,7 @@ Read `README.md` for the public benchmark contract and metric definitions.
 - `quail_b/prompts.py`, `rendering.py`, and `predicates.py`: prompt definitions
   and exact rendered text.
 - `quail_b/data.py` and `labels.py`: published inputs and reference labels.
+- `quail_b/sec.py`: download SEC filings, select the sample, and check the data.
 - `quail_b/run.py`, `scoring.py`, `minimum.py`, and `reporting.py`: adapter
   validation, scoring, token accounting, and result reports.
 - `tests/`: tests for the public contract and benchmark data.

README.md

@@ -448,6 +448,11 @@ compare a filter's answers with the dataset's labels.
 | WRENCH-3 | What kind of exploit did each exploiting run use? | WRENCH-1, then 11 exploit kinds with descriptions, from the dataset's categories |
 | WRENCH-4 | Which runs have the most steps that are part of an exploit? | `ai_filter` over steps, `GROUP BY`, `COUNT`, `HAVING`, `ORDER BY`, `LIMIT` |
 | WRENCH-5 | Among runs not asked to hack, how many exploited the verifier, per agent model? | `WHERE mode = 'baseline'`, `ai_filter`, `GROUP BY`, `COUNT`, `ORDER BY` |
+| WRENCH-6 | Which command or edit exploits the verifier, in each exploiting run? | WRENCH-1, then `ai_extract` |
+| WRENCH-7 | Which file or check does each exploit step tamper with? | `ai_filter` over steps, then `ai_extract` |
+
+WRENCH-6 and WRENCH-7 have no reference answers yet, so they are not part of
+the default 85 queries. Load them by ID with `accuracy=False`.
 
 ### Sales call queries
 
@@ -478,6 +483,55 @@ content can therefore disagree with its deal's stage.
 | SALES-4 | Which calls have the rep offering a discount and the customer committing to buy? | Two `ai_filter`s over one call |
 | SALES-5 | Among calls on deals in negotiation, which 25 most likely end in a commitment? | `WHERE deal_stage = 'Negotiation'`, `ai_score`, `ORDER BY`, `LIMIT` |
 
+### SEC filing queries
+
+The `SEC` queries read 100,000 8-K filings from
+[TeraflopAI/SEC-EDGAR](https://huggingface.co/datasets/TeraflopAI/SEC-EDGAR).
+A company files an 8-K to announce an important event. The sample covers all
+event types, not only CEO changes.
+
+`sec_filings` has one row per filing, with columns `id` (the SEC accession
+number), `text`, `filing_date`, and `filing_month`. A CEO departure is a
+resignation, retirement, removal, or death of the filing company's CEO,
+including an interim or co-CEO. If several CEOs leave, the answers use the
+first one the filing names.
+
+| Query | Question | Operators |
+| --- | --- | --- |
+| SEC-1 | Which filings announce a CEO departure? | `ai_filter` |
+| SEC-2 | Who is the departing CEO in each filing? | `ai_extract`; null without a departure |
+| SEC-3 | When does the CEO leave, in each filing? | `ai_extract`; null without a departure |
+| SEC-4 | Who left, when, and why, in filings that announce a CEO departure? | `ai_filter`, then three `ai_extract`s of one document |
+| SEC-5 | Is each departing CEO's successor interim or permanent? | `ai_filter`, then 3 labels |
+| SEC-6 | How many CEO departures per month and successor status? | SEC-5, then `GROUP BY`, `COUNT`, `ORDER BY` |
+| SEC-7 | Which sentence states the event each filing reports? | `ai_extract` of a long sentence |
+
+The SEC queries have no reference answers yet, so they are not part of the
+default 85 queries. Load them on their own, at `scale_factor=1.0`, with
+`accuracy=False`. `quail_b.run` needs reference answers, so it does not
+accept them. Download the sample to `results/sec/`, then call your adapter
+directly:
+
+```sh
+uv run --frozen python tools/make_sec_corpus.py
+```
+
+```python
+import quail_b
+
+suite = quail_b.load_benchmark(
+    [f"SEC-{i}" for i in range(1, 8)],
+    scale_factor=1.0,
+    data_dir="results/sec",
+    accuracy=False,
+)
+for query in suite.queries:
+    output = my_engine_adapter(query, suite.tables)
+```
+
+[sec.py](quail_b/sec.py) defines the source revision and how the sample is
+chosen. The loader checks the download against its manifest.
+
 ## Scale factors
 
 Scale factors 0.1, 0.5, and 1.0 sample 10%, 50%, and 100% of each dataset's

quail_b/benchmark.py

@@ -27,14 +27,17 @@ def select_queries(only=None, *, scale_factor=0.1) -> tuple[QuerySpec, ...]:
     """Validate a scale factor and return the requested query definitions."""
     if scale_factor not in PUBLISHED_CORPORA:
         raise ValueError("scale factor must be 0.1, 0.5, or 1.0")
-    available = queries()
+    available = queries(include_draft=True)
     ids = list(queries()) if only is None else (
         [only] if isinstance(only, str) else list(only))
     unknown = set(ids) - available.keys()
     if unknown:
         raise ValueError(f"unknown query IDs: {sorted(unknown)}")
     if not ids or len(ids) != len(set(ids)):
         raise ValueError("query IDs must be nonempty and unique")
+    sec = [query_id for query_id in ids if query_id.startswith("SEC-")]
+    if sec and (len(sec) != len(ids) or scale_factor != 1.0):
+        raise ValueError("run SEC queries separately at scale_factor=1.0")
     if isinstance(only, (set, frozenset)):
         ids = [query_id for query_id in available if query_id in only]
     return tuple(available[query_id] for query_id in ids)
@@ -53,17 +56,19 @@ class Benchmark:
 
 def load_benchmark(only=None, *, scale_factor=0.1, data_dir=None,
                    collection_id=None, accuracy=True, root=None) -> Benchmark:
-    """Load selected inputs and labels from the public benchmark.
+    """Load selected inputs and labels, including explicit local SEC inputs.
 
     Args:
         only: Query IDs, or None for all queries.
-        scale_factor: Published scale factor: 0.1, 0.5, or 1.0.
-        data_dir: Optional directory of input Parquet files to validate.
+        scale_factor: Published scale factor: 0.1, 0.5, or 1.0. SEC uses 1.0.
+        data_dir: Directory of input files to validate. Required for SEC.
         collection_id: Reference collection ID, or None for the active one.
         accuracy: Whether to load reference labels for scoring.
         root: Published data root, or None for the public S3 bucket.
     """
     specs = select_queries(only, scale_factor=scale_factor)
+    if specs[0].id.startswith("SEC-"):
+        return _load_sec_benchmark(specs, data_dir, collection_id, accuracy, root)
     corpus_id = PUBLISHED_CORPORA[scale_factor]
     manifest = _read_json(
         root, f"{GROUND_TRUTH_ROOT}/corpora/{corpus_id}/manifest.json")
@@ -89,6 +94,31 @@ def read(name):
     for name in names:
         if identity["tables"][name] != manifest["tables"][name]:
             raise ValueError(f"{name} does not match published corpus {corpus_id}")
+    truth = _load_truth(specs, scale_factor, corpus_id,
+                        collection_id, accuracy, root)
+    return Benchmark(specs, tables, scale_factor, corpus_id, truth)
+
+
+def _load_sec_benchmark(specs, data_dir, collection_id, accuracy, root):
+    from quail_b.sec import SEC_TABLE, load_sec_table, sec_identity
+
+    if data_dir is None:
+        raise ValueError(
+            "SEC needs data_dir from tools/make_sec_corpus.py")
+    if accuracy and collection_id is None:
+        raise ValueError(
+            "SEC reference labels are not published; use accuracy=False "
+            "for adapter inputs or provide collection_id")
+    table = load_sec_table(data_dir)
+    tables = {SEC_TABLE: table}
+    corpus_id = sec_identity(table)["corpus_id"]
+    truth = _load_truth(specs, 1.0, corpus_id,
+                        collection_id, accuracy, root)
+    return Benchmark(specs, tables, 1.0, corpus_id, truth)
+
+
+def _load_truth(specs, scale_factor, corpus_id,
+                collection_id, accuracy, root):
     truth = None
     if accuracy:
         # only the label sets the selected queries score against
@@ -117,4 +147,4 @@ def read(name):
                 raise ValueError(
                     f"reference labels for {key} use a different prompt format; "
                     "regenerate labels for the current benchmark")
-    return Benchmark(specs, tables, scale_factor, corpus_id, truth)
+    return truth

quail_b/plans/SEC-1.json

@@ -0,0 +1,134 @@
+{
+  "extension_urns": [
+    {
+      "extension_urn_anchor": 1,
+      "urn": "extension:org.fsdatalab.quail_b:functions_ai"
+    }
+  ],
+  "extensions": [
+    {
+      "extension_function": {
+        "extension_urn_reference": 1,
+        "function_anchor": 1,
+        "name": "ai_filter:str_str"
+      }
+    }
+  ],
+  "relations": [
+    {
+      "root": {
+        "input": {
+          "project": {
+            "common": {
+              "emit": {
+                "output_mapping": [
+                  3
+                ]
+              }
+            },
+            "expressions": [
+              {
+                "selection": {
+                  "direct_reference": {
+                    "struct_field": {}
+                  },
+                  "root_reference": {}
+                }
+              }
+            ],
+            "input": {
+              "filter": {
+                "common": {
+                  "direct": {},
+                  "hint": {
+                    "alias": "filter-1"
+                  }
+                },
+                "condition": {
+                  "scalar_function": {
+                    "arguments": [
+                      {
+                        "value": {
+                          "literal": {
+                            "string": "{0}\n\nDoes this filing announce a CEO departure? A CEO departure is an announced resignation, retirement, removal, death, or other end of service as the filing company's chief executive officer, including an interim or co-CEO, effective now or later. Hypothetical departures, past departures given as background, and departures from other roles or other companies do not count."
+                          }
+                        }
+                      },
+                      {
+                        "value": {
+                          "selection": {
+                            "direct_reference": {
+                              "struct_field": {
+                                "field": 1
+                              }
+                            },
+                            "root_reference": {}
+                          }
+                        }
+                      }
+                    ],
+                    "function_reference": 1,
+                    "output_type": {
+                      "bool": {
+                        "nullability": "NULLABILITY_REQUIRED"
+                      }
+                    }
+                  }
+                },
+                "input": {
+                  "read": {
+                    "base_schema": {
+                      "names": [
+                        "id",
+                        "text",
+                        "filing_month"
+                      ],
+                      "struct": {
+                        "nullability": "NULLABILITY_REQUIRED",
+                        "types": [
+                          {
+                            "string": {
+                              "nullability": "NULLABILITY_REQUIRED"
+                            }
+                          },
+                          {
+                            "string": {
+                              "nullability": "NULLABILITY_REQUIRED"
+                            }
+                          },
+                          {
+                            "string": {
+                              "nullability": "NULLABILITY_REQUIRED"
+                            }
+                          }
+                        ]
+                      }
+                    },
+                    "common": {
+                      "direct": {},
+                      "hint": {
+                        "alias": "f"
+                      }
+                    },
+                    "named_table": {
+                      "names": [
+                        "sec_filings"
+                      ]
+                    }
+                  }
+                }
+              }
+            }
+          }
+        },
+        "names": [
+          "f"
+        ]
+      }
+    }
+  ],
+  "version": {
+    "minor_number": 103,
+    "producer": "quail-b"
+  }
+}