fix(eval): commit each published score to the corpus a reader can download (#9811)

Verified live on api.loopover.ai: /v1/public/stats publishes
ai_consensus_defect at decided 460 / confirmed 287, /v1/public/eval-corpus
serves 460 real cases whose checksum re-derives from the downloaded bytes, and
/v1/public/eval-scores serves `{"records":[]}`.

The commitment came only from a persisted calibration.*_backtest_run audit
event. #9639 fixes that writer, but it runs inside a review pass, and hosted
review execution is retired (src/index.ts acks-and-drops review-execution jobs
off the queue), so on loopover.ai the event never exists. The walkthrough tells
an anonymous reader to fetch .records and re-derive recordDigest; they got an
empty array, explained by a doc line attributing it to an empty corpus that
demonstrably is not empty.

Each record now falls back to its OWN rule's published corpus checksum -- the
exact bytes /v1/public/eval-corpus serves, over the same window. That is not the
placeholder commitment #9215 forbids: it is a hash over an artifact the reader
downloads and re-hashes, which is what the `reproducible` tier asserts. A
persisted run still wins where one exists, so self-host is unchanged.

Resolving per rule also fixes a latent bug: one run's checksum was stamped onto
every record, so with more than one published rule every record but one would
have committed to a different rule's cases.

TRUNCATION HAD TO BE MADE DETECTABLE FIRST. PUBLIC_EVAL_CORPUS_MAX_CASES was
5_000 while the corpus is built from a rule-history read that
listAuditEventsByType hard-clamps to 2_000 -- and queryRuleHistory passed no
limit at all, so it took the default of 500. The cap could never bind, and
`truncated` was structurally always false, while /v1/public/stats counts
`decided` with an unbounded SQL COUNT(*). The two surfaces agreed only while the
window stayed under 500 cases; at 460 they were 40 from silently diverging, with
the corpus serving a prefix and reporting completeness. queryRuleHistory now
takes an explicit bound and reports `saturated`, the corpus ORs that into
`truncated`, and a truncated corpus is never published as a commitment -- its
checksum would cover a prefix of the window the score covers.

Closes #9805

Author: JSONbored

apps/loopover-ui/content/docs/verify-this-review.mdx

@@ -138,11 +138,28 @@ transport — recompute `recordDigest` over the record's own remaining fields an
 curl -s "https://api.loopover.ai/v1/public/eval-scores" | jq '.records'
 ```
 
-That array is empty whenever the latest backtest run's corpus is empty. A checksum over zero cases
-is byte-identical for every rule and every window, so it commits to nothing a reader could re-derive
-the scores from — publishing a record against it would assert a reproducibility that does not exist.
-An empty `records` array therefore means *the numbers are not currently committed to a corpus*, never
-that the numbers are zero.
+Each record's `commitments.corpusChecksum` is the checksum of the corpus you downloaded in step 2 —
+the same rule, the same window, the same bytes. So you can close the loop yourself: hash the corpus,
+read the record, compare.
+
+```bash
+curl -s "https://api.loopover.ai/v1/public/eval-corpus?ruleId=ai_consensus_defect" | jq -r '.checksum'
+curl -s "https://api.loopover.ai/v1/public/eval-scores" \
+  | jq -r '.records[] | select(.workUnit.ruleId == "ai_consensus_defect") | .commitments.corpusChecksum'
+```
+
+A rule is **omitted** from `records` — rather than published with a commitment you could not check —
+in exactly three cases:
+
+- **its corpus is empty.** A checksum over zero cases is byte-identical for every rule, every window
+  and every deployment, so it commits to nothing you could re-derive the scores from;
+- **its corpus is truncated** (`truncated: true` in step 2). The checksum would then cover a prefix of
+  the window while the record's `decided`/`confirmed` cover all of it, so re-deriving from the
+  published cases would give you different numbers than the ones published;
+- **the deployment has neither a persisted backtest run nor a usable corpus** for that rule.
+
+An empty `records` array therefore means *these numbers are not currently committed to a corpus* —
+never that the numbers are zero.
 
 The aggregated run history is readable directly too, again against the deployment's own database
 (operator / self-host):

packages/loopover-engine/src/calibration/signal-tracking.ts

@@ -49,7 +49,21 @@ export interface SignalStore {
   recordHumanOverride(event: HumanOverrideEvent): Promise<void>;
   /** Every fired + override event for `ruleId` at or after `sinceMs` (epoch millis), oldest first. A host MAY
    *  scope this further (e.g. to one repo) internally; the interface itself is unscoped beyond `ruleId`. */
-  queryRuleHistory(ruleId: string, sinceMs: number): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[] }>;
+  /**
+   * Read one rule's history since `sinceMs`.
+   *
+   * `limit` bounds EACH of the two reads (#9805). It is part of the interface rather than an implementation
+   * detail because a caller that publishes the result has to know whether it saw the whole window --
+   * /v1/public/eval-corpus reported `truncated: false` over a read it could not have completed, precisely
+   * because the bound was invisible from here.
+   *
+   * `saturated` is true when either read came back AT its bound, i.e. rows were almost certainly left behind.
+   */
+  queryRuleHistory(
+    ruleId: string,
+    sinceMs: number,
+    limit?: number,
+  ): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[]; saturated: boolean }>;
 }
 
 /** Per-rule confusion-style report over a window: how many times it fired, how many of those got an explicit

packages/loopover-miner/lib/signal-tracking-store.ts

@@ -100,7 +100,12 @@ export function createSignalTrackingStore(eventLedger: SignalTrackingLedger): Si
         payload: toHumanOverridePayload(event),
       });
     },
-    async queryRuleHistory(ruleId: string, sinceMs: number): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[] }> {
+    // #9805: the interface carries a `limit`/`saturated` pair so a publishing caller can tell a complete read
+    // from a truncated one. This store reads the WHOLE local event ledger in memory -- there is no bound to
+    // hit and nothing is ever left behind -- so `limit` is accepted for interface conformance and ignored,
+    // and `saturated` is unconditionally false. That is a true statement here, not a stub: reporting `true`
+    // would claim missing rows that do not exist.
+    async queryRuleHistory(ruleId: string, sinceMs: number): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[]; saturated: boolean }> {
       const sinceIso = new Date(sinceMs).toISOString();
       const fired: RuleFiredEvent[] = [];
       const overrides: HumanOverrideEvent[] = [];
@@ -127,7 +132,7 @@ export function createSignalTrackingStore(eventLedger: SignalTrackingLedger): Si
           });
         }
       }
-      return { fired, overrides };
+      return { fired, overrides, saturated: false };
     },
   };
 }

src/api/routes.ts

@@ -304,6 +304,7 @@ import { isFairnessAnalyticsEnabled, resolveFairnessAnalyticsManifestOverride }
 import { isRagEnabled } from "../review/rag-wire";
 import { loadDecisionLedgerTip, loadPublicDecisionRecord, loadPublicLedgerRow, verifyDecisionLedger } from "../review/decision-record";
 import { buildEvalScoreRecordsFromRulePrecision, filterEvalScoreRecords } from "../review/eval-score-records";
+import { buildPublicCorpusCommitments } from "../review/public-eval-corpus";
 import { anchorSigningInput, buildLedgerAnchorPayload, currentAnchorKey, parseAnchorPublicKeys, publicAnchorStatus, signLedgerAnchorPayload } from "../review/ledger-anchor";
 import { resolveProofPage } from "../review/proof-summary";
 import { renderProofBadgeSvg } from "./proof-badge";
@@ -905,7 +906,13 @@ export function createApp() {
     // IO-touching source (the benchmark_run records from #9265) is exactly where real error handling belongs,
     // added when that source actually exists, not as untestable defensive code here.
     const precision = await loadPublicRulePrecision(c.env);
-    const records = await buildEvalScoreRecordsFromRulePrecision(precision, new Date().toISOString());
+    // #9805: the per-rule fallback commitment, when no backtest run is persisted. Loaded HERE rather than
+    // inside the record builder so that module stays pure -- and loaded through the same
+    // loadPublicEvalCorpus the /v1/public/eval-corpus route serves, so the checksum a record commits to is by
+    // construction the one a reader re-derives from the bytes they downloaded, not a parallel computation
+    // that could drift from it.
+    const corpusChecksumByRuleId = await buildPublicCorpusCommitments(c.env, precision.rules.map((rule) => rule.ruleId));
+    const records = await buildEvalScoreRecordsFromRulePrecision(precision, new Date().toISOString(), corpusChecksumByRuleId);
     const filtered = filterEvalScoreRecords(records, {
       subject: c.req.query("subject"),
       since: c.req.query("since"),

src/review/eval-score-records.ts

@@ -76,6 +76,18 @@ async function finalizeRecord(input: EvalScoreRecordDigestInput): Promise<EvalSc
  * a record whose commitments cannot be independently re-derived (no corpus checksum to point at) is not
  * publishable, so this deliberately emits nothing rather than a record with a placeholder commitment.
  *
+ * #9805: when no backtest run is persisted, the commitment falls back to `corpusChecksumByRuleId` -- the
+ * checksum of the corpus `/v1/public/eval-corpus` publishes for that same rule, over the same window. That is
+ * not a placeholder standing in for a real commitment: it is a hash over an artifact the reader can download
+ * and re-hash themselves, which is exactly what the `reproducible` trust tier asserts. It exists because a
+ * deployment with review execution retired (the hosted Worker: see src/index.ts) never persists a backtest
+ * run at all, so the entire surface was empty while a complete, downloadable corpus sat behind the next
+ * endpoint over.
+ *
+ * The commitment is resolved PER RULE, not once for the whole batch. Each rule's score is computed over its
+ * own corpus, so stamping one checksum across every record would have every record but one committing to a
+ * different rule's cases -- latent today only because a single rule clears the publication floor.
+ *
  * Also returns an empty array when the run's checksum is {@link EMPTY_CORPUS_CHECKSUM}. A hash over zero
  * cases is the same 32 bytes for every rule, every window, and every deployment, so it points at nothing a
  * consumer could re-derive the scores from -- it is a placeholder commitment wearing a real hash's clothes,
@@ -90,14 +102,31 @@ async function finalizeRecord(input: EvalScoreRecordDigestInput): Promise<EvalSc
  * concept here, so `0` is the correct value, not a masked null). PURE -- no IO, no clock (the caller
  * supplies `issuedAt`).
  */
-export async function buildEvalScoreRecordsFromRulePrecision(precision: PublicRulePrecision, issuedAt: string): Promise<EvalScoreRecord[]> {
-  if (!precision.latestBacktestRun) return [];
-  const { corpusChecksum } = precision.latestBacktestRun;
-  if (corpusChecksum === EMPTY_CORPUS_CHECKSUM) return [];
+export async function buildEvalScoreRecordsFromRulePrecision(
+  precision: PublicRulePrecision,
+  issuedAt: string,
+  // #9805: per-rule fallback commitments, supplied by the caller so this module stays PURE. Only rules whose
+  // published corpus is a usable commitment belong in here -- the route drops empty and truncated ones before
+  // building it, because a truncated corpus's checksum covers a subset of the cases the score covers.
+  corpusChecksumByRuleId: ReadonlyMap<string, string> = new Map(),
+): Promise<EvalScoreRecord[]> {
+  // A persisted backtest run still wins where one exists, so a deployment that executes reviews keeps exactly
+  // today's behaviour and this change cannot silently move a self-host commitment.
+  const runChecksum =
+    precision.latestBacktestRun && precision.latestBacktestRun.corpusChecksum !== EMPTY_CORPUS_CHECKSUM
+      ? precision.latestBacktestRun.corpusChecksum
+      : null;
   const windowStart = new Date(Date.parse(issuedAt) - precision.windowDays * 24 * 60 * 60 * 1000).toISOString();
 
+  // A rule with no usable commitment is OMITTED rather than published with a placeholder -- the #9215
+  // requirement this module has always enforced, now applied per rule instead of to the whole batch.
+  const publishable = precision.rules.flatMap((row) => {
+    const corpusChecksum = runChecksum ?? corpusChecksumByRuleId.get(row.ruleId) ?? null;
+    return corpusChecksum === null || corpusChecksum === EMPTY_CORPUS_CHECKSUM ? [] : [{ row, corpusChecksum }];
+  });
+
   const records = await Promise.all(
-    precision.rules.map((row) =>
+    publishable.map(({ row, corpusChecksum }) =>
       finalizeRecord({
         schemaVersion: EVAL_SCORE_RECORD_SCHEMA_VERSION,
         subject: { kind: "agent", id: ORB_GATE_SUBJECT_ID },

src/review/public-eval-corpus.ts

@@ -29,12 +29,18 @@
 import { buildBacktestCorpus } from "@loopover/engine/calibration/backtest-corpus";
 import { canonicalJson, sha256Hex } from "./decision-record";
 import { NON_ATTRIBUTABLE_OVERRIDE_PROVENANCES, PUBLIC_PRECISION_WINDOW_DAYS } from "./public-rule-precision";
-import { createSignalStore } from "./signal-tracking-wire";
+import { createSignalStore, MAX_RULE_HISTORY_LIMIT } from "./signal-tracking-wire";
 
 /** Hard cap on published cases. The window is already bounded, but an unbounded array on an
  *  unauthenticated route is a footgun the moment a rule gets noisy; a truncated corpus is reported
  *  honestly via `truncated` rather than silently trimmed. */
-export const PUBLIC_EVAL_CORPUS_MAX_CASES = 5_000;
+// #9805: was 5_000, which could never bind. The corpus is built from a rule-history read that
+// listAuditEventsByType hard-clamps to MAX_RULE_HISTORY_LIMIT rows, so a cap above that ceiling is
+// unreachable and `truncated` was structurally always false -- while /v1/public/stats counts `decided` with
+// an UNBOUNDED SQL COUNT(*). The two surfaces therefore agreed only while the window stayed under the read
+// bound, and would have silently diverged after that: a complete-looking corpus serving a prefix of the
+// cases the published precision was computed over. Pinned to the real ceiling so the cap and the read agree.
+export const PUBLIC_EVAL_CORPUS_MAX_CASES = MAX_RULE_HISTORY_LIMIT;
 
 /** One published case: a {@link BacktestCase} minus `targetKey`, with `metadata` narrowed to the single
  *  key the shipped classifier reads. `metadata` is omitted entirely (never `undefined`) when the firing
@@ -124,8 +130,12 @@ export async function loadPublicEvalCorpus(env: Env, ruleId: string, nowMs: numb
   const sinceMs = nowMs - PUBLIC_PRECISION_WINDOW_DAYS * 24 * 60 * 60 * 1000;
   let fired: Awaited<ReturnType<ReturnType<typeof createSignalStore>["queryRuleHistory"]>>["fired"] = [];
   let overrides: Awaited<ReturnType<ReturnType<typeof createSignalStore>["queryRuleHistory"]>>["overrides"] = [];
+  // A read that came back at its bound almost certainly left rows behind, and a corpus that silently omits
+  // them must say so -- committing a published score to a prefix, while claiming completeness, is exactly the
+  // unverifiable-artifact problem this endpoint exists to solve.
+  let saturated = false;
   try {
-    ({ fired, overrides } = await createSignalStore(env).queryRuleHistory(ruleId, sinceMs));
+    ({ fired, overrides, saturated } = await createSignalStore(env).queryRuleHistory(ruleId, sinceMs, MAX_RULE_HISTORY_LIMIT));
   } catch {
     // Fall through to an empty corpus rather than 500ing an unauthenticated route.
   }
@@ -144,8 +154,42 @@ export async function loadPublicEvalCorpus(env: Env, ruleId: string, nowMs: numb
     ruleId,
     windowDays: PUBLIC_PRECISION_WINDOW_DAYS,
     caseCount: cases.length,
-    truncated,
+    // Either bound truncates: the cap on the built cases, or the read that fed it. Reporting only the former
+    // is what made this field always-false.
+    truncated: truncated || saturated,
     checksum: await checksumPublicEvalCorpus(cases),
     cases,
   };
 }
+
+/**
+ * #9805: the publishable commitment for each of `ruleIds` -- the checksum of the corpus this deployment
+ * serves at `/v1/public/eval-corpus?ruleId=...`, for rules whose corpus can actually back a claim.
+ *
+ * A rule is OMITTED (rather than mapped to a checksum a reader would be misled by) when:
+ *
+ *   • the corpus is empty -- `checksumPublicEvalCorpus([])` is the same 32 bytes for every rule, every
+ *     window and every deployment, so it commits to nothing re-derivable. This is also where a failed read
+ *     lands, since loadPublicEvalCorpus degrades to an empty corpus rather than throwing a public route;
+ *   • the corpus is TRUNCATED at PUBLIC_EVAL_CORPUS_MAX_CASES -- the checksum would then cover a prefix of
+ *     the window while the record's `decided`/`confirmed` cover all of it. A reader who re-derived scores
+ *     from the published cases would get different numbers and reasonably conclude the published ones were
+ *     wrong. Omitting the record says "not committed"; publishing it would say something false.
+ *
+ * Sequential rather than Promise.all: each call is its own D1 read over a 90-day window, and the rule list
+ * is the handful that clear the publication floor -- fanning them out buys nothing and makes the read
+ * burst on an unauthenticated route.
+ */
+export async function buildPublicCorpusCommitments(
+  env: Env,
+  ruleIds: readonly string[],
+  nowMs: number = Date.now(),
+): Promise<Map<string, string>> {
+  const commitments = new Map<string, string>();
+  for (const ruleId of ruleIds) {
+    const corpus = await loadPublicEvalCorpus(env, ruleId, nowMs);
+    if (corpus.caseCount === 0 || corpus.truncated) continue;
+    commitments.set(ruleId, corpus.checksum);
+  }
+  return commitments;
+}

src/review/signal-tracking-wire.ts

@@ -69,6 +69,15 @@ function toHumanOverrideEvent(ruleId: string, row: { targetKey: string | null; m
  *  are NOT fail-open the same way: a read error propagates, since a caller computing a precision report needs
  *  to know its input is incomplete rather than silently scoring against a partial (possibly empty) history.
  */
+/** listAuditEventsByType's own default, restated so callers that do not pass a limit keep today's behaviour
+ *  explicitly rather than by inheritance (#9805). */
+export const DEFAULT_RULE_HISTORY_LIMIT = 500;
+
+/** The most rows one queryRuleHistory read can return: listAuditEventsByType hard-clamps its limit to this,
+ *  so asking for more silently yields this many. Anything built on top of a rule-history read is bounded by
+ *  it, and a cap declared ABOVE it can never be the thing that actually truncates. */
+export const MAX_RULE_HISTORY_LIMIT = 2_000;
+
 export function createSignalStore(env: Env): SignalStore {
   return {
     async recordRuleFired(event: RuleFiredEvent): Promise<void> {
@@ -93,15 +102,28 @@ export function createSignalStore(env: Env): SignalStore {
         createdAt: event.occurredAt || nowIso(),
       }).catch(() => undefined);
     },
-    async queryRuleHistory(ruleId: string, sinceMs: number): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[] }> {
+    // #9805: `limit` is explicit rather than left to listAuditEventsByType's default of 500. The published
+    // corpus needs to know whether it saw the WHOLE window, and a caller that cannot choose the bound cannot
+    // tell a complete read from a truncated one. Defaulted so every existing caller is byte-identical.
+    //
+    // `saturated` is the honest signal: the row count came back exactly at the bound, so there are almost
+    // certainly more rows the caller did not see. It is deliberately not "did we hit MAX_CASES" -- the read
+    // bound is what actually limits the corpus, and conflating the two is how /v1/public/eval-corpus came to
+    // report `truncated: false` over a read it could not have completed.
+    async queryRuleHistory(
+      ruleId: string,
+      sinceMs: number,
+      limit: number = DEFAULT_RULE_HISTORY_LIMIT,
+    ): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[]; saturated: boolean }> {
       const sinceIso = new Date(sinceMs).toISOString();
       const [firedRows, overrideRows] = await Promise.all([
-        listAuditEventsByType(env, ruleFiredEventType(ruleId), sinceIso),
-        listAuditEventsByType(env, humanOverrideEventType(ruleId), sinceIso),
+        listAuditEventsByType(env, ruleFiredEventType(ruleId), sinceIso, limit),
+        listAuditEventsByType(env, humanOverrideEventType(ruleId), sinceIso, limit),
       ]);
       return {
         fired: firedRows.map((row) => toRuleFiredEvent(ruleId, row)),
         overrides: overrideRows.map((row) => toHumanOverrideEvent(ruleId, row)),
+        saturated: firedRows.length >= limit || overrideRows.length >= limit,
       };
     },
   };