From 29461aa50b49ad8720b368da09d15f66a7ac869d Mon Sep 17 00:00:00 2001 From: JSONbored <49853598+JSONbored@users.noreply.github.com> Date: Tue, 28 Jul 2026 19:06:20 -0700 Subject: [PATCH] fix(review): stop publishing a reproducibility freeze point that commits to an empty corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `/v1/public/stats` reported a `latestBacktestRun.corpusChecksum` of 4f53cda1…, which is `checksumCases([])` — SHA-256 over the canonicalized empty case list. The fairness page rendered it as a "Reproducibility freeze point", and `/v1/public/eval-scores` published two records committed to it with `trust.tier: "reproducible"` alongside decided=460/confirmed=287. A hash over zero cases is the same 32 bytes for every rule, every window and every deployment, so it points at nothing a skeptic can re-derive anything from. Treat it the same way as a missing run: `loadPublicRulePrecision` now reports `latestBacktestRun: null` for it, which clears the freeze point from the fairness page and, through the builder's existing guard, empties the eval-scores response. The record builder additionally refuses the value on its own, since it is the code that stamps the `reproducible` tier and should not depend on its caller to be honest. The scores are unaffected — they come from a different dataset (`signal.human_override:*` audit events), so an empty corpus never means the numbers are zero, only that they are uncommitted, which is exactly the state the record spec says must not be published. Docs: the walkthrough's step 4 referenced a bare `/v1/public/stats`, which resolves against loopover.ai and 404s — only the API host serves it. Fetch it absolutely, matching the convention in what-you-can-verify.mdx. Step 1 and the track-record command shell out to `wrangler d1 execute --remote` against the deployment's own database, so they need that deployment's Cloudflare credentials and are not runnable by a third party; each step is now marked with who can actually run it instead of the page claiming nothing needs a key. The freeze point covers whichever single rule the latest run backtested, not every rule on the report, and is described that way now. --- .../content/docs/verify-this-review.mdx | 64 +++++++++++++++---- .../content/docs/what-you-can-verify.mdx | 5 +- src/review/eval-score-records.ts | 12 +++- src/review/public-rule-precision.ts | 18 +++++- test/unit/eval-score-records.test.ts | 22 ++++++- test/unit/public-rule-precision.test.ts | 29 +++++++++ 6 files changed, 131 insertions(+), 19 deletions(-) diff --git a/apps/loopover-ui/content/docs/verify-this-review.mdx b/apps/loopover-ui/content/docs/verify-this-review.mdx index da39dd3777..1cace0e727 100644 --- a/apps/loopover-ui/content/docs/verify-this-review.mdx +++ b/apps/loopover-ui/content/docs/verify-this-review.mdx @@ -11,16 +11,27 @@ website only build trust if a skeptic can check them without asking anyone's per page is the end-to-end walkthrough: export the same corpus snapshot the numbers come from, verify its checksum, replay the same scorer over it, and compare what you get against what is published. -Everything below runs read-only against a database export and pure functions from -`@loopover/engine`. Nothing posts anywhere, nothing needs an API key. +Everything below runs read-only against a corpus export and pure functions from `@loopover/engine`. +Nothing posts anywhere, and nothing needs a LoopOver API key. + + + **Step 1 is not yet runnable by a stranger.** Exporting the corpus reads the deployment's own + database — with `--remote` that means the operator's Cloudflare D1, which needs *their* Cloudflare + credentials, not a LoopOver key. So today steps 1–3 are reproducible by an operator or by anyone + running a self-host deployment against their own data, and step 4 is the part a third party can + check unauthenticated. No anonymous corpus download exists yet; until one does, this page marks + each step with who can actually run it rather than implying everyone can run all of them. + For the wider contract — every claim, its artifact, and its trust assumption, including the ones you cannot check — see [what you can verify](/docs/what-you-can-verify). -## 1. Export the corpus snapshot +## 1. Export the corpus snapshot *(operator / self-host)* Every rule's fired/override history exports as a versioned, checksummed JSON snapshot -([backtest & calibration](/docs/backtest-calibration) explains how that history is recorded): +([backtest & calibration](/docs/backtest-calibration) explains how that history is recorded). +`--remote` shells out to `wrangler d1 execute … --remote`, so it reads the deployed database and +requires that deployment's own Cloudflare credentials: ```bash npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --remote @@ -34,10 +45,12 @@ npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch The snapshot's `checksum` field is a SHA-256 over the canonicalized cases (keys sorted, so property order can never change the hash). The fairness report's *reproducibility freeze point* -shows the checksum of the corpus behind the latest persisted backtest run — an export of the same -window reproduces the same checksum, byte for byte. +shows that checksum for the **most recent persisted backtest run**, whichever single rule that run +covered — re-exporting that rule over the same window reproduces the same checksum, byte for byte. +It is one run's freeze point, not a per-rule commitment for every rule on the report, so compare it +against an export of the rule the run actually covered. -## 2. Verify the checksum +## 2. Verify the checksum *(anyone, given a snapshot)* The manifest is self-verifying: recompute the hash over its own `cases` array and compare it to the recorded `checksum`. The canonicalization lives in `scripts/backtest-corpus-export-core.ts` @@ -53,7 +66,7 @@ console.log(recomputed.checksum === saved.checksum ? "checksum OK" : "CHECKSUM M ' ``` -## 3. Replay the scorer +## 3. Replay the scorer *(anyone, given a snapshot)* The published precision comes from the same pure functions any Node script can import: `scoreBacktest` replays a classifier over the labeled cases, and `compareBacktestScores` applies @@ -75,19 +88,42 @@ console.log(report); - **`null` is never `0`.** Precision and recall stay `null` below the decided-sample floor; the fairness report renders that as *insufficient data*, never as a zero. -## 4. Compare against the published numbers +## 4. Compare against the published numbers *(anyone)* + +The [fairness report](/fairness) renders each rule's decided-case count and measured precision from +the public stats endpoint's `rulePrecision` block. That endpoint is served by the API host, not by +this site, so fetch it absolutely — a bare `/v1/public/stats` resolves against `loopover.ai` and +404s: + +```bash +curl -s "https://api.loopover.ai/v1/public/stats" | jq '.rulePrecision' +``` + +The same per-rule numbers are also published as digest-committed +[EvalScoreRecords](/docs/what-you-can-verify), each independently re-derivable without trusting the +transport — recompute `recordDigest` over the record's own remaining fields and compare: + +```bash +curl -s "https://api.loopover.ai/v1/public/eval-scores" | jq '.records' +``` + +That array is empty whenever the latest backtest run's corpus is empty. A checksum over zero cases +is byte-identical for every rule and every window, so it commits to nothing a reader could re-derive +the scores from — publishing a record against it would assert a reproducibility that does not exist. +An empty `records` array therefore means *the numbers are not currently committed to a corpus*, never +that the numbers are zero. -The [fairness report](/fairness) renders each rule's decided-case count and measured precision -from the public stats endpoint (`/v1/public/stats`, the `rulePrecision` block). The aggregated -run history is also readable directly: +The aggregated run history is readable directly too, again against the deployment's own database +(operator / self-host): ```bash npx tsx scripts/backtest-track-record.ts --db loopover --remote ``` Your replayed `confirmed / decided` for a rule should match the published precision for the same -window; the freeze-point checksum ties the published numbers to the exact corpus you just -verified. +window. The freeze-point checksum ties the published numbers to the corpus you just verified only +for the rule the latest run actually backtested — for any other rule it is a timestamped pointer to +a different run, not a commitment to that rule's own cases. ## 5. Verify an attested run (when a run carries one) diff --git a/apps/loopover-ui/content/docs/what-you-can-verify.mdx b/apps/loopover-ui/content/docs/what-you-can-verify.mdx index 3df962efc3..6fa1d12b71 100644 --- a/apps/loopover-ui/content/docs/what-you-can-verify.mdx +++ b/apps/loopover-ui/content/docs/what-you-can-verify.mdx @@ -176,7 +176,10 @@ replayable history — not hand-entered numbers. This is the walkthrough in [Verify this review](/docs/verify-this-review): export the checksummed corpus, verify its checksum, re-run the same public scoring functions from `@loopover/engine`, and -compare. **Anyone** can do this for public repositories. +compare. The checksum verification and the scorer replay are pure functions **anyone** can run over +a snapshot they hold, and the published numbers themselves are fetchable unauthenticated — but the +export step that produces the snapshot reads the deployment's own database and needs that +deployment's credentials, so it is an operator / self-host step today rather than an anonymous one. **Private repositories are the exception.** A hosted tenant's review history cannot be published diff --git a/src/review/eval-score-records.ts b/src/review/eval-score-records.ts index 64e9f9305f..c1b6520764 100644 --- a/src/review/eval-score-records.ts +++ b/src/review/eval-score-records.ts @@ -4,7 +4,7 @@ // It adds no new scoring and no new trust -- the numbers are the same ones `/v1/public/stats` already // publishes, just committed to a corpus checksum and made independently re-derivable per-record. import { canonicalJson, contentDigest } from "./decision-record"; -import type { PublicRulePrecision } from "./public-rule-precision"; +import { EMPTY_CORPUS_CHECKSUM, type PublicRulePrecision } from "./public-rule-precision"; export const EVAL_SCORE_RECORD_SCHEMA_VERSION = 1 as const; @@ -76,6 +76,14 @@ async function finalizeRecord(input: EvalScoreRecordDigestInput): Promise { if (!precision.latestBacktestRun) return []; const { corpusChecksum } = precision.latestBacktestRun; + if (corpusChecksum === EMPTY_CORPUS_CHECKSUM) return []; const windowStart = new Date(Date.parse(issuedAt) - precision.windowDays * 24 * 60 * 60 * 1000).toISOString(); const records = await Promise.all( @@ -145,3 +154,4 @@ export async function verifyEvalScoreRecordDigest(record: EvalScoreRecord): Prom // Re-exported so callers that only import this module never need a second import from decision-record.ts // just to canonicalize something alongside a record (e.g. logging, or a future signed-bundle wrapper). export { canonicalJson, contentDigest }; +export { EMPTY_CORPUS_CHECKSUM }; diff --git a/src/review/public-rule-precision.ts b/src/review/public-rule-precision.ts index 18b7d9cc7a..c5136cf3f1 100644 --- a/src/review/public-rule-precision.ts +++ b/src/review/public-rule-precision.ts @@ -23,6 +23,15 @@ export const PUBLIC_PRECISION_MIN_DECIDED = 10; // duplication rule-calibration-trend.ts documents for its identical queries. const HUMAN_OVERRIDE_EVENT_TYPE_PREFIX = "signal.human_override:"; +/** `checksumCases([])` — SHA-256 over the canonicalized empty case list (the two-byte string `"[]"`), i.e. what + * a corpus export produces for a rule with no labeled cases at all. A hash over zero cases is the same 32 + * bytes for every rule, every window and every deployment, so it is not the "independently-verifiable freeze + * point" {@link PublicRulePrecision.latestBacktestRun} claims to be — it points at nothing a skeptic could + * re-derive anything from. Hard-coded because the canonicalization that produces it + * (`scripts/backtest-corpus-export-core.ts`) runs on `node:crypto` and is unimportable from the Workers + * runtime; this module's own test re-derives it from `sha256Hex("[]")` so it can never drift. */ +export const EMPTY_CORPUS_CHECKSUM = "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945"; + export type PublicRulePrecisionRow = { ruleId: string; decided: number; @@ -40,7 +49,9 @@ export type PublicRulePrecision = { /** All three reversal shapes counted over the window — the "counted against ourselves" number. */ reversals: { reopened: number; reverted: number; superseded: number }; /** The latest persisted backtest run carrying a corpus checksum — the independently-verifiable freeze - * point — or null when no run has been recorded yet. */ + * point — or null when no run has been recorded yet, or when the latest run's corpus was empty (see + * {@link EMPTY_CORPUS_CHECKSUM}: a commitment to nothing is not a freeze point, so it is reported as + * absent rather than published as though it were verifiable). */ latestBacktestRun: { corpusChecksum: string; at: string } | null; }; @@ -102,6 +113,9 @@ export async function loadPublicRulePrecision(env: Env, nowMs: number = Date.now reverted: reversalCount("reversal_reverted"), superseded: reversalCount("reversal_superseded"), }, - latestBacktestRun: latest && typeof latest.checksum === "string" && latest.checksum !== "" ? { corpusChecksum: latest.checksum, at: latest.created_at } : null, + latestBacktestRun: + latest && typeof latest.checksum === "string" && latest.checksum !== "" && latest.checksum !== EMPTY_CORPUS_CHECKSUM + ? { corpusChecksum: latest.checksum, at: latest.created_at } + : null, }; } diff --git a/test/unit/eval-score-records.test.ts b/test/unit/eval-score-records.test.ts index 1b8479ddf1..910e58ca3f 100644 --- a/test/unit/eval-score-records.test.ts +++ b/test/unit/eval-score-records.test.ts @@ -6,9 +6,10 @@ import { ORB_GATE_SUBJECT_ID, OUTCOME_CONFIRMED_PRECISION_SCORING_RULE_VERSION, verifyEvalScoreRecordDigest, + EMPTY_CORPUS_CHECKSUM, type EvalScoreRecord, } from "../../src/review/eval-score-records"; -import { contentDigest } from "../../src/review/decision-record"; +import { contentDigest, sha256Hex } from "../../src/review/decision-record"; import type { PublicRulePrecision } from "../../src/review/public-rule-precision"; const ISSUED_AT = "2026-07-27T12:00:00.000Z"; @@ -29,6 +30,25 @@ describe("buildEvalScoreRecordsFromRulePrecision (#9266)", () => { expect(records).toEqual([]); }); + it("refuses to publish records whose freeze point commits to an empty corpus", async () => { + // Regression: production published decided=460/confirmed=287 alongside sha256("[]") -- a hash that is + // byte-identical for every rule and every window, so it committed to nothing a consumer could re-derive. + const records = await buildEvalScoreRecordsFromRulePrecision( + { ...PRECISION_WITH_FREEZE_POINT, latestBacktestRun: { corpusChecksum: EMPTY_CORPUS_CHECKSUM, at: "2026-07-27T10:00:00.000Z" } }, + ISSUED_AT, + ); + expect(records).toEqual([]); + }); + + it("EMPTY_CORPUS_CHECKSUM is the exporter's own checksum over zero cases", async () => { + // scripts/backtest-corpus-export-core.ts hashes `JSON.stringify(cases.map(canonicalizeCase))`, which for + // an empty list is the two-byte string "[]" -- re-derived here so the hard-coded constant cannot drift + // from the exporter that produces the value it guards against. (That exporter's canonicalization and + // canonicalJson coincide ONLY on the empty case, so this hashes the literal preimage rather than + // round-tripping [] through either one.) + expect(EMPTY_CORPUS_CHECKSUM).toBe(await sha256Hex("[]")); + }); + it("builds one record per rule, committed to the freeze point's corpus checksum", async () => { const records = await buildEvalScoreRecordsFromRulePrecision(PRECISION_WITH_FREEZE_POINT, ISSUED_AT); expect(records).toHaveLength(2); diff --git a/test/unit/public-rule-precision.test.ts b/test/unit/public-rule-precision.test.ts index fdf4e5d85d..085c028dba 100644 --- a/test/unit/public-rule-precision.test.ts +++ b/test/unit/public-rule-precision.test.ts @@ -3,7 +3,9 @@ import { loadPublicRulePrecision, PUBLIC_PRECISION_MIN_DECIDED, PUBLIC_PRECISION_WINDOW_DAYS, + EMPTY_CORPUS_CHECKSUM, } from "../../src/review/public-rule-precision"; +import { sha256Hex } from "../../src/review/decision-record"; import { recordAuditEvent } from "../../src/db/repositories"; import { createSignalStore } from "../../src/review/signal-tracking-wire"; import { createTestEnv } from "../helpers/d1"; @@ -95,6 +97,33 @@ describe("loadPublicRulePrecision (#8230)", () => { expect(block.latestBacktestRun).toEqual({ corpusChecksum: "newest111", at: new Date(NOW - 30_000).toISOString() }); }); + it("reports no freeze point when the latest run's corpus was empty", async () => { + // Regression: production's latest run carried sha256("[]") -- the checksum of ZERO cases -- and the + // fairness page rendered it as a "reproducibility freeze point" while /v1/public/eval-scores published + // records committed to it. A hash over no cases is identical everywhere, so it verifies nothing. + const env = createTestEnv(); + await seedVerdicts(env, "ai_consensus_defect", 15, 5); + await recordAuditEvent(env, { + eventType: "calibration.logic_backtest_run", + targetKey: "rule", + outcome: "completed", + metadata: { corpusChecksum: EMPTY_CORPUS_CHECKSUM, comparison: {} }, + createdAt: new Date(NOW - 1000).toISOString(), + }); + + const block = await loadPublicRulePrecision(env, NOW); + expect(block.latestBacktestRun).toBeNull(); + // The scores come from a different dataset (human-override events) and are unaffected -- an empty corpus + // means the numbers are uncommitted, never that they are zero. + expect(block.rules).toEqual([{ ruleId: "ai_consensus_defect", decided: 20, confirmed: 15, precision: 0.75 }]); + }); + + it("EMPTY_CORPUS_CHECKSUM is the exporter's own checksum over zero cases", async () => { + // scripts/backtest-corpus-export-core.ts hashes `JSON.stringify(cases.map(canonicalizeCase))`, which for + // an empty list is the two-byte string "[]" -- re-derived here so the constant cannot drift from it. + expect(EMPTY_CORPUS_CHECKSUM).toBe(await sha256Hex("[]")); + }); + it("degrades fail-safe on a broken store and reports null freeze point on a fresh ledger", async () => { const empty = await loadPublicRulePrecision(createTestEnv(), NOW); expect(empty).toEqual({