Skip to content

Commit 16a6fdc

Browse files
committed
test(stats): assert the totals/parity invariant instead of describing it
#9963 was an Orb publishing `totals.handled: 0` in the same payload as `reviewParity.verdicts: 2123`, both derived from the same ledger. #9967 fixed the aggregate and covered it well -- but per surface, which is the blind spot the defect lived in. Both halves were individually correct and individually green while production served the contradiction, and the verifier's own source says why per-surface testing cannot catch this class: "the two are computed by different code over the same ledger, which is the divergence no amount of per-surface testing catches, because each surface is individually self-consistent." `getPublicStats` and `loadReviewParityRollups` remain separate reads with separate gating, so nothing stopped the next change to either from reintroducing the disagreement with every existing test still green. These build BOTH halves over one seeded ledger and compare them through the verifier's own `checkStatsParity` -- the function that printed the original FAIL against the Orb -- rather than restating its rule, so the test cannot drift from the tool that decides whether production is publishing a contradiction. Covers the case a hand-written equality assertion would get wrong: a re-decided pull request appends a verdict row, so parity counts 3 while `handled` counts 2 pull requests. That is consistent, not a contradiction, and only parity EXCEEDING all-time handled is the accounting error. Mutation-tested against #9967's fix: reverting the ledger fallback fails exactly the two behavioural assertions, while the empty-ledger case and the failing-direction guard correctly stay green. Closes #9971
1 parent dcbfe2a commit 16a6fdc

1 file changed

Lines changed: 108 additions & 0 deletions

File tree

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
import { describe, expect, it } from "vitest";
2+
3+
import { checkStatsParity } from "../../packages/loopover-mcp/lib/verify-public-claims";
4+
import { getPublicStats } from "../../src/review/public-stats";
5+
import { loadReviewParityRollups } from "../../src/review/review-parity-rollups";
6+
import { createTestEnv } from "../helpers/d1";
7+
8+
// #9971 (follow-up to #9963): the CROSS-SURFACE invariant, asserted rather than described.
9+
//
10+
// #9963 was an Orb publishing `totals.handled: 0` in the same payload as `reviewParity.verdicts: 2123`, both
11+
// derived from the same ledger. #9967 fixed the aggregate and tested it thoroughly -- but per-surface. That is
12+
// the exact blind spot the defect lived in, and the verifier's own source names it:
13+
//
14+
// "the two are computed by different code over the same ledger, which is the divergence no amount of
15+
// per-surface testing catches, because each surface is individually self-consistent."
16+
//
17+
// Both halves were individually correct and individually green while production served the contradiction.
18+
// `getPublicStats` and `loadReviewParityRollups` are still separate reads with separate gating, so nothing
19+
// stops the next change to either from reintroducing the disagreement -- unless something builds BOTH from one
20+
// ledger and compares them, which is what this file does.
21+
//
22+
// The comparison runs the verifier's OWN `checkStatsParity` rather than restating its rule. That function is
23+
// what printed the original FAIL against the Orb; asserting through it means this test cannot drift from the
24+
// tool that decides whether production is publishing a contradiction, which a hand-written `>=` could.
25+
const NOW = Date.parse("2026-07-30T12:00:00.000Z");
26+
const REPO = "JSONbored/loopover";
27+
28+
/** Insert one ledger verdict. Direct SQL rather than `persistDecisionRecord`: this exercises a READ, and going
29+
* through the writer would drag in digesting and the hash-chain append without making the row any more real. */
30+
async function seedVerdict(env: Env, pull: number, headSha = `sha${pull}`): Promise<void> {
31+
await env.DB.prepare(
32+
`INSERT INTO decision_records (id, repo_full_name, pull_number, head_sha, action, reason_code, record_digest, record_json, created_at)
33+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
34+
)
35+
.bind(
36+
`record:${REPO}#${pull}@${headSha}`,
37+
REPO,
38+
pull,
39+
headSha,
40+
"merge",
41+
"gate_pass",
42+
"d".repeat(64),
43+
"{}",
44+
new Date(NOW - 3_600_000).toISOString(),
45+
)
46+
.run();
47+
}
48+
49+
/** The self-hosted Orb's configuration: public stats on, and NO own-ledger repo allowlist -- which is what
50+
* silently skipped every own-ledger query and left the headline at zero. */
51+
const orbEnv = () => createTestEnv({ LOOPOVER_PUBLIC_STATS: "true", LOOPOVER_PUBLIC_STATS_REPOS: "" });
52+
53+
describe("published totals and parity rollups cannot contradict each other (#9971)", () => {
54+
it("REGRESSION: the verifier's stats-parity claim PASSES for a self-hosted Orb", async () => {
55+
const env = orbEnv();
56+
for (let pull = 1; pull <= 12; pull += 1) await seedVerdict(env, pull);
57+
58+
const [stats, parity] = await Promise.all([getPublicStats(env, NOW), loadReviewParityRollups(env, { nowMs: NOW })]);
59+
60+
// Both surfaces read the same ledger, so they must see the same 12 pull requests. Before #9967:
61+
// handled=0 beside verdicts=12.
62+
expect(parity.verdicts).toBe(12);
63+
expect(stats.totals.handled).toBe(12);
64+
expect(checkStatsParity(stats, parity).status).toBe("pass");
65+
});
66+
67+
it("stays consistent when a pull request is re-decided, which moves the two counts differently", async () => {
68+
// The sharpest case, and the one a hand-written equality assertion would get wrong. `decision_records`
69+
// holds one row per VERDICT, so a re-evaluation adds a row: parity counts verdicts and goes to 3, while
70+
// `handled` counts distinct pull requests and stays at 2. That is not a contradiction -- parity being the
71+
// LARGER of the two in the same direction as real volume is exactly what the verifier tolerates, and only
72+
// parity EXCEEDING all-time handled beyond its tolerance is the accounting error.
73+
const env = orbEnv();
74+
await seedVerdict(env, 1);
75+
await seedVerdict(env, 2);
76+
await seedVerdict(env, 2, "sha2-rev2");
77+
78+
const [stats, parity] = await Promise.all([getPublicStats(env, NOW), loadReviewParityRollups(env, { nowMs: NOW })]);
79+
80+
expect(parity.verdicts).toBe(3);
81+
expect(stats.totals.handled).toBe(2);
82+
expect(checkStatsParity(stats, parity).status).toBe("pass");
83+
});
84+
85+
it("INVARIANT: an empty ledger agrees trivially rather than failing", async () => {
86+
// The hosted Worker's shape (review execution retired, ledger empty by design). 0 and 0 agree, and the
87+
// claim must not read a quiet deployment as a broken one.
88+
const env = orbEnv();
89+
90+
const [stats, parity] = await Promise.all([getPublicStats(env, NOW), loadReviewParityRollups(env, { nowMs: NOW })]);
91+
92+
expect(parity.verdicts).toBe(0);
93+
expect(stats.totals.handled).toBe(0);
94+
expect(checkStatsParity(stats, parity).status).toBe("pass");
95+
});
96+
97+
it("MUTATION GUARD: the same claim FAILS when handled is zeroed beneath a populated rollup", async () => {
98+
// Proves the assertions above are driven by the numbers rather than passing for any payload at all. This
99+
// is byte-for-byte the contradiction production published, and the claim has to reject it.
100+
const env = orbEnv();
101+
for (let pull = 1; pull <= 12; pull += 1) await seedVerdict(env, pull);
102+
const parity = await loadReviewParityRollups(env, { nowMs: NOW });
103+
104+
const result = checkStatsParity({ totals: { handled: 0 } }, parity);
105+
expect(result.status).toBe("fail");
106+
expect(result.detail).toContain("exceeding the all-time handled count of 0");
107+
});
108+
});

0 commit comments

Comments
 (0)