Skip to content

CERTIFICATION_GATES.md

Latest commit

 

History

History
84 lines (65 loc) · 4.66 KB

CERTIFICATION_GATES.md

File metadata and controls

84 lines (65 loc) · 4.66 KB
Raw

Certification Gates — the one authored claim-rules doc

Authored 2026-07-12 (direction audit). This file OWNS the claim rules that were previously scattered across seven files. Generated artifacts (board, readiness, handoff) restate slices of this; when they disagree, fix the generator or this file — do not fork a third statement.

1. The claim ladder

From docs/dogfood/official-score-boundary.md (the honesty anchor):

  • Product proof — local/live ProofLoop receipts, chart packs, dashboard exports, model/cost receipts. Proves the app, agent loop, graph, memory, reports, and provider setup behave. It NEVER becomes an official benchmark score by wording.
  • Official proof — an imported upstream scorer, Harbor/Gandalf receipt, or benchmark-accepted judge receipt, stored under docs/eval/proofloop-official-scores/.
  • Blocked official proof — explicit blocked_external status naming the missing scorer, credential, task bundle, or acceptance path.

Proxy model sweeps, local deterministic gates, LangSmith mirrors, and product UI runs are product proof unless the upstream benchmark accepts that path.

2. Status vocabulary (the trap that mints false claims)

From the benchmark board's interpretation section — memorize the first row:

Status Means Does NOT mean
proven (official score) The official scorer/verifier result was imported The benchmark was passed. BTB is proven at mean reward 0.2519, pass-rate 0.0000
proven (product path) ProofLoop has evidence for the app workflow An official leaderboard score
registered Tracked, adapter contract exists Live-proofed; do not sell it
not_applicable Internal/product harness lane A blocked official lane
blocked Scorer/verifier path not imported Product proof is missing (it may exist)
needs_scaffold_or_run Local work must be attempted before "external-blocked" is allowed Externally blocked
proxy_only Local/proxy evidence exists Any official claim
flipEligible (receipt field) The lane EARNED a board-status flip Anything, if false — the lane stays blocked regardless of summaries

Wording rules for outward copy stay in docs/WEDGE.md; buyer-facing surfaces say "verification you run", never "certification".

3. Two-layer certification (decided policy)

  1. Headless capability lane — all tasks run isolated (e.g. Harbor for BTB), generic-only, no answer-key writers, official scorer.
  2. Stratified browser lane — a seeded sample re-run through the real product UI as the TRANSFER-AGREEMENT gate; the sample must include failures.
  3. Divergence fails closed. If browser-lane results disagree with the headless lane, the claim is blocked — never averaged.
  4. Never claim "all tasks browser-verified" unless that is literally what ran.

Enforcement: the exploration loop proposes, it never grades or promotes itself (evaluateScaffoldAcceptance() requires an external adversarial-review approval — see AGENTS.md "two-loop architecture").

4. Long-running goals

Gate/supervise/resume take --goal <id>. Current goals (committed receipt: docs/eval/PROOFLOOP_GOAL_LEDGER.md; raw state gitignored under .proofloop/goals/): dev-audience-ready · official-scores · voice-agent-implementation · voice-agent-merge-packet. Note: npm run proofloop:gate / proofloop:resume are hardwired to --goal official-scores; other goals need the explicit flag.

5. Generated files → regeneration commands

Never hand-edit these; regenerate them. Hand-editing a receipt is evidence falsification.

File Regenerated by
docs/eval/PROOFLOOP_BENCHMARK_BOARD.md npm run benchmark:proofloop:board
docs/eval/OFFICIAL_BENCHMARK_READINESS.md npm run benchmark:official:readiness
docs/eval/scaffold-handoff.md (gitignored) npm run scaffold:check
docs/eval/PROOFLOOP_GOAL_LEDGER.md proofloop CLI goal-ledger receipt (via npm run proofloop -- gate/supervise)
README proof-release block src/eval/proofReleasePublishing.ts
.proofloop/lanes/<id>/* receipts lane scripts — see docs/eval/BENCHMARK_RUNBOOK.md per-lane receipts row
CLAUDE.md / AGENTS.md marker block proofloop CLI init (src/eval/proofloopAgentFriendlyCli.ts)

6. Receipt-commit policy

Regenerated lane receipts commit atomically with the board that cites them. A board flip without its supporting receipt in the same commit is invalid (this is exactly the SpreadsheetBench C15 defect found 2026-07-12 — see docs/audit/DIRECTION_AUDIT_2026-07-12.md). HEAD must never tell a different certification story than the working tree for more than one working session.