diff --git a/docs/CHEAPEST-DISCONFIRMING-TEST.md b/docs/CHEAPEST-DISCONFIRMING-TEST.md new file mode 100644 index 0000000..549259a --- /dev/null +++ b/docs/CHEAPEST-DISCONFIRMING-TEST.md @@ -0,0 +1,101 @@ +# Cheapest Disconfirming Test + +Every serious claim in Open Problem Lab should identify the cheapest test that could make the repository lower confidence in it. + +This is not a demand for perfect certainty. It is a demand for an explicit reversal path. + +## Why this matters + +AI-assisted research makes it easy to produce plausible claims and expensive-looking evidence packages. The harder question is usually missing: + +> What would change our mind? + +Without that question, the protocol can drift into verification theater: sources are cited, schemas pass, prose sounds careful, but no one knows the smallest observation that would weaken the claim. + +A claim that cannot name its cheapest disconfirming test is usually not ready to become load-bearing repository knowledge. + +## Definition + +A cheapest disconfirming test is the lowest-cost safe check that would materially reduce confidence in a claim if it failed. + +It should be: + +- **specific** — tied to one claim, not a vague research agenda; +- **safe** — does not require unsafe action or misuse; +- **cheap** — uses the smallest credible check before larger work; +- **decision-relevant** — failure would change status, scope, priority, or confidence; +- **auditable** — another contributor can see what was tested and why it matters. + +Cheap does not mean weak. It means the project should try to falsify claims efficiently before spending reviewer attention, compute, field access, or money. + +## Minimum field + +Add this field to issue descriptions, PR bodies, evidence records, or review notes when a claim is intended to become canonical: + +```markdown +**Cheapest disconfirming test:** +If [specific check] shows [specific failure condition], then this claim should be [narrowed / disputed / rejected / moved to exploratory / sent for replication]. +``` + +Examples of acceptable shapes: + +- If the same method fails on a second dated source set, the claim should be marked disputed. +- If the effect disappears after excluding a known biased dataset, the claim should be narrowed to the original dataset only. +- If a field reviewer identifies a routine operational constraint that blocks implementation, the claim should become field-pending rather than accepted. +- If a reproduction run cannot recover the reported metric within the declared tolerance, the artifact should become replication-pending. + +## Bad patterns + +Bad tests are vague, heroic, or unfalsifiable. + +Avoid: + +- "More research is needed." +- "An expert should review this someday." +- "If the whole literature changes, update it." +- "If someone proves it wrong." +- "Run a large trial" when a source check, data audit, or small reproduction would test the load-bearing assumption first. + +The point is to name the first useful crack, not the final definitive study. + +## Reviewer rule + +A reviewer may ask for a cheapest disconfirming test before accepting a claim as canonical. + +The reviewer should especially ask for it when: + +- the claim is safety-sensitive; +- the claim is quantitative; +- the claim affects prioritization; +- the claim is based on indirect evidence; +- the claim depends on a narrow dataset, geography, population, or operational assumption; +- an agent-generated submission sounds more confident than its evidence. + +## Agent rule + +Before opening a PR, an agent should identify the load-bearing assumption in its own claim and propose the cheapest safe way to attack it. + +The agent should write: + +1. The claim I want accepted is ... +2. The load-bearing assumption is ... +3. The cheapest disconfirming test is ... +4. If that test fails, the claim should become ... + +If the agent cannot complete those lines, it should not present the claim as accepted knowledge. + +## Relationship to counter-evidence + +The disconfirming test is forward-looking: it says what would lower confidence next. + +The counter-evidence ledger is backward-looking: it records what has already weakened, bounded, or disputed a claim. + +Together, they prevent the repo from accumulating claims with no memory and no reversal path. + +## Acceptance test + +A cheapest disconfirming test is good when a skeptical reviewer can say: + +> This is the first safe thing I would check if I wanted to prove the claim too broad, too weak, or not yet ready. + +If the test would not change anything, it is not a disconfirming test. It is decoration.