Skip to content

Commit bfbe4db

Browse files
committed
feat: add claim and replication schemas — the core thesis objects
The repo's thesis is 'CI/CD for truth claims': make false AI-generated knowledge harder to merge. But the protocol had no persistent claim object — only transient submissions and single-source evidence records. This commit adds the missing core objects: - schemas/claim.schema.json: a persistent, falsifiable statement with a verification lifecycle (unverified -> dry-lab-verified -> replicated -> accepted -> field-tested), evidence links, failure modes, a kill condition, and required reviewer roles. High-safety claims cannot advance without red-team review. Falsified claims stay in the record so contributors do not repeat them. - schemas/replication.schema.json: an independent replication record with replicator identity, environment, input hash, result, and divergence tracking. Self-replication is not replication. Evidence schema strengthened: - Added model-prediction, computational-analysis, and wet-lab-confirmation evidence types. The thesis envisions computational claims (molecule candidates, theorem sketches, benchmark results) but the enum had no way to represent them. - Added optional doi and archive_url fields to insulate records against link rot. Agent-submission schema strengthened: - Added kill_condition as a required field. SKILL.md pattern #3 already required it in prose; the schema now enforces it. Protocol drift fixed. Validator updated to compile and validate all new schemas and examples. Stale pack counts fixed: AGENTS.md had a 21-pack table; ROADMAP.md said 15. Actual count is 100. Replaced with pointers to generated indexes, following the repo's own rule: 'Never trust an embedded count in any hand-written file.' AI-slop language cleaned: replaced vague metaphorical 'landscape' in the skills-training pack with precise terms. Legitimate ecological uses of 'landscape' (landscape simplification, agricultural landscapes) in biodiversity packs were correctly left intact. All validation passes: pnpm validate, pnpm reproducibility:check, pnpm build.
1 parent e7198d0 commit bfbe4db

19 files changed

Lines changed: 387 additions & 46 deletions

‎AGENTS.md‎

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ Correct, narrow, and falsifiable beats prolific.
2424
## Key Components
2525

2626
- `problem-packs/`: canonical problem-pack directories with task maps across multiple domains. Read generated indexes for live counts. When calibrating pack quality, use strong exemplars such as `climate-health/dengue-heat-vietnam` for operational humility and `public-health/birth-registration-access-global` for measure-family discipline across survey, CRVS, and health-touchpoint evidence.
27-
- `schemas/`: JSON schemas for machine-checkable protocol objects.
27+
- `schemas/`: JSON schemas for machine-checkable protocol objects. Includes the claim schema (the core thesis object: persistent claims with a verification lifecycle, evidence links, failure modes, kill conditions, and required reviewers) and the replication schema (independent replication records with environment, input hash, and divergence tracking).
2828
- `.github/ISSUE_TEMPLATE/`: structured issue forms.
2929
- `.github/workflows/`: validation, source verification, reproducibility, and Wiki publishing.
3030
- `agents/`: role guides for structured agent contributions.
@@ -34,29 +34,16 @@ Correct, narrow, and falsifiable beats prolific.
3434

3535
## Active Problem Packs
3636

37-
| ID | Domain | Region |
38-
| ------------------------------------------------------ | --------------------------------------- | ------------------------------ |
39-
| `air-quality/indoor-air-pollution-sub-saharan-africa` | air-quality, public-health | sub-saharan-africa |
40-
| `air-quality/pm25-monitoring-south-asia` | air-quality, public-health | south-asia |
41-
| `biodiversity/coral-bleaching-great-barrier-reef` | biodiversity, climate-health | great-barrier-reef |
42-
| `biodiversity/deforestation-amazon` | biodiversity, climate-health | amazon |
43-
| `climate-adaptation/sea-level-rise-small-islands` | climate-adaptation, disaster-resilience | small-island-developing-states |
44-
| `climate-health/dengue-heat-vietnam` | climate-health, public-health | vietnam |
45-
| `climate-health/heat-stress-urban-south-asia` | climate-health, public-health | south-asia |
46-
| `climate-health/malaria-early-warning-africa` | climate-health, public-health | sub-saharan-africa |
47-
| `disaster-resilience/cyclone-early-warning-bangladesh` | disaster-resilience, climate-health | bangladesh |
48-
| `disaster-resilience/earthquake-vulnerability-nepal` | disaster-resilience | nepal |
49-
| `education/girls-education-sub-saharan-africa` | education | sub-saharan-africa |
50-
| `education/learning-loss-post-pandemic` | education | global |
51-
| `energy-access/clean-cooking-sub-saharan-africa` | energy-access, public-health | sub-saharan-africa |
52-
| `energy-access/mini-grid-rural-sub-saharan-africa` | energy-access | sub-saharan-africa |
53-
| `food-security/drought-early-warning-horn-of-africa` | food-security, disaster-resilience | east-africa |
54-
| `food-security/locust-outbreak-east-africa` | food-security, climate-health | east-africa |
55-
| `public-health/lead-exposure-urban-global` | public-health | global |
56-
| `public-health/stunting-sub-saharan-africa` | public-health, food-security | sub-saharan-africa |
57-
| `sanitation/open-defecation-india` | sanitation, public-health | india |
58-
| `water-security/glacial-melt-hindu-kush` | water-security, climate-adaptation | hindu-kush-himalaya |
59-
| `water-security/groundwater-depletion-india` | water-security, food-security | india |
37+
The portfolio grows continuously. Do not rely on a static count or a hand-maintained table. Read the live generated indexes:
38+
39+
- [`tasks-available.json`](tasks-available.json) — machine-readable index of every scoped task
40+
- [`agent-radar.json`](agent-radar.json) — routing layer ranking first moves and unlock paths
41+
- [`docs/wiki/Problem-Packs.md`](docs/wiki/Problem-Packs.md) — auto-generated reader-facing list
42+
43+
When calibrating pack quality, use these strong exemplars:
44+
45+
- `climate-health/dengue-heat-vietnam` — operational humility, bounded claims, analytic-use warnings
46+
- `public-health/birth-registration-access-global` — measure-family discipline across survey, CRVS, and health-touchpoint evidence
6047

6148
## Agent Working Rules
6249

@@ -105,6 +92,29 @@ A contribution may only be merged if at least one of these is true:
10592

10693
Pure prose polish without one of the above is not a reason to merge.
10794

95+
## The Claim Lifecycle
96+
97+
The thesis of this repository is that verification is the scarce resource. The claim object is the protocol object that makes that thesis machine-checkable. A claim is a persistent, falsifiable statement that exists independently of any single submission or review. It accumulates evidence and failure modes over time, carries its own kill condition, and moves through a verification lifecycle.
98+
99+
```text
100+
unverified -> dry-lab-verified -> needs-replication -> replicated -> accepted -> field-tested
101+
\-> rejected
102+
\-> falsified
103+
\-> deprecated
104+
```
105+
106+
- `unverified`: A claim has been submitted but no evidence has been reviewed.
107+
- `dry-lab-verified`: Evidence records support the claim at the computational or literature level. No wet-lab or field confirmation.
108+
- `needs-replication`: Evidence has been reviewed but independent replication is required before the claim can advance.
109+
- `replicated`: An independent replicator has confirmed the result. The replication record is linked.
110+
- `accepted`: The claim has passed all required reviews and replication. It is canonical repo truth.
111+
- `field-tested`: The claim has been tested in a real-world operational context. The highest status.
112+
- `rejected`: The claim did not survive review. Terminal.
113+
- `falsified`: The kill condition was met. Terminal. A falsified claim stays in the record so future contributors do not repeat it.
114+
- `deprecated`: The claim was accepted but its evidence has decayed (source rot, dataset revision, superseded by a stronger claim). Terminal but recoverable through a new submission.
115+
116+
A claim with no evidence is `unverified` regardless of prose. A claim with no kill condition is not a claim. A high-safety claim without red-team review cannot advance past `dry-lab-verified`. The schema enforces these constraints; reviewers enforce the rest.
117+
108118
## Roles for Top AI Agents
109119

110120
Strong models should pick a role and stay in it for the duration of a PR. Mixing roles in one submission hides which judgment failed.

‎CLAUDE.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,13 @@ Machine-checkable schemas for every artifact type. Read before creating or editi
7878
| Evidence record | `schemas/evidence.schema.json` |
7979
| Agent submission | `schemas/agent-submission.schema.json` |
8080
| Review | `schemas/review.schema.json` |
81+
| Claim | `schemas/claim.schema.json` |
82+
| Replication | `schemas/replication.schema.json` |
8183

8284
All schemas have `description` fields on every property.
8385

86+
The claim schema is the core protocol object: a persistent, falsifiable statement with a verification lifecycle (`unverified` to `field-tested`), evidence links, failure modes, a kill condition, and required reviewer roles. Replication records track independent attempts to reproduce a claim, with environment, input hash, and divergence tracking.
87+
8488
## Submission Quality Standard
8589

8690
**One task. One role. One claim.** Submissions that mix roles or make multiple independent claims are returned for splitting.
@@ -90,7 +94,7 @@ Every evidence record requires:
9094
- Claim: one specific, falsifiable statement
9195
- Source: title and stable URL
9296
- Source date and access date
93-
- Evidence type: `primary-source` | `peer-reviewed-study` | `dataset` | `field-report` | `expert-review` | `replication` | `negative-result`
97+
- Evidence type: `primary-source` | `peer-reviewed-study` | `dataset` | `field-report` | `expert-review` | `replication` | `negative-result` | `model-prediction` | `computational-analysis` | `wet-lab-confirmation`
9498
- Method: how you assessed what the source proves
9599
- Limitations: what the source cannot prove (be specific — "results may vary" is not a limitation)
96100
- Confidence: high / medium / low / unknown

‎ROADMAP.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ The V0 system uses GitHub itself as the operating surface. No custom web app. Ev
1010

1111
### What V0 delivers
1212

13-
- **15 problem packs** across 11 domains: air quality, biodiversity, climate adaptation, climate health, disaster resilience, education, energy access, food security, public health, sanitation, and water security.
13+
- **100 problem packs** across 11 domains: air quality, biodiversity, climate adaptation, climate health, disaster resilience, education, energy access, food security, public health, sanitation, and water security.
1414
- **Machine-checkable schemas** for problems, tasks, evidence records, agent submissions, and reviews.
1515
- **Five structured issue forms**: problem, task, agent submission, review, and safety flag.
1616
- **Automated validation**: schema checks, label coverage, wiki freshness, source verification, and reproducibility checks run on every pull request.

‎agents/literature-scout.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -36,15 +36,18 @@ Every Literature Scout submission must include:
3636

3737
These are the allowed values in `evidence.schema.json`. Use exactly these strings.
3838

39-
| Type | When to use |
40-
| --------------------- | ------------------------------------------------------------------------------------------------------ |
41-
| `primary-source` | Original data, official statistics, WHO fact sheets, program records |
42-
| `peer-reviewed-study` | Published research with methods that can be replicated, including systematic reviews and meta-analyses |
43-
| `dataset` | A standalone dataset (use for data-source records) |
44-
| `field-report` | Reports from field operations, NGO program evaluations, grey literature |
45-
| `expert-review` | Qualitative assessment from named domain experts |
46-
| `replication` | An independent replication of a prior claim |
47-
| `negative-result` | A well-documented finding that a signal, method, or claim did not hold |
39+
| Type | When to use |
40+
| ------------------------ | ------------------------------------------------------------------------------------------------------ |
41+
| `primary-source` | Original data, official statistics, WHO fact sheets, program records |
42+
| `peer-reviewed-study` | Published research with methods that can be replicated, including systematic reviews and meta-analyses |
43+
| `dataset` | A standalone dataset (use for data-source records) |
44+
| `field-report` | Reports from field operations, NGO program evaluations, grey literature |
45+
| `expert-review` | Qualitative assessment from named domain experts |
46+
| `replication` | An independent replication of a prior claim |
47+
| `negative-result` | A well-documented finding that a signal, method, or claim did not hold |
48+
| `model-prediction` | An AI or computational model prediction (e.g., predicted antimicrobial peptide activity) |
49+
| `computational-analysis` | A computational analysis or simulation (e.g., molecular docking, epidemiological back-testing) |
50+
| `wet-lab-confirmation` | A wet-lab experimental result that confirms or refutes a prior computational prediction |
4851

4952
## What Good Looks Like
5053

‎examples/AGENTS.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@ Examples here are operator aids. They should mirror canonical workflow rules clo
77
## Key Components
88

99
- `AGENT-BOOTSTRAP-PROMPT.md`: one-shot operator prompt that routes an external agent into the repo correctly.
10-
- `agent-submission.example.json`: valid sample submission shape.
10+
- `agent-submission.example.json`: valid sample submission shape, including kill_condition.
1111
- `review.example.json`: valid sample review shape.
12+
- `claim.example.json`: valid sample claim object — the core thesis object with verification lifecycle, evidence links, failure modes, and kill condition.
13+
- `replication.example.json`: valid sample replication record with environment, input hash, and divergence tracking.
1214

1315
## Diagrams (Mermaid)
1416

‎examples/agent-submission.example.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
"A reader may treat climate association as a local causal threshold.",
1717
"A contributor may skip district-level denominator checks."
1818
],
19+
"kill_condition": "A reviewer finds that the cited sources do not support framing-level claims about dengue-climate association in Viet Nam, or that a listed source has been retracted.",
1920
"confidence": "medium",
2021
"suggested_next_issue": "Inventory public district-level dengue surveillance data access."
2122
}

‎examples/claim.example.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"id": "dengue-heat-rainfall-association-vietnam",
3+
"problem_id": "climate-health/dengue-heat-vietnam",
4+
"claim": "Interactions among temperature, hydrometeorology, infrastructure, and mobility are associated with dengue emergence in Viet Nam at district level, based on 23 years of surveillance data. This association is framable but does not prove a local causal threshold.",
5+
"domain": ["climate-health", "public-health"],
6+
"status": "dry-lab-verified",
7+
"evidence": ["gibb-vietnam-dengue-2023", "who-dengue-fact-sheet-2025"],
8+
"failure_modes": [
9+
"A reader treats the association as a local causal threshold for a specific district.",
10+
"Passive surveillance data reflects reporting access, not only true incidence.",
11+
"Infrastructure and mobility variables are proxy measures that may not capture the true confounders."
12+
],
13+
"kill_condition": "A replication on an independent surveillance dataset for Viet Nam finds no statistically significant association between temperature, rainfall, or infrastructure interactions and district-level dengue incidence after controlling for reporting access.",
14+
"review_required": ["domain-reviewer", "replicator"],
15+
"safety_level": "medium",
16+
"submitter": "agent-literature-scout-1",
17+
"created_date": "2026-06-06",
18+
"last_updated": "2026-06-06",
19+
"confidence": "medium",
20+
"limitations": [
21+
"The peer-reviewed model uses passive surveillance data that can reflect reporting access, not only true incidence.",
22+
"The study covers Viet Nam nationally; district-level generalization to other countries is not supported."
23+
]
24+
}

‎examples/replication.example.json‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"id": "dengue-heat-association-replication-1",
3+
"claim_id": "dengue-heat-rainfall-association-vietnam",
4+
"replicator": "agent-data-cleaner-2",
5+
"replication_date": "2026-06-10",
6+
"method": "Re-extracted district-level dengue surveillance data from the original supplementary tables, recomputed the temperature-rainfall-infrastructure interaction terms using the reported model specification, and compared coefficient signs and significance against the published results.",
7+
"environment": {
8+
"runtime": "python 3.12.3",
9+
"os": "ubuntu-24.04",
10+
"packages": "pandas 2.2.2, statsmodels 0.14.2, numpy 1.26.4",
11+
"notes": "CI runner with 4 vCPU and 16 GB RAM. Used the author-supplied supplementary data file directly."
12+
},
13+
"input_hash": "sha256:a1b2c3d4e5f6789012345678abcdef0123456789abcdef0123456789abcdef01",
14+
"result": "partially-confirmed",
15+
"divergence": "Interaction terms for temperature and rainfall replicated within reported confidence intervals. The infrastructure-mobility interaction term was significant in the original but marginal (p=0.06) in the replication, likely due to a minor difference in district exclusion criteria when handling missing mobility data. The core association is confirmed; the infrastructure pathway needs a tighter specification.",
16+
"confidence": "medium"
17+
}

‎problem-packs/education/skills-training-youth-employment-global/evidence.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,9 @@
5050
},
5151
"source_date": "2023-09-01",
5252
"access_date": "2026-06-18",
53-
"method": "Reviewed training-data landscape assessments, provider-registry documentation, and employment-outcome tracking rates across LMICs.",
53+
"method": "Reviewed training-data assessments, provider-registry documentation, and employment-outcome tracking rates across LMICs.",
5454
"limitations": [
55-
"The originally cited 'Skills training data systems landscape assessment' could not be confirmed to exist; re-sourced to the World Bank/ILO/UNESCO TVET systems report.",
55+
"The originally cited 'Skills training data systems assessment' could not be confirmed to exist; re-sourced to the World Bank/ILO/UNESCO TVET systems report.",
5656
"This report describes TVET system fragmentation broadly; it does not quantify provider-registry coverage country by country.",
5757
"A dedicated source is still needed to substantiate the 'no single dataset' claim for any specific country."
5858
],

‎problem-packs/education/skills-training-youth-employment-global/evidence.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Use this source for the best available evidence on training-program employment e
1616

1717
### Training Provider Data Fragmentation
1818

19-
Use this source (World Bank/ILO/UNESCO TVET systems report, 2023) to document that TVET provision in most LMICs is diverse and fragmented across government ministries, private providers, and NGOs, complicating system-wide data on providers and outcomes. (Re-sourced June 2026: the prior 'landscape assessment' citation could not be confirmed; the 'no single dataset' and 'fewer than 10 percent track outcomes' specifics still need a dedicated source.)
19+
Use this source (World Bank/ILO/UNESCO TVET systems report, 2023) to document that TVET provision in most LMICs is diverse and fragmented across government ministries, private providers, and NGOs, complicating system-wide data on providers and outcomes. (Re-sourced June 2026: the prior 'training-system assessment' citation could not be confirmed; the 'no single dataset' and 'fewer than 10 percent track outcomes' specifics still need a dedicated source.)
2020

2121
### Informal Sector Training Outcome Gap
2222

0 commit comments

Comments
 (0)