Skip to content

feat: add claim and replication schemas — the core thesis objects - #47

Merged
cschanhniem merged 1 commit into
mainfrom
feat/claim-and-replication-schemas
Jun 28, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/claim-and-replication-schemas

Conversation

@cschanhniem

Copy link
Copy Markdown
Contributor

Summary

The repo's thesis is "CI/CD for truth claims": make false AI-generated knowledge harder to merge. But the protocol had no persistent claim object — only transient submissions and single-source evidence records. This PR adds the missing core objects.

Changes

New schemas

  • schemas/claim.schema.json — the core protocol object. A persistent, falsifiable statement with a verification lifecycle (unverified → dry-lab-verified → needs-replication → replicated → accepted → field-tested), evidence links, failure modes, a kill condition, and required reviewer roles. High-safety claims cannot advance without red-team review. Falsified and deprecated are terminal states; falsified claims stay in the record so contributors do not repeat them.

  • schemas/replication.schema.json — independent replication records with replicator identity (must differ from original submitter), environment (runtime, OS, packages), input hash, result enum (confirmed / partially-confirmed / failed / inconclusive), and divergence tracking. Self-replication is not replication.

Strengthened schemas

  • schemas/evidence.schema.json — added model-prediction, computational-analysis, and wet-lab-confirmation evidence types. The thesis envisions computational claims (molecule candidates, theorem sketches, benchmark results) but the enum had no way to represent them. Added optional doi and archive_url fields to insulate records against link rot.

  • schemas/agent-submission.schema.json — added kill_condition as a required field. SKILL.md pattern Add agent merge bar note #3 already required it in prose; the schema now enforces it. Protocol drift fixed.

New examples

  • examples/claim.example.json — worked example using the dengue-heat-vietnam pack
  • examples/replication.example.json — worked example with a partially-confirmed replication result
  • examples/agent-submission.example.json — updated to include kill_condition

Validator

  • scripts/validate-repo.mjs — compiles and validates all new schemas and example files

Docs

  • AGENTS.md — replaced stale 21-pack table with pointers to generated indexes (actual count: 100). Added claim lifecycle section.
  • ROADMAP.md — fixed "15 problem packs" to "100 problem packs"
  • CLAUDE.md — updated schemas table and evidence type list
  • schemas/AGENTS.md — updated component and flow diagrams
  • examples/AGENTS.md — added new example files to key components
  • agents/literature-scout.md — updated evidence types table with new types
  • README.md — updated repository map description

AI-slop cleanup

  • Replaced vague metaphorical "landscape" in the skills-training pack with precise terms ("full scope of skills-training availability", "training-system assessment")
  • Legitimate ecological uses of "landscape" (landscape simplification, agricultural landscapes, mosaic landscapes) in biodiversity and climate packs were left intact

Protocol Notes

  • The kill_condition field in agent-submission.schema.json was required by SKILL.md pattern Add agent merge bar note #3 but absent from the schema. This is exactly the kind of prose-rule-vs-schema drift the protocol should catch. A future validator rule could scan agent guides for required fields and check them against the corresponding schema.
  • The stale pack count in AGENTS.md (21) and ROADMAP.md (15) vs actual (100) suggests the validator could check that hand-written files do not embed pack counts, or that they match the generated index. Currently the rule exists only in prose ("Never trust an embedded count").

Verification

  • pnpm validate — passes
  • pnpm reproducibility:check — passes (516 tasks)
  • pnpm build — passes (100 packs)
  • pnpm format:check — passes

The repo's thesis is 'CI/CD for truth claims': make false AI-generated
knowledge harder to merge. But the protocol had no persistent claim
object — only transient submissions and single-source evidence records.

This commit adds the missing core objects:

- schemas/claim.schema.json: a persistent, falsifiable statement with a
  verification lifecycle (unverified -> dry-lab-verified -> replicated ->
  accepted -> field-tested), evidence links, failure modes, a kill
  condition, and required reviewer roles. High-safety claims cannot
  advance without red-team review. Falsified claims stay in the record
  so contributors do not repeat them.

- schemas/replication.schema.json: an independent replication record
  with replicator identity, environment, input hash, result, and
  divergence tracking. Self-replication is not replication.

Evidence schema strengthened:
- Added model-prediction, computational-analysis, and wet-lab-confirmation
  evidence types. The thesis envisions computational claims (molecule
  candidates, theorem sketches, benchmark results) but the enum had no
  way to represent them.
- Added optional doi and archive_url fields to insulate records against
  link rot.

Agent-submission schema strengthened:
- Added kill_condition as a required field. SKILL.md pattern #3 already
  required it in prose; the schema now enforces it. Protocol drift fixed.

Validator updated to compile and validate all new schemas and examples.

Stale pack counts fixed: AGENTS.md had a 21-pack table; ROADMAP.md said
15. Actual count is 100. Replaced with pointers to generated indexes,
following the repo's own rule: 'Never trust an embedded count in any
hand-written file.'

AI-slop language cleaned: replaced vague metaphorical 'landscape' in the
skills-training pack with precise terms. Legitimate ecological uses of
'landscape' (landscape simplification, agricultural landscapes) in
biodiversity packs were correctly left intact.

All validation passes: pnpm validate, pnpm reproducibility:check,
pnpm build.
@cschanhniem
cschanhniem merged commit 53222fa into main Jun 28, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant