docs: self-document agent-submission schema and add reviewer guide

Add description fields to every property in agent-submission.schema.json,
making it self-documenting for AI agents that parse schemas: what each field
requires, what counts as a valid value, what common mistakes look like, and
why the field exists. Addresses the most critical path for AI agent contributors.

Add docs/REVIEW-GUIDE.md — a step-by-step guide for domain reviewers covering:
evidence quality check, method review, failure mode review, done condition check,
review record format (accepted/needs-changes/needs-replication/reject), replication
protocol, common review mistakes, and how reviewer reputation is earned. This is
the primary resource for recruiting and onboarding domain experts.

Author: cschanhniem

docs/REVIEW-GUIDE.md

@@ -0,0 +1,146 @@
+# Review Guide
+
+Domain reviewers are the most valuable contributors to Open Problem Lab. This guide explains how to conduct a review, what to look for, and how to write a review record.
+
+## What Reviewing Means Here
+
+A review is not a judgment of the contributor. It is a judgment of whether the evidence and method are sufficient to justify the claim at the stated confidence level, in the context of the problem pack's safety requirements.
+
+You may reject for:
+- Weak or undated evidence
+- Unclear method
+- Hidden assumptions
+- Missing failure modes
+- Unsafe downstream use
+- Unverifiable claims
+
+You do not need to prove the claim false. The submitter must prove it reliable enough to merge.
+
+## Review Types
+
+| Type | When you are needed |
+|------|---------------------|
+| `domain-reviewer` | You have expertise in the problem area (epidemiology, climate science, water engineering, etc.) |
+| `replicator` | You independently rerun a quantitative artifact and confirm reproducibility |
+| `red-team-reviewer` | You adversarially stress-test claims and failure modes |
+| `field-reality-reviewer` | You have direct field experience with the decision context |
+
+A single PR may need multiple review types. The task's `reviewer_needed` field specifies which.
+
+## Step-by-Step Review Process
+
+### 1. Understand the Task
+
+Read the task in `tasks.json` to understand:
+- What outcome was expected
+- What inputs were specified
+- What the done condition is
+- What safety risk level applies
+
+### 2. Check Evidence Quality
+
+For each evidence record in `evidence.json`:
+- Can you access the URL?
+- Does the source_date look accurate?
+- Does the claim accurately represent what the source says?
+- Are the limitations honest? (Too few limitations is usually a red flag, not a sign of quality)
+- Is the evidence type correctly classified?
+
+**Red flags in evidence:**
+- Claims that exceed what the source can prove at the stated grain
+- Source dates that are missing or implausible
+- URLs that return errors or require unapproved access
+- Limitations that are vague ("results may vary")
+- Citing a secondary source (a report about a study) instead of the primary
+
+### 3. Check the Method
+
+For the submission's method field:
+- Could you replicate the reasoning from the evidence to the claim?
+- Is there a step where the submitter makes an inference the evidence does not support?
+- Are the assumptions listed complete?
+
+A method that says "reviewed the literature" without specifying which sections, what criteria, and what conclusions were drawn is not a reviewable method.
+
+### 4. Check the Failure Modes
+
+For each failure mode listed:
+- Is it specific enough to be useful?
+- Does it cover the most plausible misuse scenario?
+- For high-risk submissions: has a red-team-reviewer signed off?
+
+**Things to look for:**
+- Grain mismatch: is the claim being made at a grain the evidence cannot support?
+- Causal overreach: does the claim imply causation where the evidence shows only association?
+- Operational overreach: does the output imply actionability it has not earned?
+
+### 5. Check the Done Condition
+
+Compare the submission to the task's done condition. Ask:
+- Is the done condition met? (Binary: yes or no)
+- If not, what specifically is missing?
+
+### 6. Write the Review
+
+Use the review issue template or the `review.schema.json` format:
+
+```json
+{
+  "review_id": "task-id-review-N",
+  "target_id": "task-id",
+  "reviewer_role": "domain-reviewer",
+  "verdict": "accepted | needs-changes | needs-replication | reject",
+  "strengths": ["..."],
+  "concerns": ["..."],
+  "required_changes": ["..."],
+  "replication_needed": true | false,
+  "confidence": "high | medium | low"
+}
+```
+
+**Verdict guide:**
+- `accepted`: Evidence is sufficient, method is sound, failure modes are honest, done condition is met.
+- `needs-changes`: Specific changes would make this acceptable. List them in `required_changes`.
+- `needs-replication`: The claim is plausible but needs independent reproduction before acceptance.
+- `reject`: The claim is unsupported, the method is invalid, or the output is unsafe to publish.
+
+### 7. Post the Review
+
+Post the review as a structured comment on the pull request. Use the `review.schema.json` format or the Review issue template.
+
+If you are requesting replication, tag a replicator in your review comment.
+
+## Common Review Mistakes
+
+| Mistake | Why it matters |
+|---------|---------------|
+| Approving because the prose is well-written | Quality prose can hide weak evidence |
+| Requesting cosmetic changes instead of flagging the real issue | Cosmetic reviews waste everyone's time |
+| Being vague in required_changes | "Improve the analysis" is not a reviewable change request |
+| Not reading the task done condition | The review is against the done condition, not your personal standard |
+| Applying high-safety-risk standards to low-risk framing | Mismatch in standards delays useful work |
+
+## Replication Protocol
+
+If your review verdict is `needs-replication`:
+
+1. State specifically what artifact must be replicated and what the expected outcome is.
+2. The replicator should follow the `reproducibility_steps` in the submission exactly.
+3. The replicator should post a replication record: what they ran, what they got, and whether it matched.
+4. If replication fails, the replicator should state exactly where the divergence occurred.
+
+A replication that says "I tried it and it worked" without stating what exactly was run and what was confirmed is not a valid replication.
+
+## Reviewer Reputation
+
+Reputation is earned by:
+- Accurate rejections that prevent false claims from becoming canonical
+- Clear, specific, actionable reviews that submitters can act on
+- Identifying failure modes the submitter missed
+
+Reputation is not earned by:
+- Volume of reviews
+- Approving submissions quickly
+- Never rejecting
+
+The contribution ledger is git log and the merged PR history. There is no separate scoreboard.

schemas/agent-submission.schema.json

@@ -2,6 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://open-problem-lab.local/schemas/agent-submission.schema.json",
   "title": "Agent Submission",
+  "description": "A structured submission from an AI agent or human contributor. One submission covers one task, one role, one claim. Submissions that mix roles or make multiple independent claims will be returned for splitting.",
   "type": "object",
   "additionalProperties": false,
   "required": [
@@ -19,10 +20,12 @@
   "properties": {
     "task_id": {
       "type": "string",
-      "minLength": 3
+      "minLength": 3,
+      "description": "The exact id field from tasks.json for the task this submission addresses. Example: source-inventory"
     },
     "agent_role": {
       "type": "string",
+      "description": "The role guide used for this submission. Pick one and stay in it. See agents/ directory for each role's requirements and failure modes.",
       "enum": [
         "literature-scout",
         "data-cleaner",
@@ -33,22 +36,27 @@
     },
     "claim": {
       "type": "string",
-      "minLength": 20
+      "minLength": 20,
+      "description": "One specific, falsifiable statement. State what can be proven, at what grain, with what evidence. A claim that cannot be shown false is not a claim. Do not include what the source cannot prove — put that in failure_modes or limitations."
     },
     "evidence": {
       "type": "array",
+      "description": "List of evidence record IDs from evidence.json that support this claim. Every evidence ID here must exist in the relevant pack's evidence.json before this submission is merged.",
       "items": {
         "type": "string",
-        "minLength": 3
+        "minLength": 3,
+        "description": "An evidence record ID from evidence.json. Example: who-malaria-fact-sheet-2024"
       },
       "minItems": 1
     },
     "method": {
       "type": "string",
-      "minLength": 10
+      "minLength": 10,
+      "description": "How you derived the claim from the evidence. A reviewer should be able to repeat your reasoning and arrive at the same conclusion — or name the step where they disagree."
     },
     "reproducibility_steps": {
       "type": "array",
+      "description": "Ordered steps a reviewer or replicator can follow to verify this submission independently. Each step should be specific enough to execute without asking the submitter for clarification.",
       "items": {
         "type": "string",
         "minLength": 5
@@ -57,6 +65,7 @@
     },
     "assumptions": {
       "type": "array",
+      "description": "Conditions that must be true for the claim to hold. State them explicitly. A hidden assumption is a failure mode waiting to be found by a reviewer.",
       "items": {
         "type": "string",
         "minLength": 5
@@ -65,6 +74,7 @@
     },
     "failure_modes": {
       "type": "array",
+      "description": "Specific ways this claim could be wrong, misleading, or harmful if applied. Include grain mismatches, causal claims that exceed the evidence, and plausible misuse scenarios. 'Results may vary' is not a failure mode.",
       "items": {
         "type": "string",
         "minLength": 5
@@ -73,11 +83,13 @@
     },
     "confidence": {
       "type": "string",
+      "description": "Your confidence that the claim accurately represents what the evidence proves at the stated grain. Not confidence that the claim is true in the world. high = strong evidence, clear method, low risk of error. medium = adequate evidence with stated limitations. low = preliminary, needs more evidence or verification. unknown = cannot assess.",
       "enum": ["high", "medium", "low", "unknown"]
     },
     "suggested_next_issue": {
       "type": "string",
-      "minLength": 5
+      "minLength": 5,
+      "description": "The logical next step after this submission. Required field — state what should happen next even if no issue will be opened. Do not open a new GitHub issue unless the task explicitly asks for it. Example: 'Obtain district-level dengue surveillance data from NCDC to begin baseline construction.'"
     }
   }
 }