Churn simulator: validate 80% completion at 30% churn over 72h

[jeremymanning]

Description

Per spec T144c and whitepaper Phase 2 requirement: "Over a 72-hour continuous run, 80% of submitted test jobs complete correctly with 30% simulated node churn."

Requirements

• Build configurable churn simulator: random node kill/rejoin at configurable rate
• Integrate with multi-node test harness
• Track job completion rates under churn
• Validate checkpoint/resume works across node failures
• Run for configurable duration (target: 72 hours for Phase 2)
• Report completion rate, data loss events, and recovery metrics

Success Criteria

• Simulator can kill/rejoin nodes at configurable rate (target: 30% churn)
• Job completion rate ≥ 80% at 30% churn over 72 hours
• Zero data loss events during churn
• Checkpoint/resume works correctly across node failures
• Metrics reported: completion rate, recovery time, data loss count

Testing (Principle V)

• Run on 20+ node testbed with real hardware
• 30% churn rate → verify 80% job completion
• Verify zero data loss during churn
• Measure checkpoint/resume latency under various churn rates

[jeremymanning]

Partially addressed by spec 004 / PR #59 — not closing. Statistical churn simulator exists (ChurnSimulator with configurable node count + churn rate). HOWEVER, this is a model, not a real kill-rejoin harness over libp2p. A 72-hour real-hardware churn run has not been performed.

Remaining work tracked here.

[jeremymanning]

Spec 005 progress — PR #61

STATUS: Real kill-rejoin harness script written. 72-hour run deferred.

Done (commit b78970b):

• scripts/churn-harness.sh — real kill-rejoin over libp2p. Spawns N local daemon processes, submits workloads at 1/s, and on a Poisson schedule (computed from --rotation-rate-per-hour) kills and restarts one random node. Writes evidence bundle under evidence/phase1/churn/<ts>/. Default: 1-hour smoke; pass --duration-s 259200 for canonical 72-hour SC-005 evidence run.

Remaining:

• T054 src/churn/simulator.rs refactor to expose the same API backed by the real harness not yet done (statistical simulator unchanged).
• T056 72-hour real-hardware run not yet executed — operator-driven. Can be kicked off in screen session in next session. Also: harness currently runs all nodes on the local machine; extending to SSH-remote spawn across tensor01/tensor02 not yet wired.
• --dry-run flag used in the script for workload submission is invented — needs to be either added to the submitter CLI or swapped for a real flag in next session.

See notes/session-2026-04-19-spec-005-kickoff.md Known Bugs #5 for details.