P0: Build automations conformance, chaos, SLO, and operator diagnostics

[BunsDev]

Parent program: #854
Depends on: #855, #856, #857
Foundation evidence: #816

Outcome

Create an executable, implementation-independent certification plane for Coven Automations v1. A release must prove—not merely claim—that schedule semantics, state transitions, identity/authority binding, crash recovery, duplicate prevention, privacy controls, and client compatibility hold at the exact candidate revision and in the packed artifacts users install.

The same work must give operators safe diagnostics for missed, duplicated, stuck, unauthorized, stale-identity, repeatedly failing, cancelled, ambiguous, or delivery-failed automations without editing SQLite directly.

Conformance architecture

Add a standalone runner and versioned vectors, for example:

conformance/automations/
  manifest.json
  schemas/
  vectors/
    definitions/
    schedules/
    state-machines/
    idempotency/
    events/
    identity-authority/
    receipts/
    privacy/
  scenarios/
    restart/
    duplicate-process/
    runtime-failures/
    delivery-failures/
  runner/
  reports/

Requirements:

• vectors are consumable without linking Coven private modules;
• every vector declares profile/version, prerequisites, input, expected events/state/receipt, and refusal behavior;
• the runner can target an in-process implementation, a daemon endpoint, or a packaged release;
• output is machine-readable and includes exact source/release revisions and artifact digests;
• failures identify invariant, object IDs, event cursor, expected/observed state, and safe reproduction command;
• reports redact secrets, raw credentials, private memory, unrestricted prompts, and irrelevant paths.

Conformance profiles

Do not collapse all confidence into one compliant bit. Publish at least:

1. Structural — schema, canonicalization, state-machine, and compatibility validity.
2. Scheduler Reliability — time, timezone/DST, occurrence fencing, misfire, overlap, retry, cancellation, lease, crash/restart, and backpressure.
3. Runtime Authority — authenticated principal, familiar embodiment, capability/approval decision, runtime descriptor, and fail-closed dispatch.
4. Continuity — exact familiar root/revision and historical rehydration/correlation semantics.
5. Privacy — access control, minimization, redaction, retention, erasure/tombstone, and changefeed projection.
6. Interoperability — SDK/Cave/Psyche/runtime clients consuming pinned artifacts and replaying events correctly.
7. Full — all required v1 profiles at one immutable compatibility set.

Every result must name exactly which profile and artifact versions passed.

Required test inventory

Unit and property tests

• definition/schema parsing and normalization;
• supported/unsupported RRULE vocabulary;
• state-machine legal/illegal transitions;
• occurrence fence uniqueness;
• request adoption/idempotency conflict behavior;
• terminal-state monotonicity;
• bounded logs/events/receipts;
• canonical digest and receipt verification;
• event reducer determinism under duplicate delivery;
• randomized operation sequences preserving invariants.

Deterministic virtual-time tests

• daily/weekly/multiple-hour schedules;
• UTC and IANA timezones;
• DST gap/fold;
• leap day, year/month boundary, and host timezone change;
• clock jumps and suspend/resume;
• startup immediate reconcile and definition-change wake;
• latest-only misfire and future explicit policies;
• backoff/jitter, retry exhaustion, quarantine, and recovery.

Crash/restart and fault injection

Inject failure before/after every durable and external boundary:

• planning, claim, adoption, dispatch, session creation, first event, terminal observation, settlement, delivery, receipt, and event publication;
• process kill and daemon restart;
• SQLite busy/lock/I/O/corruption simulation;
• disk full and permission loss;
• runtime timeout/unavailable/lost/duplicate/out-of-order response;
• identity/authority service unavailable or stale;
• approval expiry/revocation race;
• client disconnect/reconnect and cursor replay.

Duplicate and fencing tests

• two scheduler processes against one store;
• stale scheduler leader;
• stale occurrence claimant;
• duplicate manual command with one adoption key;
• duplicate runtime callback/event;
• retry after ambiguous response;
• stale client update against a newer definition revision.

Security, authority, and privacy tests

• forged familiar/principal/approval/runtime binding;
• nonce and command replay;
• capability escalation and confused deputy;
• stale/revoked identity or authority;
• TOCTOU between decision and dispatch;
• forged lifecycle/event/receipt state;
• unauthorized history/changefeed reads;
• secret/prompt/memory/path leakage in logs and errors;
• retention/redaction/erasure with minimum audit evidence retained.

Cross-repository canaries

Pin immutable producer artifacts and prove:

• SDK generated types and verifier accept all valid vectors and reject invalid ones;
• Cave projections/reducers render duplicate/reconnect/recovery states truthfully;
• Psyche adapter preserves automation/occurrence/run/attempt and binding correlation;
• Familiar Contract and Threads vectors bind exactly as expected;
• runtime descriptors/capabilities are selected and pinned correctly;
• released/packed artifacts work without source-relative imports.

Golden end-to-end scenarios

At minimum:

1. paused creation and explicit activation;
2. twice-daily familiar routine;
3. daemon restart before due time;
4. daemon restart after claim;
5. sleep through multiple occurrences with latest-only misfire;
6. DST spring gap;
7. DST fall fold;
8. duplicate scheduler process;
9. run-now command replay;
10. runtime unavailable before launch;
11. runtime lost after possible side effect;
12. timeout and cancel race;
13. delivery commit failure after successful runtime outcome;
14. revoked familiar revision;
15. approval required/approved/consumed;
16. approval expired or replayed;
17. runtime capability downgrade;
18. repeated retryable failure and quarantine;
19. event subscriber disconnect/replay with duplicates;
20. legacy Codex import remains paused and source-preserving;
21. retention/redaction without broken receipt verification;
22. direct Coven and Psyche-orchestrated runs sharing one binding/receipt contract.

Load and SLO certification

Define a supported local profile and measure at least:

• definitions and due occurrences per minute;
• planning, claim, dispatch, and start latency distributions;
• scheduler pass duration;
• queue depth and oldest eligible age;
• active runs and runtime admission pressure;
• SQLite write contention and busy retries;
• event/changefeed throughput and replay latency;
• database/log/receipt growth over retention windows;
• restart recovery latency;
• CPU and memory at idle and under load.

Ratify release SLOs from measured baselines. The gate must include zero duplicate dispatches for a single fence, zero silent eligible-occurrence loss, zero false success under injected failures, and bounded recovery/resource growth.

Operator diagnostics

Add a stable diagnostic surface through CLI and control actions, with read-only behavior unless an explicit safe operation is requested:

coven automations doctor
coven automations status
coven automations explain <automation-id>
coven automations occurrence <occurrence-id>
coven automations run <run-id>
coven automations attempts <run-id>
coven automations leases
coven automations events --after <cursor>
coven automations schedule --from ... --to ... --dry-run
coven automations reconcile <run-id> --dry-run
coven automations retry <run-id> --expected-state ...
coven automations cancel <run-id> --reason ...

doctor should report:

• daemon/scheduler leadership and last successful pass;
• store schema/integrity and migration state;
• due/eligible/claimed/running/recovery-required counts;
• expired/stale leases and fences;
• planning/start lag;
• repeatedly failing/quarantined routines;
• unresolved authority/identity/approval refusals;
• delivery and receipt failures;
• event cursor/changefeed health;
• disk/retention pressure;
• exact safe next steps.

No diagnostic should recommend deleting rows or blindly rerunning ambiguous mutating work.

Agent-readiness and CI interface

Integrate with the organization contract:

./scripts/agent-bootstrap
./scripts/agent-check fast
./scripts/agent-check full
./scripts/agent-check automations-conformance

• fast covers deterministic unit/schema/state vectors without network or ambient credentials.
• full covers integration, restart, packed artifacts, and all locally supported profiles.
• release certification runs chaos, load/SLO, cross-repository canaries, privacy/security, and exact-artifact verification.
• unsupported platform checks are reported separately from failures.
• all commands leave the worktree clean and emit machine-readable receipts.

Evidence packet

Every automation-affecting PR must include:

• objective, acceptance criteria, and non-goals;
• canonical contracts/artifact revisions consulted;
• lifecycle/authority/privacy impact;
• exact tests and results;
• fault/crash points exercised;
• migration and rollback;
• performance/SLO delta;
• generated artifacts and provenance;
• cross-repository canaries;
• unresolved uncertainty.

The final release packet additionally pins source commit, dependency lockfiles, packed artifact digests, conformance runner/vector versions, environment matrix, and all profile results.

Acceptance criteria

• Independent conformance runner and versioned vectors exist.
• Structural, scheduler, authority, continuity, privacy, interoperability, and full profiles are reported separately.
• All golden scenarios run deterministically and produce machine-readable evidence.
• Crash/fault/duplicate/security/privacy tests prove no silent loss, duplicate dispatch, false success, or unauthorized execution.
• Load/SLO gates are ratified and pass on the exact release artifact.
• Operator diagnostics explain every supported unhealthy state without raw database mutation.
• SDK, Cave, Psyche, Familiar Contract, Threads, and runtime canaries pin immutable artifacts.
• Canonical agent bootstrap/check commands and PR evidence packets are enforced.

Non-goals

• Claiming legal/security certification beyond the tested profiles.
• Replacing normal unit/integration tests with one end-to-end suite.
• Hiding unsupported states behind a generic health badge.
• Auto-repairing ambiguous external side effects.

Bead packet

Create one P0 certification Bead mapped exactly to this issue. It depends on #855, #856, and #857 and blocks every v1 release/public reliability claim. Child tasks may partition vectors, runner, diagnostics, chaos, load, security/privacy, and cross-repo canaries, but completion requires one exact-release evidence packet.