P0: Harden automation time, retries, cancellation, fencing, and crash recovery

[BunsDev]

Parent program: #854
Protocol dependency: #855
Foundation: #816

Outcome

Make the Coven automation scheduler deterministic, restart-safe, bounded, and diagnosable under real operating conditions: sleep/wake, clock changes, daylight-saving transitions, daemon restart, duplicate processes, slow or unavailable runtimes, cancellation races, delivery failure, and ambiguous side effects.

The completion criterion is not “the 60-second thread fired.” It is that every eligible occurrence reaches one explainable durable disposition without silent loss, duplicate execution, false success, or an unrecoverable stale lease.

Current implementation baseline

The landed scheduler already provides valuable foundations:

• SQLite occurrence fencing;
• claim leases and expired-lease recovery;
• latest-only misfire behavior;
• overlap refusal;
• daemon tick and scheduled dispatch;
• shared manual/scheduled runtime launch;
• run ledger and atomic output delivery.

The remaining reliability contract is incomplete:

• scheduling is driven by a fixed sleeping thread and wall-clock reads;
• startup does not have a separately specified immediate-reconciliation contract;
• definition changes do not yet have a canonical wake/replan mechanism;
• local timezone is not an immutable IANA timezone binding;
• DST gap/fold and host-timezone-change semantics are not published;
• retries/backoff/jitter and failure classification are not first-class;
• cancellation acknowledgement and cancellation/completion races are not fully modeled;
• scheduler leadership/fencing for more than one process is not specified;
• shutdown/drain behavior and stuck-runtime recovery need explicit state transitions;
• virtual-time, crash-point, chaos, and sustained-load certification are absent.

Time abstraction

Introduce an injected scheduler clock and wake source used by production and tests:

trait AutomationClock {
    fn now_utc(&self) -> DateTime<Utc>;
    fn monotonic_now(&self) -> InstantLike;
    fn sleep_until_or_wake(&self, deadline: InstantLike, wake: &WakeSignal) -> WakeReason;
}

Requirements:

• production waits use a monotonic deadline recalculated from authoritative wall time;
• tests use deterministic virtual time with no wall-clock sleeps;
• creation/update/activation/pause/cancel and daemon shutdown can wake the scheduler;
• scheduler startup performs an immediate reconcile before the first timed wait;
• every pass records scheduled start, actual start, duration, planned/claimed/dispatched/recovered counts, and error class;
• a backward/forward wall-clock jump produces a recorded replan rather than duplicate or permanently delayed work;
• no correctness assertion depends on a sub-second timing margin on shared CI.

Timezone and DST contract

Replace ambiguous durable local semantics with an explicit versioned timezone binding:

• accept UTC and IANA TZID in the stable contract;
• treat local as compatibility input resolved to an exact IANA zone at activation/import where the platform can prove it;
• record the timezone database/version used for planning where practical;
• changing host timezone must not silently reinterpret an active definition;
• changing a definition timezone creates a new definition revision and replans only future occurrences.

Ratify and vector-test safe v1 defaults:

• nonexistent local wall time during a spring-forward gap is skipped with a durable reason rather than silently shifted;
• repeated wall time during a fall-back fold produces one occurrence under an explicit deterministic fold rule, with the chosen offset recorded;
• leap-day, month/year boundary, end-of-month, and supported weekly rules are deterministic;
• unsupported RRULE vocabulary fails at validation rather than being approximated;
• system sleep and daemon downtime use the declared misfire policy, not one event per missed minute.

Alternative gap/fold policies may be added later as explicit versioned variants; they must never be inferred.

Retry and failure classification

Define failure classes before adding automatic retry:

• validation/contract failure — never retry;
• identity/authority/capability refusal — never retry without new authorization evidence;
• scheduler/store transient — retry scheduler operation, not the familiar action;
• runtime admission/unavailable before side effect — bounded retry allowed;
• runtime started with known failed outcome — retry only when action policy permits;
• ambiguous runtime/side-effect outcome — recovery_required, no automatic retry;
• delivery commit failure after successful execution — retry idempotent delivery only, never rerun the familiar action;
• cancellation/timeout — explicit policy, never silently converted to retry.

Implement bounded exponential backoff with deterministic full jitter in production and seeded deterministic jitter in tests. Persist retry policy, notBefore, attempt number, failure class, and exhaustion reason. Add a circuit/quarantine state for repeated failures so a broken routine does not create an unbounded storm.

Cancellation and timeout

• Model cancellation as requested → acknowledged/reconciled → terminal.
• Persist who requested cancellation, when, scope, and reason.
• Fence cancel against the exact run/attempt and current runtime correlation.
• Define completion-vs-cancel and timeout-vs-completion races with one convergent terminal result.
• A runtime that cannot confirm stop enters recovery-required or a narrowly defined timeout terminal state; it is not assumed stopped.
• Disabling/deleting a definition prevents future planning but does not rewrite an in-flight run.
• Daemon shutdown has a bounded drain policy and records every active attempt left for restart reconciliation.

Lease, leadership, and fencing

Even for a local-first v1, prevent accidental duplicate schedulers:

• use a scheduler-leader lease or equivalent single-writer guard in the Coven store;
• issue monotonic fencing generations for scheduler ownership and occurrence claims;
• require the current fence on every state transition and settlement;
• reject stale workers/processes after lease transfer;
• prove two daemon processes sharing one store cannot both dispatch the same occurrence;
• define what remains unsupported for network filesystems or multi-host shared SQLite.

Multi-host routing remains P2; this issue prevents unsafe accidental concurrency without pretending SQLite is a distributed consensus system.

Crash/restart matrix

Inject a crash or process stop after each consequential boundary:

1. definition revision commit;
2. occurrence planning;
3. claim commit;
4. command adoption;
5. runtime dispatch request;
6. runtime session creation;
7. first runtime event;
8. runtime terminal observation;
9. run settlement;
10. output/delivery temp write;
11. delivery rename/commit;
12. receipt commit;
13. event/changefeed publication.

For each point, restart must converge to exactly one durable, explainable disposition. Missing evidence must never become success, and an ambiguous mutating action must never be replayed automatically.

Backpressure and retention

• Bound eligible/claimed work per pass.
• Bound global and per-routine concurrency.
• Bound runtime admissions and delivery work.
• Avoid holding SQLite writer transactions during runtime/network waits.
• Index due, lease-expiry, active-run, history, and event queries.
• Bound logs/events/receipts and enforce retention without breaking required audit/recovery evidence.
• Report queue depth, oldest eligible age, planning lag, claim lag, start lag, active count, and database contention.

Operator controls

Provide safe commands/actions for:

• scheduler status and last pass;
• due/eligible/claimed/running/recovery-required lists;
• inspect occurrence/run/attempt/lease/fence;
• explain next occurrence and timezone decision;
• pause/resume without deleting history;
• request cancellation;
• reconcile one ambiguous run;
• retry an explicitly retryable failed attempt;
• quarantine/unquarantine;
• dry-run schedule evaluation over a time range;
• database integrity and retention diagnostics.

No command may mutate raw lifecycle rows directly.

Verification

Deterministic tests

• virtual-time next/due computation;
• wake-on-definition/cancel/shutdown;
• immediate startup reconcile;
• UTC and IANA timezone vectors;
• DST gap/fold vectors;
• clock forward/backward jumps;
• sleep/wake and latest-only misfire;
• seeded backoff/jitter;
• retry exhaustion/quarantine;
• cancellation races;
• lease renewal/expiry/fencing;
• duplicate scheduler processes;
• delivery-only retry;
• bounded batch/backpressure.

Integration and chaos

• kill/restart at every crash boundary;
• SQLite busy/locked/transient I/O failures;
• runtime unavailable, slow, duplicate, lost, and out-of-order observations;
• daemon shutdown with active work;
• corrupt/unreadable definition or event data;
• disk full during run/delivery/receipt commit;
• sustained due-occurrence load and history retention.

Proposed SLO gates

Ratify exact targets from measured baselines, but require at minimum:

• zero duplicate dispatches for one occurrence fence in certification;
• zero silent missed eligible occurrences;
• bounded p99 planning/start lag under the supported local load profile;
• bounded recovery time after daemon restart;
• bounded database growth and scheduler pass duration;
• zero false-success outcomes under injected failures;
• zero unbounded retry, log, event, or receipt growth.

Acceptance criteria

• Production scheduling uses the injected clock/wake abstraction; tests use virtual time.
• Startup, shutdown, definition changes, clock jumps, sleep/wake, timezone, and DST semantics are documented and executable.
• Retry is failure-class-aware, bounded, persisted, and refuses ambiguous or unauthorized replay.
• Cancellation and timeout races converge to one authoritative state.
• Scheduler and occurrence fencing prevent duplicate dispatch by competing local processes.
• The complete crash matrix passes without silent loss, duplicate execution, or false success.
• Operator commands diagnose and safely reconcile every supported nonterminal/stuck state.
• Load/SLO and retention evidence is attached to the exact release candidate.

Non-goals

• Active-active distributed scheduling.
• Multi-host consensus over SQLite.
• Automatically retrying unknown external side effects.
• Adding every RRULE feature or timezone policy in v1.

Bead packet

Create one P0 Bead mapped exactly to this issue. It depends on #855 for normative states/errors and the #816 foundation. It blocks conformance certification and production-ready claims. Split implementation into test-first child tasks for clock/timezone, retries, cancellation, leader/fencing, crash matrix, diagnostics, and load/retention; retain this issue as the public outcome.

[BunsDev]

Execution started.

• Canonical Cave Bead: cave-vfz8o (P0, shared surface, exact external reference to P0: Harden automation time, retries, cancellation, fencing, and crash recovery #856)
• Coven claim: issue-856
• First isolated slice: feat/856-automation-clock-wake from 8a796807b37d4ad33eaeca37498debf1ca55dd49

Dependency-ordered implementation plan retained in this parent outcome:

1. injected clock/wake + immediate startup reconcile;
2. explicit restart reconciliation ordering;
3. persisted retry/failure classification;
4. cancellation convergence;
5. scheduler leadership/fencing;
6. crash matrix;
7. backpressure/retention;
8. operator diagnostics and exact-candidate load/SLO evidence.

The first slice is strictly Rust scheduler authority and consumes #855. It will not duplicate #857 principal/familiar/approval/receipt authority.