Pulse cron loop can freeze indefinitely on a held pipe, and the failure breaker never recovers (v7.1.1, present on current main)

[jacobo-ortiz]

Three sites in LifeOS/install/LIFEOS/PULSE/{lib,pulse}.ts share two latent defect classes that can silently take the Pulse cron loop down. All verified against v7.1.1 and current main (both identical at these sites).

1. Spawn timeout does not unblock the loop (lib.ts, three sites)

spawnScript (line 326) and spawnClaude (line 376) both do:

const timer = setTimeout(() => proc.kill("SIGTERM"), timeoutMs)
const output = await new Response(proc.stdout).text()   // can hang forever

The timeout signals the child but the code still unconditionally awaits the pipe read. Two measured reproductions (bun test against the shipped lib.ts, 2,000 ms timeout):

• sleep 300 & exit 0 (a grandchild inherits and holds the stdout pipe): the await hung 300,013 ms, the grandchild's full lifetime.
• trap '' TERM; sleep 300 (child ignores SIGTERM): hung 300,021 ms.

Because the cron loop in pulse.ts runs jobs sequentially, one held pipe freezes every cron job behind it. Observed in production on v5.0.0-era code: a single job logged durationMs = 5283540 (88 minutes) and no cron ran for 2 days; voice and telegram kept working so nothing surfaced. The same pattern also appears in the email dispatch case (lib.ts line 272): SIGTERM plus a bare await proc.exited, with stdout/stderr piped but never drained, so a blocked or signal-immune gws child hangs dispatch too (adjacent to #1537, which covers gws being unshipped; this is the hang pattern itself).

Also note the failure path reads stderr only after the main await, so even with a race around stdout alone, a grandchild holding stderr re-hangs the error branch. The read of both pipes needs to live inside the race.

Proposed fix: race the reads against a timeout that rejects, so the caller always unblocks regardless of orphan or pipe state:

Promise.race([
  Promise.all([stdoutText, stderrText, proc.exited]),  // drain both pipes inside the race
  timeoutThatRejects,                                   // SIGKILL best effort, then reject
])

with clearTimeout in finally. SIGKILL is best effort (not a tree kill): the unblock comes from the reject, not from the child dying. With this shape the same reproductions above reject at roughly 2,001 ms.

2. Circuit breaker never recovers (pulse.ts line 904)

Any job at consecutiveFailures >= MAX_FAILURES (3) is skipped. The only reset to 0 happens on a successful run (line 937), which a skipped job can never reach. A job that trips the breaker (three transient failures: a network blip, a temporarily missing binary) is quarantined permanently, and the state persists across restarts via state/state.json.

Proposed fix: make the breaker half-open. Add a cooldown (for example 30 minutes); within it, skip as today; after it, log a half-open retry and fall through. Success resets to 0 through the existing line 937, failure restarts the cooldown through the existing catch block. No state schema change needed.

Context

• SecurityValidator.hook.ts hangs when stdin doesn't close properly #452 documents the SIGTERM-vs-blocked-IO insight on a different surface (hook stdin).
• Pulse cron scheduler and launchd LaunchAgents don't catch up missed nightly jobs (machine off/asleep at scheduled time) #1513's fixes (isDue catch-up) make missed jobs fire on wake, which slightly raises the odds of hitting site 1 on a flaky first tick after wake.
• Local soak of this fix shape on v5-era code: 25 days, all jobs consecutiveFailures: 0, zero loop freezes.

[danielmiessler]

Fixed in source, and thanks for the measured reproductions. Part 1: all three spawn sites (spawnScript, spawnClaude, and email dispatch) now race pipe-drain plus exit against a hard deadline: SIGTERM at timeout, SIGKILL 10s later, and a rejecting deadline 10s after that which unblocks the loop even if a grandchild still holds the pipe. Verified with a live repro that hung forever before and now resolves at the deadline. Part 2 you'll be glad to know was already fixed after 7.1.1: the breaker has a 6-hour half-open retry cooldown. Both ship with the next release.