wcore-swarm::heartbeat_test worker_writes_heartbeat_during_long_running_task is now the single most frequent false red on the merge test gate. It has reddened three separate merges on 2026-07-31, each time costing a diagnosis plus two confirming re-runs (~6 minutes of gate time per incident), and each time it was innocent — the merging lanes touched zero wcore-swarm files.
It is not a defect in the product and not a defect in the test's logic. It is starvation.
The test polls every 40ms and requires ≥3 distinct heartbeats from a worker emitting them 150ms apart over a ~450ms lifetime. Miss enough 40ms ticks and the observed count comes up short even though the worker behaved perfectly.
Measured:
| condition |
result |
wcore-swarm alone, --profile ci, 3 reps |
150/150 passed, ~2.2s each |
| the heartbeat test alone, 16 consecutive runs |
16 passed (incl. under 8 background cargo check jobs) |
| full workspace (13,701 tests, 592 binaries) |
fails intermittently, all 3 attempts of retries = 2 exhausted |
threads-required = 4 on package(wcore-swarm) (landed 2026-07-31) reduced but did not eliminate it: the rate went from 2-of-3 full runs red to roughly 1-in-4.
Why this matters more than a flaky test normally would
The merge test gate is now a required step in the cadence. A gate with a ~25% false-red rate is a gate people learn to re-run until it goes green — at which point it has stopped being a gate. That is the exact failure mode this project treats as equivalent to a gate that cannot fail.
The fix, and the trap to avoid
The sibling defect in this crate was root-caused and fixed properly: status_output_cap_kills_git_descendant was reading a PID file that > had created but printf had not yet written, and it now polls for a parseable value with a bounded deadline. That fix took its test from flaky to 0-of-4 red.
This one needs the equivalent: stop sampling for heartbeats and instead wait deterministically for the worker to have written N of them, then assert monotonicity from the recorded values.
Do not "fix" it by loosening >= 3 to >= 1, or by widening the poll interval until the assertion cannot fail. The test exists to prove a worker emits progressive liveness during a long task; an assertion that passes on a single heartbeat proves nothing and would be worse than the flake.
wcore-swarm::heartbeat_test worker_writes_heartbeat_during_long_running_taskis now the single most frequent false red on the merge test gate. It has reddened three separate merges on 2026-07-31, each time costing a diagnosis plus two confirming re-runs (~6 minutes of gate time per incident), and each time it was innocent — the merging lanes touched zerowcore-swarmfiles.It is not a defect in the product and not a defect in the test's logic. It is starvation.
The test polls every 40ms and requires ≥3 distinct heartbeats from a worker emitting them 150ms apart over a ~450ms lifetime. Miss enough 40ms ticks and the observed count comes up short even though the worker behaved perfectly.
Measured:
wcore-swarmalone,--profile ci, 3 repscargo checkjobs)retries = 2exhaustedthreads-required = 4onpackage(wcore-swarm)(landed 2026-07-31) reduced but did not eliminate it: the rate went from 2-of-3 full runs red to roughly 1-in-4.Why this matters more than a flaky test normally would
The merge test gate is now a required step in the cadence. A gate with a ~25% false-red rate is a gate people learn to re-run until it goes green — at which point it has stopped being a gate. That is the exact failure mode this project treats as equivalent to a gate that cannot fail.
The fix, and the trap to avoid
The sibling defect in this crate was root-caused and fixed properly:
status_output_cap_kills_git_descendantwas reading a PID file that>had created butprintfhad not yet written, and it now polls for a parseable value with a bounded deadline. That fix took its test from flaky to 0-of-4 red.This one needs the equivalent: stop sampling for heartbeats and instead wait deterministically for the worker to have written N of them, then assert monotonicity from the recorded values.
Do not "fix" it by loosening
>= 3to>= 1, or by widening the poll interval until the assertion cannot fail. The test exists to prove a worker emits progressive liveness during a long task; an assertion that passes on a single heartbeat proves nothing and would be worse than the flake.