You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3bb4263
Browse filesBrowse the repository at this point in the historyBrowse files
worker: drain retirement latch and hand-back retry budget
Plan 6 tasks 1 to 4 as one change set.
- Shell test helper: shell argv tests bound their poll wait and narrow the no-/proc scan.
- CI retry visibility: a retried test step reports its attempts.
- Native exit naming: powershell_runner failures name the native exit code.
- Drain retirement: unreached paths of a retiring drain are handed back without charging a retry, through a handBack hook that drainOnce supplies.
@@ -435,7 +438,7 @@ export function processDirtyBatch(
435
438
if(drainShouldHandBack()){
436
439
// No retry is charged, since nothing failed. Persisted before the caller clears the claimed file that named them, so a crash in between duplicates a path (its sha gate absorbs that) and never drops one.
// Each deferred requeue is also written at once to a sidecar named like a stage (b) fallback claim, because the queue files that named these paths are deleted before the live append below: a kill, or a throw out of stage (b), in between would otherwise lose them. This cycle's stage (a) listed its files before the sidecar existed, so only a later cycle recovers it, and the deferral holds.
// (a) Crash recovery: absorb any `.draining` (and `.draining.alt-*` fallback) files abandoned by a previous crashed or stuck drain. Multiple files can accumulate when a Windows sharing violation keeps the primary `.draining` name locked across cycles (see stage (b)'s fallback-claim comment) -- recover every one of them, not just the first, so a single stuck file can never starve the rest of the queue from ever draining.
// Only process this content if it was not already folded into a batch on a prior cycle (see unclearedDrainingSnapshots below). Without this guard, a draining file that outlives both cleanup attempts (e.g. a persistent Windows sharing violation) would be re-read and its paths reprocessed on every cycle.
// Only clear the recovered file now that its batch has been durably processed (or recognized above as already processed) -- never before -- so a crash partway through processDirtyBatch leaves the file in place for the next startup to recover, instead of deleting it up front and losing every path it named.
563
571
try{
@@ -600,7 +608,7 @@ export function drainOnce(
600
608
}
601
609
if(readOk){
602
610
// Deliberately NOT wrapped in the try above: a throw from processDirtyBatch (e.g. the process crashing mid-batch) must propagate to the caller, not be swallowed as a "read failure", so the cleanup below never runs and the claimed file survives.
// Re-read the claimed file ONE more time, right before deleting it, to catch a path appended by a writer that raced the claim-rename above. The rename-to-claim pattern assumes no other writer can touch the live queue's underlying file once it has been renamed to the claim target -- but that isn't guaranteed on Windows (FILE_SHARE_DELETE lets a concurrent open-for-append follow the same underlying file object across the rename) or on POSIX (a narrow rename/open race exists there too). A dirty path appended during that window would otherwise be silently deleted along with the rest of this cycle's already-processed content and never reindexed. Forward anything found beyond what was already processed back into the live queue so the next drain cycle picks it up.
reason: 'the only production caller is drainOnce, which passes its own index/remove/dir/requeue explicitly; these defaults are reached by direct-call tests only, and drainOnce\'s defaults (covered below) are what ships',
// Provenance: HAND-DERIVED from a macOS CI failure of the test above: macOS has no /proc, and its BSD `ps -e` lists the caller's own processes that have a terminal, with their environment, so a child started with no terminal was invisible to the kin scan, which also ran only where /proc exists. The branch is forced by making the /proc test fail, and `ps` is a function that answers as macOS does: `-wwAE` prints the owned parent with the token, `-e -o ...` prints nothing, and `-A -o ...` is the real listing.
415
-
it('ends an owned child whose environment does not carry the token where there is no /proc and ps -e lists nothing',async()=>{
415
+
it('ends an owned child whose environment does not carry the token where there is no /proc and ps lists only the owned parent and the late child',async()=>{
// The /proc test is forced false, and the environ scan is narrowed to the two pid files, so the row does not pay the whole-table /proc grep under load; the kin listing the row tests is still the real fixture above.
/** A path a retiring drain hands back was never tried, so handing it back must not use up its retry budget. Pinned in-thread: the daemon's pid file goes missing during the first batch's heartbeat, so the remaining paths are unreached, and the live queue path is a directory, so appending them to it fails and each takes the hand-back fallback. */
// The drain's per-path heartbeat runs after the retire decision for that path, so the first call marks the point where the daemon's pid file goes missing mid-batch.
0 commit comments