Skip to content

[front] fix: keep the reaper from sleeping a sandbox with an exec in flight - #30197

Open
fontanierh wants to merge 4 commits into
mainfrom
fn-reaper-exec-guard
Open

[front] fix: keep the reaper from sleeping a sandbox with an exec in flight#30197
fontanierh wants to merge 4 commits into
mainfrom
fn-reaper-exec-guard

Conversation

@fontanierh

@fontanierh fontanierh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Description

The lifecycle lock serializes sandbox transitions but has never covered a running exec, so the reaper can pause a sandbox while an exec is still committing pod-state writes. The sleep flow flushes SQLite state to the GCS replica, pauses at the provider, then flips status — and an exec whose writes land after the flush loses them silently, because sleepers are later destroyed without a re-flush. Flagged during review of #30194, which widened admission into this window; the window itself predates it.

Fix: exec-activity counters in Redis, consulted by the flows that flush and then transition. SandboxResource.exec brackets every workload exec with a monotonic started increment and a matching finished one, each record an atomic MULTI that refreshes both keys' TTLs so the pair cannot drift apart (execRoot is excluded: root commands are lifecycle plumbing, including the pre-sleep flush itself). dangerouslySleepIfRunning and the kill sweep's dangerouslyDestroyIfKillRequested — where losing the race is permanent, since a destroyed sandbox is never re-flushed — skip their transition when an exec is in flight before the flush, and re-check after it — started being monotonic catches even an exec that began and finished entirely during the flush, which an in-flight gauge alone would miss (and which lastActivityAt cannot signal, being throttled to one write per 30s). Skipping is not an error: status stays running and the reaper retries next cycle, exactly like a pod that was never idle. An unreadable signal fails closed the same way.

The counters are best-effort on the write side (an exec never fails because Redis hiccuped) and both expire 10 minutes after the last record, so a replica that dies mid-exec un-wedges the sleep via TTL rather than a count that never settles. Deferring a kill is safe: a busy pod's next invocation escalates through ensureActive's kill-requested branch, which recreates on access regardless. A residual window remains between the post-flush read and the provider pause — milliseconds against the previous seconds-long flush — and pauseForApproval keeps today's behavior, since aborting an approval pause has UX cost and the approval flow's own exec is blocked waiting.

Tests

sandbox_resource suite: quiet sandbox sleeps with the activity read on both sides of the flush, in-flight exec skips the sleep before flushing anything, an exec that started and finished during the flush skips the pause, unreadable signal fails closed, an in-flight exec defers the kill-sweep destroy, and exec brackets start/end records including when the provider throws. front typechecks clean.

Risk

False positives only delay a sleep by one reaper cycle. A lost finished record blocks sleep for at most the 10-minute TTL. The exec-side Redis cost is two INCR+EXPIRE round trips per exec, off the latency-critical path's measured contention point.

Deploy Plan

Normal deploy. No migration, no flag. Worth watching the reaper's "not sleeping" log lines for a stuck sandbox that never sleeps, which would indicate a leaked in-flight count before its TTL.

@vercel

vercel Bot commented Aug 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
playground Ignored Ignored Preview Aug 10, 2026 2:50pm
storybook Ignored Ignored Preview Aug 10, 2026 2:50pm

Request Review

@dust-agent dust-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coding Rules LGTM \o/

@fontanierh
fontanierh force-pushed the fn-reaper-exec-guard branch from 44c141d to 49456b3 Compare August 10, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant