Skip to content

Commit 2ce8d20

Browse files
authored
fix(container): kill orphaned in-container process on execute() timeout (#31)
When subprocess.run([docker exec ...], timeout=N) hits TimeoutExpired, Python SIGKILLs the local docker exec CLI. The docker daemon does not propagate that into the container (only SIGINT/SIGTERM propagate gracefully), so the test process spawned inside the container keeps running. With sleep-as-PID-1 cleanrooms it never gets reaped: it just orphans onto PID 1 and keeps burning CPU while the next docker exec call competes for the same resources. On a multi-arm evaluation run over the full task set, this surfaces as: a TUI-style task hits the 3600s outer timeout, returns rc=-1, programbench retries the branch, the retry SIGKILLs another docker exec, the second orphan accumulates alongside the first, and so on. Observed cost on one run: ~2 hr wasted per arm per hang-prone task, plus measurable CPU contention slowing surrounding evals in the same stripe. Fix: on TimeoutExpired, send a follow-up docker exec <c> bash -c "kill -KILL -1; sleep 0.2; kill -KILL -1; true" which signals every non-PID-1 process in the container's PID namespace. Safe here because ContainerEnvironment starts cleanrooms with a long- lived sleep as PID 1; the only non-PID-1 processes are the test subprocess tree we just spawned. Bounded with a 30s timeout on the kill exec itself, and warns if it fails so the failure is debuggable instead of silent. Behavior on successful runs is unchanged.
1 parent 31952f0 commit 2ce8d20

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

‎src/programbench/container.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,37 @@ def execute(self, command: str, *, timeout: int | None = None) -> dict[str, Any]
8989
"exception_info": "",
9090
}
9191
except subprocess.TimeoutExpired:
92+
# When subprocess.run hits its host-side timeout, Python SIGKILLs
93+
# the local `docker exec` CLI. The docker daemon does not
94+
# propagate that into the container (only SIGINT/SIGTERM
95+
# propagate gracefully), so the process spawned inside the
96+
# container keeps running. With sleep-as-PID-1 cleanrooms it
97+
# never gets reaped and competes with the next docker exec call.
98+
#
99+
# Sweep the in-container processes by SIGKILL-ing every non-
100+
# PID-1 process. Safe here because the cleanroom's only PID-1
101+
# process is the long-lived `sleep` that keeps the container
102+
# alive; everything else is the test subprocess tree we just
103+
# spawned. Bounded with a short timeout so a stuck daemon
104+
# cannot wedge us a second time.
105+
try:
106+
subprocess.run(
107+
[
108+
self.executable,
109+
"exec",
110+
self.container_id,
111+
"bash",
112+
"-c",
113+
"kill -KILL -1 2>/dev/null; sleep 0.2; kill -KILL -1 2>/dev/null; true",
114+
],
115+
capture_output=True,
116+
text=True,
117+
timeout=30,
118+
)
119+
except Exception as e:
120+
log.warning(
121+
"in-container teardown after execute() timeout failed: %s", e
122+
)
92123
return {
93124
"output": "",
94125
"returncode": -1,

0 commit comments

Comments
 (0)