Repository navigation
Commit 2ce8d20
authored
fix(container): kill orphaned in-container process on execute() timeout (#31)
When subprocess.run([docker exec ...], timeout=N) hits TimeoutExpired,
Python SIGKILLs the local docker exec CLI. The docker daemon does not
propagate that into the container (only SIGINT/SIGTERM propagate
gracefully), so the test process spawned inside the container keeps
running. With sleep-as-PID-1 cleanrooms it never gets reaped: it just
orphans onto PID 1 and keeps burning CPU while the next docker exec
call competes for the same resources.
On a multi-arm evaluation run over the full task set, this surfaces as:
a TUI-style task hits the 3600s outer timeout, returns rc=-1,
programbench retries the branch, the retry SIGKILLs another docker
exec, the second orphan accumulates alongside the first, and so on.
Observed cost on one run: ~2 hr wasted per arm per hang-prone task,
plus measurable CPU contention slowing surrounding evals in the same
stripe.
Fix: on TimeoutExpired, send a follow-up
docker exec <c> bash -c "kill -KILL -1; sleep 0.2; kill -KILL -1; true"
which signals every non-PID-1 process in the container's PID namespace.
Safe here because ContainerEnvironment starts cleanrooms with a long-
lived sleep as PID 1; the only non-PID-1 processes are the test
subprocess tree we just spawned. Bounded with a 30s timeout on the
kill exec itself, and warns if it fails so the failure is debuggable
instead of silent.
Behavior on successful runs is unchanged.1 parent 31952f0 commit 2ce8d20
1 file changed
Lines changed: 31 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
92 | 123 | | |
93 | 124 | | |
94 | 125 | | |
| |||
0 commit comments