Skip to content

fix(sandbox): bound paused proxies to node capacity - #1542

Open
0xAlcibiades wants to merge 3 commits into
paradigmxyz:mainfrom
0xAlcibiades:alcibiades/bound-paused-proxy-capacity
Open

fix(sandbox): bound paused proxies to node capacity#1542
0xAlcibiades wants to merge 3 commits into
paradigmxyz:mainfrom
0xAlcibiades:alcibiades/bound-paused-proxy-capacity

Conversation

@0xAlcibiades

Copy link
Copy Markdown

Closes #1541

Bounded retention of paused sandboxes' iron-proxy pods, sized from node capacity. Two commits, the Rust half independently cherry-pickable.

Read this first: what #1502 changed

#1502 made pause() tear the proxy down, so steady-state retention is now zero and this is not the fix for the incident in #1541 on an up-to-date deployment. I would rather say that up front than have you find it in review.

What it still covers:

  1. Upgrade state. Sandboxes paused before feat(k8s): sandbox priority class, pause-proxy teardown, quota-ready init resources #1502 keep their proxy indefinitely; nothing else deletes them. A deployment carrying paused sessions across that upgrade inherits the pod-budget exhaustion with no path out but hand-deletion.
  2. Crash window. pause() patches replicas: 0 then deletes the proxy. A death between the two leaves a paused CR with a live proxy that nothing reclaims.
  3. Regression cover, and the per-node pod-budget logging, which is useful on its own — today the budget is only visible by reconstructing it from FailedScheduling events.

If you read #1502 as making this redundant, say so and I will close it. I think 1 and 2 are real and unaddressed, and 3 is worth having regardless.

Commit 1 — fix(sandbox): bound paused proxies to node capacity

New paused_proxy_retention module in centaur-sandbox-agent-k8s:

  • Censuses steering nodes' status.allocatable.pods and scheduled (non-terminating) pod load, and identifies retained paused proxies: iron-proxy pods whose Sandbox CR is at replicas: 0.
  • Per node, retention holds at most allocatable − load − margin slots; beyond that the longest-paused go first, ordered by the existing centaur.ai/paused-at annotation with a sandbox-id tiebreak so selection is deterministic.
  • Optional absolute cap, which never expands a node's headroom.
  • Eviction deregisters the proxy in iron-control by its durable OID (a pod annotation, so it works across api-rs restarts) and deletes the proxy pod, service and network policies. The Sandbox CR is untouched — resume takes the existing rebuild path, so only the evicted session pays a cold create.
  • Never touches running or warm sandboxes, in-flight resumes (in-process guard set at resume() entry and cleared on every exit path), unorderable pauses, or nodes with an unreadable budget. That last one is fail-open on purpose: evicting against an unknown budget is worse than not evicting.

Commit 2 — feat(chart): paused-proxy retention knobs and node-budget RBAC

Renders the three env vars, adds the read-only ClusterRole/Binding (pods, nodes get/list) that the namespace-scoped sandbox-manager Role cannot provide, documents the knobs, and adds them to values.schema.json.

The split keeps commit 1 cherry-pickable on its own: without the chart half the sweep cannot read nodes, fails open, and evicts nothing.

Testing

10 new tests — 8 on eviction selection (headroom, saturation to zero, cap interaction, cap-never-expands-headroom, unobservable node, tie-breaking, config toggle) and 2 on arg parsing.

cargo test -p centaur-sandbox-agent-k8s -p centaur-api-server: 122 and 56 pass. cargo fmt --all --check and cargo clippy --all-targets -- -D warnings clean. helm lint passes and helm template renders the env block and both RBAC objects.

…node-budget binding

Review fixes for the paused-proxy retention sweep:
- The ClusterRoleBinding's ServiceAccount subject lacked a namespace, which
  the API server rejects for a cluster-scoped binding, leaving the sweep's
  pods+nodes reads ungranted.
- The pod census excluded every iron-proxy pod from per-node load, so the
  proxies of running sandboxes were not counted and the sweep retained more
  than the node headroom allowed. Count all scheduled pods as load and take
  back out only the evictable paused proxies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paused sandboxes retain their iron-proxy pod with no bound against node pod capacity

1 participant