ST4 worker: follow-ups deferred during review

[pdettori]

Follow-ups from the ST4 worker (#87, PR #172) that were seen, judged, and deliberately
not fixed. None affects the acceptance battery or runtime correctness on the paths the
harness exercises today.

Filed as its own issue rather than a note in the repo, because an in-repo list went stale
within one commit: it was generated from an implementation ledger that recorded findings
as they were raised and never reconciled against the code after the final review wave,
so it advertised three blockers that were already fixed. Everything below was audited
against feat/st4-go-worker at review time.

Item numbers are stable and never reused. Resolved items move to Resolved at the
bottom keeping their original number, because commit messages and a test comment
(packages/knative-server/test/worker-deployment.test.ts) cite them as "#173 item N".
Last reconciled against main at 1b5d4ef, 2026-09-06 — each open item below was
re-verified in the tree at that commit, not just re-read.

Worth doing, in rough priority order

• 1. A dropped refusal leaves an exec unanswered. The receive goroutine sends through a
  non-blocking trySend so it can never stall and miss an Abort — but both frames it
  sends are terminal, and the busy: queue full refusal is uncached, so dropping it
  loses it outright. Worse, the drop correlates with the condition that produces it:
  outbound fills when the pool is saturated, which is when the queue fills. Under
  overload the caller gets nothing and waits out its own deadline — the opposite of the
  design's stated intent. Restructuring frame egress (a priority channel, or a dedicated
  terminal-frame forwarder with its own bounded queue) was declined as too much change
  for the end of a 21-commit branch. The drop is now logged as a warning naming the
  unanswered req_id, so it is at least diagnosable.
  Still open at remote-worker/internal/session/loop.go:145-159, which now documents the
  hazard in a comment but does not restructure frame egress.

• 2. gRPC MaxCallRecvMsgSize is unconfigured, so a base64 write payload above the
  4 MiB default kills the whole stream rather than one exec, triggering a reconnect.
  Needs a decision on both ends plus a documented max write size — relay-side
  coordination, which is why it is not in the PR.
  Still open: no message-size option is set anywhere in remote-worker/ or
  packages/sandbox-relay/.

• 4. Unify the session's state plumbing. abortReq/finish close over inflight/mu
  while recvLoop/accept take the map and a *sync.Mutex as positional parameters —
  seven parameters, two of them raw synchronization primitives. For a file whose central
  safety property is "every access is under mu", one convention would make that locally
  checkable instead of requiring a whole-file audit. A small struct with methods collapses
  both signatures. Wanted specifically because this is the reference implementation
  other language ports get written against.
  Still open at loop.go:248 (recvLoop) and loop.go:278 (accept). Pure refactor —
  best folded into whichever change next touches that file.

• 5. Most tests never join the Serve goroutine. Two of them observe teardown; the rest
  launch Serve and end without waiting, so a teardown deadlock or a wait group that
  never reaches zero would leave a hung goroutine while the suite reported PASS.

Smaller

• 6. Drain-watchdog grace is wall-clock, not activity-based. After a timeout fires, a
  drain that is merely slow (not wedged) can be force-closed between reads, dropping
  legitimate trailing output. Narrow — the run is already out of budget — and it is the
  remedy shape the review itself proposed. Resetting the timer on read activity closes it.

• 8. No //go:build unix constraint. syscall.Setpgid/Kill break a Windows build
  with a compile error rather than a clear "unsupported platform" message. The worker is
  a Linux container, so no second implementation is needed — just the tag.

• 9. settle() is a fixed sleep in an otherwise poll-based test file. Justified: there
  is no exported observable for "the inflight slot has been released". An exported test
  hook would let it become a condition wait.

• 10. The images install findutils, which none of the harness's operations use.
  Still present in remote-worker/Dockerfile and remote-worker/Dockerfile.runtime.

Resolved

• 3. SANDBOX_TOKEN was a literal env value in worker-deployment.yaml, filled by
  sed in deploy-incluster.sh, so it landed in the Deployment spec, oc describe, and
  any GitOps mirror. Done in PR fix: open-issue triage, first session (#190, #191, #192, #182, #173 item 3) #226 (e33bf3e and follow-ups): the token now arrives
  via secretKeyRef on the worker Deployment, the OCP overlay, and both deploy scripts,
  and both relay and worker are restarted after a token rotation, since env from a
  secretKeyRef is resolved only at pod start. worker-example.yaml keeps its literal
  dev-token on purpose. Guarded by
  packages/knative-server/test/worker-deployment.test.ts, which fails if the literal
  returns or if the secretKeyRef stops matching the Secret name/key the deploy script
  creates.

• 7. The memory coupling is documented but unenforced. BufferCap × 2 streams ×
  MaxConcurrent must fit the pod limit; both sides carried the arithmetic in a comment,
  but nothing stopped a future WORKER_MAX_CONCURRENT entry in the Deployment from
  invalidating the 256Mi limit silently. Done before this issue's first triage pass
  (65160bf, strengthened by a1ba25b, both 2026-08-28): the same
  worker-deployment.test.ts reads BufferCap from runner.go, DefaultConcurrency from
  loop.go, and the manifest's WORKER_MAX_CONCURRENT override if one exists, then
  asserts the pod memory limit covers 2 × concurrency × BufferCap. It throws loudly
  rather than skipping if a constant is reformatted out of reach.

Platform note, not a follow-up

TestRunReturnsWhenPipeHolderEscapesGroup needs setsid to detach a pipe holder from the
process group, so it skips on macOS and runs on Linux CI. It is the test that hangs
rather than fails if the drain watchdog regresses, so its coverage is real — just not on a
developer's Mac.

Assisted-By: Claude Code