feat: multi-session harness multiplexing — N concurrent sessions per process

[cwiklik]

Problem

The harness runs one session per process (KEDA ScaledJob model). At scale this means 1 pod per active session — 10,000 concurrent sessions = 10,000 pods. The harness spends 80-90% of wall-clock time idle, waiting for LLM responses and tool calls, yet each session exclusively owns a full pod.

Root cause

Pi was designed as a single-user CLI agent. runTurn is not concurrent-safe — five categories of global mutable state prevent multiple sessions from sharing a process:

Problem	Location	Severity
process.env.ANTHROPIC_API_KEY mutation per call	harness/src/run-turn.ts	Showstopper — concurrent sessions with different auth tokens corrupt each other
stdoutTakeoverState — global process.stdout.write replacement	pi-fork: coding-agent/src/core/output-guard.ts	Showstopper — one session's capture swallows another's output
sessionResourceCleanups — global Set, cleanupSessionResources() fires ALL	pi-fork: ai/src/session-resources.ts	Showstopper — cleanup for session A tears down session B's resources
fileMutationQueues — global serialization queue	pi-fork: coding-agent/src/core/tools/file-mutation-queue.ts	Race — concurrent sessions serialize through same promise chain
commandResultCache — shell results cached for process lifetime	pi-fork: coding-agent/src/core/resolve-config-value.ts	Stale data — session B reads session A's cached command output

4 of 5 are in pi-fork, 1 is in the harness.

Proposed fix

Each fix is mechanical — replace module-level globals with a per-session context object threaded through the call chain:

1. process.env mutation → set credentials once at startup (or accept as function args, not env). This is the harness-side fix.
2. stdoutTakeoverState → remove entirely (server mode doesn't need stdout capture) or scope per session.
3. sessionResourceCleanups → key the Set by session ID.
4. fileMutationQueues → scope per session (each session gets its own queue).
5. commandResultCache → scope per session or add TTL.

Target architecture

              ┌─────────────────────┐
              │   Harness Pod (×N)  │
              │                     │
              │  session-A ─┐       │
              │  session-B ──┤ Pi   │  N concurrent sessions
              │  session-C ──┤ loop │  per warm process
              │  ...       ─┘       │
              └──────────┬──────────┘
                         │
              ┌──────────┴──────────┐
              │   Sandbox Pool      │  shared, leased
              │   (kubectl exec)    │  unchanged
              └─────────────────────┘

• A fixed/elastic pool of harness pods replaces KEDA ScaledJob (create/destroy per session).
• Each harness process serves ~50-100 concurrent sessions.
• Session state stays in Redis (already external).
• Sandbox pool continues unchanged — sessions lease sandboxes as today.

Impact

Metric	Current (1:1)	Multiplexed (N:1)
Pods for 10K concurrent	10,000	100-200
Activation latency	1.5-3.4s (cold start)	~10-50ms (Redis session load)
Resource for 10K concurrent	~20,000 CPU	~400 CPU

This is the single largest density lever available — roughly 100× improvement in pod count and resource footprint.

Scope

• Introduce a SessionContext type threading session ID + credentials + scoped state
• Fix the 5 global state categories listed above
• Add a concurrent session smoke test (2+ sessions interleaved on one process)
• Validate no cross-session state leakage under load
• Update deployment model from KEDA ScaledJob to elastic pod pool

Related

• Epic: Two-tier FS-free harness + shared durable sandbox pool #49 — Two-tier architecture epic (harness=brain, sandbox=hands). Assumes 1 session per harness; this issue removes that constraint.
• Sync-path saturation: bounded wait + 503 Retry-After per spec 4.3 #55 — Sync-path saturation / 503 Retry-After. With multiplexing, overload handling shifts from pod-level to session-level.

[pdettori]

Correction: the severity table above is wrong on four of five rows

Traced against main at f78081f. The table in the body was written from reading the code
rather than tracing it, so please scope from this comment (or the spec) and not from the body's
table. Full detail with citations is in
PR #228 §2 of
docs/specs/2026-09-06-p5-session-isolation-design.md.

#	Body says	Code says
1. ANTHROPIC_API_KEY	Showstopper — "concurrent sessions with different auth tokens corrupt each other"	Real, but different in kind. run-turn.ts:310-312 is write-once-if-absent, so session A's token sticks process-wide and B..N silently authenticate as A. A cross-tenant identity leak, not mutual corruption.
2. stdoutTakeoverState	Showstopper	Unreachable in server mode. takeOverStdout() is called only from main.ts:511 (CLI) and rpc-mode.ts:54; the harness enters Pi via createAgentSession.
3. sessionResourceCleanups	Showstopper — "cleanup for session A tears down session B's resources"	Largely false. The sole registrant repo-wide already filters by session: closeOpenAICodexWebSocketSessions(sessionId) closes only that session's socket (openai-codex-responses.ts:769-784), and agent-session.ts:733 passes this.sessionId.
4. fileMutationQueues	Race — "concurrent sessions serialize through same promise chain"	Wrong mechanism. The map is keyed by realpath (file-mutation-queue.ts:16-26), so different files already run in parallel. The only true global is registrationQueue (:5), which serializes one realpath() call — throughput, not correctness.
5. commandResultCache	Stale data	Low. Only caches values beginning with ! (resolve-config-value.ts:211-219). The harness supplies credentials as config/env, never as !cmd.

Two consequences that change the shape of the work:

"4 of 5 are in pi-fork" is no longer the right frame — none of them needs a pi-fork change.
Pi is already per-session on the credential path: AgentSession._getRequiredRequestAuth
(agent-session.ts:357-381) resolves auth per request from a per-instance ModelRegistry and
passes an explicit apiKey into stream options, and withEnvApiKey (stream.ts:26) consults the
environment only when no explicit key was given. The harness's env seed is what activates that
fallback beneath a mechanism that already worked. The other four globals are inert in server mode,
so the design pins them with reachability tests instead of refactoring them — refactoring would add
fork divergence to fix nothing.

The actual prerequisite is missing from the issue: no per-session credential enters the harness
at all. Every buildConfig() call site reads process.env (server.ts:66-73,
leaf-job.ts:13-18, cli.ts:13), so the pod's environment is the credential, one per
deployment. TurnConfig.anthropicAuthToken exists but no caller ever fills it from a request.
This forces an ordering the "Proposed fix" list does not have: inflow → remove the ambient
fallback → scrub. Deleting the seed first would not fix a leak; it would leave every session
unauthenticated.

Also note the proposed fix #1 ("set credentials once at startup") is only correct for a
single-tenant pod. Under mixed tenancy it is the bug, and ADR-0011/0012/0026 forbid the harness
holding a provider credential at all — so the harness carries an inert placeholder plus a subject,
and AuthBridge's static-inject swaps it on egress.

Suggested split

This issue currently bundles three separable things, and its title promises density that the
correctness work alone does not deliver:

1. Session isolation (the concurrency-safety half) — specced as P5 in PR docs(spec): P5 multi-session isolation — per-request subject, no ambient credential (#220) #228, implementation
   plan written, postponed by decision as of 2026-09-06.
2. The deployment model — KEDA ScaledJob → elastic pod pool, the pod-count/activation numbers,
   and the Sync-path saturation: bounded wait + 503 Retry-After per spec 4.3 #55 overload shift from pod-level to session-level. Deserves its own issue; unblocked by 1.
3. Per-subject resolution at the injector — required for end-to-end multi-tenancy, and it is
   Z5's deferred per-user / RFC 8693 half, living in kagenti-extensions. Track against Z5. The
   harness half is the strict prerequisite: the injector cannot key on a subject the harness never
   sends.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com