feat(cloud): relay terminal traffic to workers

[mohakchakraborty2004]

What

Implement the Docker-local version of the cloud terminal relay architecture.

Terminal traffic now uses a live relay path between the Electron client and the worker’s outbound terminal stream, while preserving an asynchronous durable mirror in the control plane/Postgres. The change also distinguishes an exited coding-agent terminal from a worker that is still provisioning.

Why

The existing cloud terminal path made the control plane’s durable transport the synchronous data path for every keystroke and output frame. This added avoidable latency and made terminal responsiveness dependent on the queued transport path.

It also treated an exited agent terminal the same as a cold worker. The UI therefore retried indefinitely on an old/exited session and remained stuck on “Connecting…”.

How

• Add a feature-flagged terminal relay fast path:
  • Electron → control-plane terminal relay → worker outbound stream → PTY → coding agent.
  • Worker output is delivered live to the connected browser, then mirrored asynchronously into Postgres for replay and audit.
  • Sequence numbers preserve byte ordering between live relay frames and durable replay.
  • Existing durable/Postgres transport remains as the fallback path.
• Add relay diagnostics for forwarded input, delivered output, and asynchronous mirroring.
• Add a stable terminal routing key to ticket responses for future affinity-aware deployment.
• Keep the relay embedded in the control-plane process for now. This validates the architecture locally without introducing a separate relay service or changing existing production routing.
• Return 410 TERMINAL_SESSION_EXITED for an agent terminal that has already finished, rather than returning retryable worker-unavailable state.
• Stop the frontend retry loop for exited terminals and show a clear terminal error instead.
• Retain retry behavior for genuine cold-start/provisioning cases (409 WORKER_UNAVAILABLE).
• Add a short agent.ready grace period so sessions that already exited cannot wait forever for an event that will never arrive.

Testing

Validated locally:

cd cloud
go test ./...

cd frontend
npm run typecheck
npx vitest run src/renderer/lib/cloud-terminal-mux.test.ts src/renderer/components/TerminalPane.test.tsx

Also verified manually with the Docker-local cloud stack:

AO_CLOUD_TERMINAL_STREAM=1 AO_CLOUD_TERMINAL_RELAY=1 npm run cloud:local

Then launch Electron against the local control plane:

cd frontend
AO_CLOUD_OFFERING=on \
AO_CLOUD_CONTROL_PLANE_URL=http://127.0.0.1:8081 \
npm run dev

Observed behavior:

• A live AI Skill Codex session successfully completed ticket issuance, terminal attach, ready state, input forwarding, live output delivery, and asynchronous durable mirroring.
• An exited Muza terminal returns 410 and stops retrying instead of staying on “Connecting…”.
• The Docker-local architecture is enabled only when both terminal stream and relay flags are set.

Checklist

• Branched from main (or continuing an existing PR branch)
• One focused change; links the related issue when applicable
• Follows [AGENTS.md](https://github.com/AgentWrapper/agent-orchestrator/blob/main/AGENTS.md) conventions and PR hygiene
• Tests added/updated for user-visible behavior where it makes sense
• Relevant CI checks pass for the area touched (go, frontend, etc.)

[i-trytoohard]

@mohakchakraborty2004 friendly nudge: if there's an issue this addresses, please link it (Fixes #N). If none exists, no action needed.

[Pritom14]

Relay review: migrations, telemetry, and one production blocker

Migrations: none. AppendTerminalOutputAt reuses the existing ao_terminal_output schema, so no schema change and no collision with the in-flight 00037 migrations.

Telemetry: too loud on the hot path. Three INFO lines per output frame (forwarded / delivered / mirrored) plus one per input chunk, gated only by logger != nil. A streaming session writes thousands of frames, so this floods CloudWatch and adds work on the data path. Suggested:

• Move the three per-frame INFO lines to DEBUG, or sample them.
• Keep terminal relay durable mirror failed (Error).
• Keep saturated_clients, but as a counter, not a per-frame INFO line.

Single-replica is a production blocker, not a testing one. The live fan-out is in-process (clients map[string]map[chan terminalRelayOutput]struct{}, delivery tagged mode=local_same_replica). It only bridges a browser and its worker when both WS connections land on the same CP task. The PR is safe to run and test today, but only at desired-count=1, which is how staging is pinned for the relay test right now. At two or more replicas the ALB can split worker and browser across tasks, and the browser silently falls back to the durable Postgres mirror (correct, but poll latency, not the live path). So the blocker is scaling the CP past one task, not merging or testing.

Mitigation (pick one):

1. Cross-replica bus (preferred, no new infra). Add Postgres LISTEN/NOTIFY on a per-terminal channel. Every frame is already written to the mirror table, so NOTIFY only has to wake the replica holding the browser to read frames after the last sequence. Reuses what is here and keeps the same-replica path as the fast path.
2. Replica affinity. Route a session's worker-WS and browser-WS to the same task via a layer that consistent-hashes on session or terminal id. ALB cookie stickiness alone does not solve this, since the worker and browser are separate clients and would not share a cookie.
3. Split the terminal data plane into its own service kept at count=1 (or sticky) while the API scales. Smallest change, least elegant.

Test the mitigation before pushing it:

1. Reproduce first. Scale staging CP to desired-count=2. Force worker and browser onto different tasks (temporary log of which task served each WS, or pin each connection). Confirm the split falls back to mirror latency.
2. Behind a flag (for example AO_CLOUD_TERMINAL_RELAY_CROSS_REPLICA), deploy the LISTEN/NOTIFY path to staging at count=2. Repeat the split. Assert live frames arrive with single-digit ms added latency, sequence stays gap-free, no dupes, ordering intact.
3. Load and failure test. Stream a heavy output burst across the split, watch DB CPU and NOTIFY volume. Kill the browser's task mid-stream, confirm reconnect resumes from the last sequence via the existing durable replay.
4. Only after that passes on staging at count=2, raise prod desired-count above 1.

Net: fine to keep deployed and merged behind its flag at count=1. Hold multi-replica until the cross-replica path lands and passes the test above.