fix(cloud): coder and terminal hardening deployed on staging (follow-up to #5023) by Pritom14 · Pull Request #5329 · Untrivial-ai/agent-orchestrator

Pritom14 · 2026-09-13T08:09:15Z

What

The cloud fixes that are deployed and tested on the staging control plane (task-def :102) but were not part of #5023. This lands exactly the deployed CP state on top of #5023.

Commits (all cloud/ or cloud-gated frontend)

perf(cloud): launch preinstalled Coder worker, plus make the Coder harness link idempotent (was PR perf(cloud): launch preinstalled Coder worker #4795, cherry-picked here).
fix(cloud): stage the non-empty-workspace checkout inside the workspace (coder checkout-permission fix).
fix(cloud): three coder UX fixes (terminal reconnect storm, empty-repo race, report fan-in) plus migration 00037 (allow queued fan-in turns).
fix(cloud): state-driven cloud terminal attach (remove SSE-wait and timeout hacks), plus two earlier terminal iterations that net into it.
fix(cloud-worker): boot the agent PTY at 80x24 instead of 120x40 so the intro banner never garbles.
fix(cloud): close a superseded epoch's terminals at retire so stale terminals reconnect state-driven, with no liveness poll.
fix(cloud): orchestrator inspects before spawn; workers report tersely (see Orchestration fixes below).

Orchestration fixes (from a live 10-worker run on project l-a, lottie-android)

Two agent-guidance bugs surfaced by the run, fixed in the orchestrator/worker system prompts and the using-ao skill (both compile into ao-worker):

Inspect before spawn. The orchestrator invented file names when decomposing a task, so two workers were handed paths that do not exist in the repo (AnimationValueCallback.java, PathInterpolator.java) and blocked with no PR. The orchestrator prompt now requires confirming a file, class, or path exists in the checkout before naming it in a worker prompt.
Terse worker reports. Worker ao report messages were multi-paragraph, so a long one landed in the orchestrator conversation looking like an injected prompt. Reports are now constrained to one or two sentences (outcome plus PR number, or the single blocking reason) with no diffs, logs, or file contents.

command_test asserts both rules. Neither touches local-app behavior.

Testing

Deployed on staging as ao-cloud-staging-api:102 and exercised end to end (coder + nodeops). cloud module gofmt, build, vet, and test are green; no cloud-client codegen drift. The orchestration fixes ride the ao-worker binary, so a CP redeploy propagates them to workers via self-update.

Notes

Adds migration 00037.
Supersedes perf(cloud): launch preinstalled Coder worker #4795, which should be closed (its commits are included here).
All changes are cloud path or cloud-gated; no local-app behavior change.

Follow-up to #5023.

Resume reliability fixes (added)

Four cloud resume/terminal reliability fixes, cherry-picked as clean commits:

Restore terminal 410 (TERMINAL_SESSION_EXITED) handling in the state-driven mux. The state-driven redesign dropped it, so an exited-agent 410 looped the ticket mint (stuck "Connecting"). This restores it.
Auto-rebake the coder workspace image on deploy, so the baked ao-worker SHA always matches the control plane and resumes skip the slow PTY upload.
Coder HTTP self-heal on a preinstalled-worker miss: fetch the correct worker over HTTP before falling back to the slow PTY upload.
Keep a resuming coder box alive through an idle-stop race, so a slow restore is not idle-killed mid bring-up.

Note: the backend false-positive 410 fix is a separate PR (#5477) because it corrects #5350 code that lives in main, not in this branch.

Link the baked harness into /etc/skel with `ln -sfn` so the layer is safe to re-run over its own output, matching the convention already used in cloud/nodeops/Sandbox.Dockerfile. Assert the link resolves to an executable in the same layer so a dangling link fails the build instead of shipping. This is idempotency hardening, not a build-blocker fix. A build failure reported as `ln: Already exists` was traced to a contaminated local `codercom/enterprise-base:ubuntu` tag on one preview host (no RepoDigests, same image id as a prior ao-coder-workspace:local), so that build re-ran this layer over its own earlier output. Against the clean upstream base the previous `ln -s` builds fine: that digest ships no /etc/skel/.local/bin at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

When the coding agent started and wrote files (e.g. .claude) into the workspace before repository checkout finished, checkout took the "clone into a staging dir and merge" path and created that staging dir in filepath.Dir(workspace) — the provider's durable root. On Coder that root is /home/coder, owned by the coder user (uid 1000), while the AO worker runs as ao-worker (uid 1001), so the staging mkdir failed with "permission denied" and the repo never checked out. The faster launch path (preinstalled worker, #4795) plus a roomier host made the agent win the startup race more often, so this latent bug went from rare to reliably hitting concurrent fan-outs (observed 3 of 6 workers failing checkout). Stage inside the workspace itself instead: it is always writable by the worker (the agent just wrote into it, which is why this path runs) and is on the same filesystem, so the entry moves stay same-filesystem renames. The hidden staging dir is removed before checkout returns. Provider-neutral. New test asserts staging happens under the workspace (uid-independent, so it holds even when tests run as root) and that the clone is merged while the agent's file is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…po race, report fan-in Root-caused via three parallel investigations of the coder session UX. 1. Terminal reconnect storm (the garbled/"Connecting…"-while-"Connected" terminal): keepTerminalAlive closed the socket on the FIRST ping timeout, but the keepalive ping shares the websocket write mutex with the output writer, so a large replay Write stalls the ping past its 5s timeout on a healthy socket. The close triggered a reconnect that replayed from the start, stalling the next ping — a self-sustaining storm (~every 25s) whose repeated from-zero replays garble the terminal. Tolerate a few consecutive ping misses (terminalPingMaxFailures) before closing; a genuinely dead socket still closes within a bounded window. 2. Empty-repository race: the worker launched the checkout and the coding agent as two unsynchronized goroutines. The agent's first task is baked into its launch argv, so it began acting the instant its process existed, racing the git clone; a fast worker start (baked worker) won that race and the agent inspected an empty workspace and gave up. Gate only the agent PTY spawn on a workspaceReady channel closed after checkout; all agent prep still overlaps the clone, and the workspace shell (a separate terminal) is untouched, so the snappy-shell UX is preserved. On checkout failure the run context is cancelled so the agent goroutine unblocks instead of parking. 3. `ao report` 409 under fan-in: the partial unique index ao_turns_one_active_per_session forbade a second QUEUED turn, so several workers reporting to one orchestrator at once collided (23505 -> 409) even though the work succeeded. Migration 00037 drops 'queued' from the index so reports FIFO-queue; the single-executing-turn guarantee is preserved and ClaimWorkerTurn already drains a multi-row queue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A cloud agent pane's mux opens no socket from mux.open(); the socket only opens after SSE agent.ready -> upgradeToAgent -> mint ticket -> open -> server ready, a chain that routinely exceeds the 3s OPEN_TIMEOUT_MS. The open-timeout therefore fired, tore down the mux (and its SSE), and reattached with countAsCloudFailure=false so the connect-failure breaker never bounded it — a self-sustaining ~4s reconnect storm. Each cycle rebuilt the SSE (replaying agent.ready from 0) and reset the replay cursor to 0, so the repeated from-0 replays garbled the pane and kept busy sessions stuck on "Connecting…"; the client closing its own socket (1000) produced no server close log. Do not arm the client open-timeout for a cloud pane. Local panes keep the 3s probe/spawn budget. Genuine cloud failures stay bounded by the existing machinery (mint 409 -> "waiting", socket close -> connect-failure breaker, CP's own ready deadline). Also guard upgradeToAgent so a re-firing agent.ready cannot re-open a socket on the same worker epoch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The previous fix removed the client open-timeout for cloud panes entirely to stop the ~4s reconnect storm. That stopped the storm but left a worse failure: if agent.ready never arrives (a stalled SSE subscription), the socket is never even attempted, so the CP-close and mint-409 recovery bounds cannot apply, and the pane hangs on "Connecting…" forever. Give a cloud pane a generous 30s open budget instead of none: - long enough that a normal slow open (agent.ready SSE -> mint -> dial -> CP ready ack, ~5-20s) completes without a storm — the timer is cleared on open; - short enough that a genuinely stalled attach recovers by rebuilding the mux and re-arming the agent.ready subscription, once, not as a storm. Local panes keep the 3s daemon-spawn budget. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…timeout hacks Replace the race/timeout-based cloud terminal attach with a server-driven one. The agent pane no longer waits on an agent.ready SSE (which stalled forever for large-event-log sessions like a busy orchestrator) and then opens the socket; it opens the agent terminal directly and drives its state off the control plane's existing starting/ready messages. All the client-side open-timeout hacks for cloud panes (the 3s that caused a ~4s reconnect storm, then the 30s band-aid) are removed; local daemon panes keep their 3s spawn budget. Readiness/recovery is bounded by the CP (its 20s ready deadline closes a never-ready socket) and the mint-409 "waiting" poll — no client clock. - frontend cloud-terminal-mux: always connect(kind) at construction; drop waitForAgentReady/subscribeAgentReady/upgradeToAgent. Guard resize so a hidden 0x0 attach sends NO resize (a 0x0 resize was rejected by the CP, which closed the socket — the real cause of the "distorted / wrong-width" agent TUI, since a parked pane looped on that and left the PTY at a stale width). A visible pane sends its real fitted size, so the TUI redraws. - CP: serialize agent-terminal find-or-create (OpenTerminal + the worker's EnsureWorkerAgentTerminal) with a transaction-scoped advisory lock so a browser open racing the worker before the agent starts converges on one terminal row instead of spawning a duplicate agent. Reuse of a running agent's terminal is unchanged (non-disruptive). Send reset for agent terminals too (from-0 replay is correct across per-epoch sequences). Fixes the orchestrator "Connecting…" stall and the distorted agent TUI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…banner never garbles The agent terminal is spawned at worker boot (StartAgent) with no client dimensions, so openTerminal fell back to 120x40. The coding agent draws its full-screen intro (welcome box, "What's new") once and commits that fixed-layout box art to scrollback, which never reflows. A viewer whose pane is narrower than 120 columns then sees every banner line overflow and wrap, garbling the intro permanently even though the live region repaints correctly on the client's resize. Fall back to the canonical 80x24 instead. Every realistic viewer pane is at least 80 columns wide, so the committed banner renders cleanly (under-filling at worst, never overflowing); the client's authoritative resize still expands the live UI to the full pane width. A client-provided size always wins over the fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…rminals reconnect state-driven When a replacement worker supersedes an epoch (resume/repair), the retire transaction updated ao_worker_connections and failed the old requests but never touched the old epoch's ao_terminal_sessions rows. Those rows stayed state=open for the full session TTL (~30m) even though nothing was behind them, so the terminal-output stream kept serving a silently-dead terminal. The only things that ever noticed were a queued keystroke or a slow ping/pong keepalive timeout. Close the retired epoch's terminal rows in the same retire transaction. The existing writeTerminalOutput loop already re-reads terminal state every tick and returns on 'closed', so the stream closes within a tick and the client re-attaches against the live epoch. No new liveness poll: detection stays driven off the authoritative state the output loop already reads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two fixes from a live 10-worker run on project l-a (lottie-android): 1. The orchestrator invented file names when decomposing a task, so two workers were handed paths that do not exist in the repo (e.g. AnimationValueCallback.java, PathInterpolator.java) and blocked without a PR. Add an inspect-before-spawn rule: confirm a file/class/path exists in the checkout before naming it in a worker prompt. 2. Worker ao report messages were multi-paragraph, so a long one landed in the orchestrator conversation looking like an injected prompt. Constrain reports to one or two sentences (outcome + PR number, or the single blocking reason); never paste diffs, logs, or file contents. Both rules land in the orchestrator/worker system prompts and the using-ao skill; command_test asserts each. Prompts compile into ao-worker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…n the state-driven mux

The coder workspace image bakes ao-worker/ao from the control-plane image so Coder's bootstrap fast path can skip the multi-megabyte PTY upload. When the CP ships a new ao-worker, the baked SHA stops matching AO_WORKER_EXPECTED_SHA256 and every spawn AND resume falls back to the slow, flaky upload until the image is rebaked by hand. Add scripts/publish-coder-workspace.sh: builds coder/Sandbox.Dockerfile against the exact control-plane image (so the baked /ao-worker and /ao bytes are identical to what the reconciler advertises), verifies the baked SHA-256s match the CP image, publishes the image (ECR by default; local-tag override for a single-host Coder daemon), and pushes a new Coder template version pinned to the immutable repo@digest. Wire it into deploy-staging.sh: when AO_CLOUD_SANDBOX_PROVIDER=coder, the rebake runs from the just-built control-plane digest before the rollout. The nodeops path is untouched. promote-production.sh stays docker-free by design, so the production rebake is documented as an explicit promote step in coder/README.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…PTY upload When a coder sandbox launches/resumes and its baked ao-worker (or ao helper) does not match the SHA the control plane advertises, the bootstrap fell straight back to the slow, flaky PTY binary upload (the __AO_PREINSTALLED_MISS__ path). On a miss, the launch-only bootstrap shell now first tries a fast HTTP pull of the exact build from the control plane at AO_CLOUD_PUBLIC_URL/api/cloud/v1/worker/binary/<expected_sha> (content-addressed and unauthenticated, like /worker/bootstrap), verifies the sha256, and installs it in place. Only if that self-heal fails does the workspace emit the miss marker and fall back to the existing PTY upload. The origin is embedded as a shell literal because worker.env is not sourced yet at the preinstalled-check stage; no new sandbox env var is required (both AO_CLOUD_PUBLIC_URL and the expected hash are already available). curl is used with a wget fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A user resume stamps startup_started_at and a short interaction lease (interactive_until), but a slow coder restore can outlast that lease. When coder then auto-stops the still-starting box for idleness, the reconciler accepted the external idle stop and re-paused the sandbox mid-resume, so the user saw "Connected" with no terminal that flipped back to "resuming" and stalled. Guard the accept-idle-stop branch in reconcileSandbox with a new startingUp(record) check: while startup_started_at is set and the startup deadline has not elapsed, refuse the provider idle stop and fall through to restore, keeping the bring-up alive. The guard is bounded by StartupTimeout, so a bring-up that never converges ages out and is paused/failed normally rather than holding compute awake forever. This complements the existing interactive_until bump on terminal attach/input (IssueTerminalTicket, queueTerminalRequest, RefreshTerminalInteraction) and AcceptSandboxProviderPause's atomic interactive_until re-check, which already fence the "user attaches while the reconciler is mid-accept" race. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Resolve the one conflict in cloud-terminal-mux.ts (+ its test) toward #5329's state-driven attach. main (#5350 relay) still carries the agent-ready SSE wait / upgradeToAgent path that #5329 deliberately removed; both sides already have the 410 TERMINAL_SESSION_EXITED handling so that converges. The renderer callers use none of the dropped options (only the test did), so taking #5329's mux + test keeps the tree consistent. #5350's relay is CP/worker-side and merged cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…lure path Command.Cleanup is only declared, never assigned, so it is always nil; every other call site guards with if command.Cleanup != nil, but main.go:339 and :349 called it unguarded. On the checkout-failure path this PR adds (cancel() -> ctx.Done()), the select fired the nil func -> worker SIGSEGV crash-loop (the recorded main.go:349 RCA) on exactly the failure this change means to handle gracefully. Guard both call sites to match the established pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mohakchakraborty2004 · 2026-09-16T23:10:35Z

Blocking issue found and fixed

startInteractiveAgent creates an agent terminal, then waits for checkout before calling StartAgent (main.go:337). The new direct cloud mux can attach immediately, and the terminal row is already open, so the UI reports ready before an agent PTY exists.

During that checkout window:

resize requests are claimed and fail because no terminal process exists;
user input is also accepted/acknowledged, then fails and is dropped because AgentTerminalID is not assigned until StartAgent.

There is no replay of those failed resize/input requests after checkout completes. This makes early terminal interaction unreliable and can leave the agent at the fallback 80×24 size.

…al-init error path Command.Cleanup is only declared, never assigned, so it is always nil. The ensureAgentTerminal error path called it unguarded, so an agent-terminal init failure would nil-panic the worker into a crash-loop. Guard it (matching the pattern used at the other call sites and in #5329). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…t, first-click delete, and worker self-heal (#5518) (#5519) * feat(cloud): session transcript capture + restore endpoints Control-plane side of cloud session delete + restore. - Migration 00037: ao_session_transcripts (org_id, session_id PK, FK to ao_sessions ON DELETE CASCADE) with tenant + service RLS policies, to durably hold the harness transcript blob and preserved git ref a deleted sandbox would otherwise lose. - Migration 00038: scope ao_projects repository-URL uniqueness to live (archived_at IS NULL) projects so a repo can be re-registered after its project is archived/deleted. - Store: PutSessionTranscript (UPSERT) / GetSessionTranscript (ErrNotFound when absent), worker-scoped via withOrg; RestoreSession un-terminates the same session_id and re-arms its sandbox for reconciliation in one tenant tx. - httpapi: TranscriptStore Put/Get abstraction (Postgres impl behind an adapter, swappable for S3); PUT/GET /worker/transcript (worker:connect, base64 body); POST /orgs/{orgId}/sessions/{sessionId}/restore -> 202. - Handler unit tests: transcript PUT/GET happy path, 404, scope + base64 guards, restore un-terminate + not-found + bad-UUID. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cloud): delete cloud projects + restore terminated cloud sessions Cloud-only DELETE/RESTORE UI, gated so local sessions/projects are untouched: - cloud-cp client: add restoreSession(orgId, sessionId) -> POST /orgs/{orgId}/sessions/{sessionId}/restore (202) with a CloudCpRestoreSessionResponse type mirroring the delete/resume shape. (deleteProject/deleteSession already existed and are reused.) - Project delete: _shell.tsx removeProject now routes CLOUD_PROJECT_KIND projects to cloudClient.deleteProject and invalidates the cloud projects/sessions queries, instead of hitting the local daemon endpoint (fixes the mis-routed cloud project delete). Local projects keep the existing path. - Session restore: useRestoreSession detects a cloud session from the cloud-sessions cache and calls restoreSession (mirroring the non-hook cloud pattern in useTerminateSession), then invalidates the cloud + workspace queries so the session shows as reviving. All existing restore entry points (archive card, terminal, notifications) now work for cloud sessions; local restore is unchanged. - Session delete copy: SessionTerminationPopover shows restore-aware copy for cloud sessions; the Sidebar remove-project dialog warns cloud deletes remove all sessions. - i18n: add termination.bodyCloud/bodyCloudNamed and shell.removeCloudProjectBody to all locale catalogs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cloud): in-sandbox capture/rehydrate for session delete + restore Add durable-restore checkpointing to the in-sandbox worker so a deleted cloud session can be rebuilt on a fresh sandbox with --resume working and uncommitted work returned. Capture (ao-worker): a throttled (~15s) checkpoint loop, started once the checkout and git credential helper are in place, locates the harness transcript (Claude first-class via CLAUDE_CONFIG_DIR/projects glob; Codex best-effort via CODEX_HOME rollout; others extensible), commits any uncommitted work to refs/ao/preserved/<session-id> through a temp index (no touch to the agent's index/worktree) and force-pushes it to origin via the worker's repo-local credential helper, then PUTs {agentSessionId, harness, transcript(base64), preservedGitRef} to /worker/transcript. Change-detection skips no-op pushes/PUTs; every step is best-effort. Rehydrate (ao-worker): on boot, after checkout and before the agent is built (gated via rehydrateDone), GET /worker/transcript; on 200 fetch and cherry-pick --no-commit the preserved ref onto the checkout and write the transcript to the harness resume path (Claude project dir encoded as [^a-zA-Z0-9]->'-'); 404 is the normal fresh-session no-op. Reconciler: verified the deleted->running restore already re-provisions a fresh sandbox (observed 'deleted', provider id NULL); added a minimal guard so a 'terminated' sandbox with a running desired-state also un-parks and re-provisions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cloud/worker): drive durable-restore capture off the Stop hook, not a timer Replace the 15s checkpoint ticker with an event trigger. The worker serves a unix socket (ao-checkpoint.sock, AO_CHECKPOINT_SOCKET); the harness Stop hook (ao hooks <harness> stop) pokes it on every turn completion, and the bridge runs one change-detected checkpoint per poke. A single-slot trigger coalesces bursts; the capture is fire-and-forget so a completed turn never waits on git or the network. No timers or polling. * fix(cloud): renumber session_restore migration 00037 -> 00039 00037 collided with 00037_turns_allow_queued_fanin (already deployed to staging on 09-11). goose keys on the numeric version, so the duplicate 37 silently skipped session_restore and ao_session_transcripts was never created, leaving capture and restore no-ops against a missing table. A fresh migrate against the combined image would instead hard-fail on the duplicate version. Renumber to the next free slot (00039); 00038 keeps the active-project unique index. Matches goose versions 38/39 now recorded on staging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): reconnect terminal after restore and make delete land on first click Two renderer bugs in the cloud delete/restore flow. Restore left the terminal stuck on the dead epoch (connected but TERMINAL ENDED, cannot type). A cloud terminal is keyed only on the unchanged session id and reuses a per-session mux factory whose replay cursor never resets, so after a restore re-provisions a fresh sandbox under the same id but a NEW worker epoch, the pane clung to the exited terminal and dialed the old replay position that the new epoch never sends. Fix: a per-session terminal-reset nonce (terminal-reset-store) that useRestoreSession bumps on a successful cloud restore, folded into the terminal cache key and the cloud mux factory key so a restore behaves like a fresh open: new pane, new factory closure, cursor at 0, re-mint against the new epoch. The nonce rides the cache key only (not the generation field) so the workspace reconcile loop still compares the raw terminalGeneration and does not dispose the freshly mounted terminal. Delete needed two or three clicks. The trash trigger only toggled the confirm popover (a second tap dismissed it), and useTerminateSession had no optimistic update: its one cache write targeted workspaceQueryKey, which does not drive the cloud board (recomputed from cloudSessionsQueryKey), so a cloud card sat still until the slow CP round trip and users re-clicked. Fix: optimistic onMutate marks the session terminated in BOTH the workspace and every cloud sessions cache so the card archives on the click, with onError rollback; and each trash trigger forces the confirm open instead of toggling it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): key the cloud terminal on the worker epoch so it reconnects on resume The recurring 'connected but TERMINAL ENDED / can't type' on cloud sessions is a terminal that never re-mints against a NEW worker epoch. When a paused session resumes (idle-pause wake) the CP starts a fresh worker at a new epoch with a new open agent terminal and the agent runs fine, but the renderer had NO epoch signal (terminalGeneration was never set) so it clung to the dead epoch's exited terminal. Nothing attached, so the CP idle-paused the session again ~3 min later -> reopen -> new epoch -> infinite loop. The earlier restore nonce covered only delete->restore, not resume. Surface the worker epoch (MAX agent-terminal worker_epoch) on the session DTO and fold it into terminalGeneration. The workspace reconcile loop already disposes and remounts a worker terminal when terminalGeneration changes, so ANY new epoch (resume, restore, re-provision, crash-recovery) now re-mints against the live agent. It is stable within an epoch, so it does not churn the pane between resumes. CP: domain.Session.WorkerEpoch + sessionSelect subquery + scanSession + the create INSERT RETURNING + sessionResponse.workerEpoch. Renderer: CloudCpSession .workerEpoch + toCloudWorkspaceSession sets terminalGeneration from it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): terminate the session on delete so it archives on the first click deleteSession only set the sandbox desired_state to 'deleted' and left the session row active (is_terminated stayed false), relying on the reconciler to terminate it later. The idle scanner races that by resetting desired_state back to 'paused' (verified: a deleted coder session's sandbox went straight back to paused), so the session was never terminated -> the board (which archives by is_terminated) kept showing the card -> the 5s poll re-added it -> the user had to click delete repeatedly. Restore, by contrast, flips is_terminated directly and lands in one click. Add TerminateSession (the exact inverse of RestoreSession): set is_terminated =true + activity_state='exited' AND request the sandbox teardown, atomically in one tenant transaction. Wire deleteSession to it. Now a delete archives the session immediately and the idle scanner cannot undo it. CP-only; no worker or migration change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): run harness hooks from the self-healed ao helper, not the baked path (#5518 pt1) The claude/codex/cursor activity hooks invoked a hardcoded /usr/local/bin/ao. The worker self-heals the current helper to <dataDir>/bin/ao and only shadows it on PATH, but the hooks used the absolute baked path, so on any deploy that changed the ao binary the Stop hook ran the STALE baked helper (no pokeCheckpoint) and durable-restore capture silently stopped until the template was rebaked. This is the recurring stale-bake capture breakage. Make a stale bake a non-event for hooks: - healHelper now ALWAYS keeps <dataDir>/bin/ao current (copies the baked helper when it already matches, downloads when stale) so the healed path is never empty or stale (new copyExecutable, atomic temp+rename). - Hooks resolve their binary via hookHelperPath(dataDir): the healed helper when present, else the baked path as a last resort. Threaded through the claude, codex, and cursor installers. - Hook removal matches on the ' hooks claude-code ' suffix instead of a fixed binary prefix, so a hook installed with any prior path is still cleaned up. Provider-agnostic; the worker binary changes, so a deploy carrying this must rebake both templates once (via deploy-staging.sh) — after which future worker changes self-heal without a rebake. Coder PTY-stream fallback reliability (#5518 pt2) is separate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud/worker): nil-guard agentCommand.Cleanup on the agent-terminal-init error path Command.Cleanup is only declared, never assigned, so it is always nil. The ensureAgentTerminal error path called it unguarded, so an agent-terminal init failure would nil-panic the worker into a crash-loop. Guard it (matching the pattern used at the other call sites and in #5329). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): show Connecting while a restored/resuming session's fresh box comes up Restoring a cloud session re-provisions a FRESH sandbox/container that rehydrates the saved context, which takes ~10s+. During that window the pane lingered on the previous box's dead terminal: typing went nowhere, then it flashed 'process exited', then finally 'Connecting'. The isRestoring flag cleared as soon as the restore API returned (near-instant), well before the new worker connected. Drive the connecting state off the session runtime instead: whenever a cloud session is active but its worker is not yet connected (and the terminal has not attached), show a calm 'Connecting…' overlay and suppress the dead terminal and the process-exited strip. It covers restore, resume-from-idle-pause, and fresh provision uniformly, and lifts the instant the new terminal attaches. Cloud only; local terminals unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): simplify the box-coming-up overlay to just 'Connecting…' Drop the 'saved context is being restored' subtitle (and its i18n key); the single Connecting label is what is wanted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): show Connecting immediately on restore, before the runtimeConnected poll The box-coming-up overlay only keyed on the polled runtimeConnected, which lags up to the 5s cloud poll. A board-initiated restore therefore left a window where the previous box's dead terminal still showed (unresponsive typing, then process exited) before the poll flipped and Connecting appeared. Add an immediate, synchronous reconnecting signal: the terminal-reset store's bump (already called by useRestoreSession) now marks the session reconnecting, the terminal clears it the instant it attaches, and isBoxComingUp ORs it with the runtimeConnected fallback. Connecting now shows the moment restore is clicked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cloud/worker): add a coarse periodic safety-net checkpoint (~25s) alongside the Stop hook Capture is event-driven off the Stop hook (turn completion), but a delete/restore in the MIDDLE of a turn (before any Stop fires) had nothing to restore from and re-ran from the task prompt. Add a coarse 25s periodic poke to the same coalescing trigger so in-progress work is captured. It is not the old 15s polling: the Stop hook is still primary, the checkpoint is change-detected (an unchanged tick is a no-op), and the trigger is a single coalescing entry point (poke) any producer can call. A future move to file-modifying tool-use events can lengthen or drop this timer without touching the bridge. Interval is a var so tests can shorten it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): don't re-cover a live terminal with Connecting on a post-attach flicker The box-coming-up overlay gated on state !== 'attached', so once a restored terminal attached, a transient reconnect (e.g. on the first keystroke, while the polled runtimeConnected still lagged false) flipped state off 'attached' and pulled the full Connecting cover back over the live terminal — the user typed, hit enter, and it went to Connecting. Gate on !hasAttached instead: once the pane has ever attached, the cover never returns; a same-epoch reconnect uses the subtle reattaching banner, and a real restore mounts a fresh pane (hasAttached resets) so the cover still shows until the new box attaches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): keep Connecting until the WORKER connects, not just the PTY attaches A restored session's terminal attaches to the sandbox PTY and replays the transcript as soon as the box is running, but the worker may still be coming up (or stuck). Gating the Connecting cover on PTY attach (!hasAttached) therefore lifted it too early, dropping the user onto a terminal that shows content but cannot accept input (the 'Restoring agent' state, runtimeConnected=false). Gate on the worker being connected instead: latch workerHasConnected when runtimeConnected first turns true, show Connecting until then, and never re-cover once the worker has connected (a transient reconnect uses the subtle banner). A restore mounts a fresh pane so the latch resets. Matches 'do not present the terminal until the box is up and the worker connected'. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud): make session restore idempotent so a rapid second restore can't cross the mapping RestoreSession un-terminated the session and re-armed the sandbox unconditionally, clearing the reconcile lease (reconcile_lease_owner='', reconcile_lease_until=NULL) with no guard. The lease is the reconciler's only exclusivity mechanism, so a second restore fired while the first restore's provision was still in flight fenced that provision and let a second one start concurrently for the same session — crossing the session<->sandbox<->workspace mapping (worker binds to the wrong box, live_workers=0, stuck 'Restoring agent'). Two changes: - Gate the un-terminate on is_terminated=true (with the row lock it takes). Once the first restore un-terminates the session, a rapid second restore matches 0 rows; return an idempotent no-op for an already-active session (distinguished from a missing one -> ErrNotFound) BEFORE touching the sandbox. - Stop clearing the reconcile lease. An expired/free lease is already claimable (ClaimSandboxes keys on reconcile_lease_until, not owner); a live lease must be left to its owner or to expire, never stomped. Verified against the staging schema (rolled-back tx): first restore un-terminates (1 row) and re-arms while PRESERVING a live lease; a second restore on the active session is a no-op (0 rows); a missing id is not-found. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud/terminal): gate restore 'Connecting' on a new worker epoch, not runtimeConnected A rapid delete->restore left the terminal showing the OLD box for ~10s: it displayed content but could not accept input, then flipped to Connecting, then finally became usable. Root cause (confirmed from CP logs + ao_terminal_sessions): on restore the fresh sandbox provisions and its worker only creates its agent terminal ~10-13s later under a NEW worker epoch. In that gap the only attachable terminal is the OLD epoch's — content replays but its box is being torn down, so typing does nothing. The previous cover keyed on runtimeConnected, which the OLD worker's lingering connection keeps true, so the cover lifted early onto the dead terminal. Fix: baseline the worker epoch at restore time (captured in useRestoreSession from the session DTO's terminalGeneration = MAX(agent terminal worker_epoch)) and show 'Connecting' until the current epoch advances PAST that baseline — i.e. the fresh worker's terminal actually exists. The epoch only moves forward, so the cover spans the whole boot gap and never flickers back once the new epoch attaches. - terminal-reset-store: bump(sessionId, currentEpoch) records baselineEpoch; markConnected clears reconnecting; drop the runtimeConnected-based markAttached. - useRestoreSession: capture the pre-restore epoch and pass it to bump. - TerminalPane: clear reconnecting when currentEpoch > baselineEpoch; isBoxComingUp = cloud && isReconnecting (epoch-gated), replacing the runtimeConnected latch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gofmt resource_handlers.go and mock useCloudOrg in shell tests - Run gofmt on cloud/internal/httpapi/resource_handlers.go to fix struct field alignment (CI cloud-build-test gate). - Add missing vi.mock for useCloudOrg in shell-new-session-shortcut.test.tsx so ShellLayout no longer calls the real useQuery without a QueryClientProvider, which was timing out all 43 tests. * fix(cloud): scope agent-terminal exit detection to the latest epoch (nodeops restore false-410) IssueTerminalTicket returned TERMINAL_SESSION_EXITED (410) for restored nodeops sessions, so the browser showed 'The coding-agent terminal has exited' with a green Connected badge instead of the Connecting cover, even though the restore succeeded and the agent was alive. Root cause: exit detection fired when ANY agent terminal was closed/failed. A restore provisions a fresh box under a NEW worker epoch and closes the OLD epoch's terminal, so an old 'closed' row is expected. The prior guard tried to suppress that by requiring an open terminal WITH a live ao_worker_connections row, but nodeops sessions do not populate ao_worker_connections, so the guard false-fired on every nodeops restore (coder populates it, so coder was unaffected). Fix: scope the closed/failed check to MAX(worker_epoch). Only the latest epoch's agent terminal state signals a real exit; a superseded old epoch's 'closed' row is ignored. Provider agnostic (no ao_worker_connections dependency) and applied to both IssueTerminalTicket exit-detection sites. Verified against staging: for the reported session the old logic returns exited true (the 410), the new logic returns false. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): remove unused workspace status binding * fix(cloud): don't false-fire terminal-exited 410 when the latest epoch has a live terminal The MAX(worker_epoch) exit-detection reported 'agent exited' whenever a closed/failed agent terminal existed at the latest epoch. But a worker whose first agent-terminal attempt fails and then reopens leaves a stale 'failed' row BESIDE a live 'open' row at the SAME epoch (seen on a fresh nodeops orchestrator spawn: epoch 5556 had both). The EXISTS matched the failed row and returned a false 410, so the browser showed 'The coding-agent terminal has exited' on a perfectly live terminal. Fix: only count a closed/failed terminal as an exit when its epoch has NO open/opening agent-terminal sibling. Verified against staging: for the affected session the old predicate returns exited=t, the new one returns exited=f. Applied to both IssueTerminalTicket exit-detection sites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cloud/terminal): retry a transient 410 instead of a permanent exited banner The mux treated any 410 TERMINAL_SESSION_EXITED from the ticket mint as a permanent agent exit: it showed 'The coding-agent terminal has exited' and stopped, never retrying. But on nodeops a worker's first agent-terminal attempt can fail then reopen at the same epoch within ~40s, and the control plane correctly reports that transient gap as a 410. So a perfectly live terminal got a permanent banner the browser never recovered from (a reload did not re-mint). Give the 410 the same bounded readiness poll the 409 WORKER_UNAVAILABLE already gets: retry as 'waiting' for up to ~60s (60 x the 1s cloud poll), reset on any successful mint, and only surface the permanent banner once the exit persists past that window (a genuine exit). No new timer: reuses the existing poll. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Revert "fix(cloud/terminal): retry a transient 410 instead of a permanent exited banner" This reverts commit c4844a7. * fix(cloud): gate browser agent ticket on a live agent terminal, not just a worker connection Root cause of the false 'coding-agent terminal has exited' banner on a fresh nodeops spawn: the worker connection registers on bootstrap, but the coding agent only starts after the repository checkout (~40s later), when the worker creates the agent terminal. IssueTerminalTicket issued a browser agent ticket on the worker connection alone, so the browser attached in that gap and find-or-created an agent terminal that no agent served; it timed out to 'failed', which then poisoned the next mint as a 410 exit. Gate the browser agent ticket on an already-live (opening/open, unexpired) agent terminal at the current epoch. The worker creates that terminal independently via EnsureWorkerAgentTerminal at agent start, so there is no deadlock: until then the browser gets ErrWorkerUnavailable (409) and shows 'Connecting', exactly as during provisioning, instead of attaching to a doomed terminal. CP-only, no worker change. The workspace shell terminal stays available earlier (kind='agent' only). Verified against staging: with only a failed terminal at the epoch the gate is closed (409); once the worker's open terminal exists the gate opens; an expired open terminal keeps the gate closed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: merge test <test@ao.local> Co-authored-by: t <t@ao.local> Co-authored-by: mohak <c.mohak2004@gmail.com>

i-trytoohard and others added 9 commits September 13, 2026 13:38

perf(cloud): launch preinstalled Coder worker

42b7981

Pritom14 mentioned this pull request Sep 13, 2026

perf(cloud): launch preinstalled Coder worker #4795

Merged

illegalcall mentioned this pull request Sep 13, 2026

feat(chat): expose preventive read-only permissions #4205

Open

i-trytoohard added bug Something isn't working comp/daemon Go daemon, process lifecycle, and backend control plane. labels Sep 13, 2026

i-trytoohard added this to the Agents & orchestration milestone Sep 13, 2026

Pritom14 and others added 8 commits September 16, 2026 01:06

fix(cloud): restore terminal 410 (TERMINAL_SESSION_EXITED) handling i…

266e10f

…n the state-driven mux

fix(cloud): buffer terminal requests before agent startup

12b6704

Pritom14 mentioned this pull request Sep 17, 2026

cloud: robust worker self-heal so a stale baked binary is a non-event (nodeops helper-path mismatch + coder PTY-stream reliability) #5518

Closed

mohakchakraborty2004 self-requested a review September 17, 2026 14:48

mohakchakraborty2004 approved these changes Sep 17, 2026

View reviewed changes

Pritom14 merged commit 177202f into main Sep 17, 2026
10 checks passed

Navigation Menu

fix(cloud): coder and terminal hardening deployed on staging (follow-up to #5023) - #5329

fix(cloud): coder and terminal hardening deployed on staging (follow-up to #5023)#5329
Pritom14 merged 17 commits into
mainfrom
fix/cloud-coder-terminal-hardening

Pritom14 commented Sep 13, 2026 •

edited

Loading

Uh oh!

mohakchakraborty2004 commented Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Pritom14 commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Commits (all cloud/ or cloud-gated frontend)

Orchestration fixes (from a live 10-worker run on project l-a, lottie-android)

Testing

Notes

Resume reliability fixes (added)

Uh oh!

mohakchakraborty2004 commented Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pritom14 commented Sep 13, 2026 •

edited

Loading