From 530110dc3dd4d9102958578631eaa8d97e834477 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 09:50:17 +0000 Subject: [PATCH 1/8] task: add completiondock activity coalescing --- ...8-25-completiondock-activity-coalescing.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 tasks/backlog/2026-08-25-completiondock-activity-coalescing.md diff --git a/tasks/backlog/2026-08-25-completiondock-activity-coalescing.md b/tasks/backlog/2026-08-25-completiondock-activity-coalescing.md new file mode 100644 index 0000000000..efa0321c65 --- /dev/null +++ b/tasks/backlog/2026-08-25-completiondock-activity-coalescing.md @@ -0,0 +1,76 @@ +# Fix CompletionDock activity twitch from ACP harness reports + +## Problem + +After PR #1874 unified ACP tool-call lifecycle tracking, the VM agent began reporting activity on every ACP `session/update` notification with a tool-call edge. Busy Codex turns can emit dozens of edges. Because those reports are fired from independent goroutines and read the mirrored host status at send time, stale `prompting` reports can arrive after authoritative `idle` reports at turn boundaries. The control plane persists `activity` with last-write-wins semantics, and the web client then verifies stale working activity and keeps the CompletionDock in the wrong morph until stale healing. + +The visible failure is a twitchy CompletionDock center button: Stop/Interrupt and Sleep flip rapidly, and Stop can be absent or stale when the agent is actually working. + +## Research findings + +- `packages/vm-agent/internal/acp/session_host_client.go:SessionUpdate` calls `nudgeHarnessActivityReport()` for every normalized ACP tool-call edge. +- `packages/vm-agent/internal/acp/session_host_harness_work.go:nudgeHarnessActivityReport()` currently single-flights only a same-instant goroutine handoff, so high-frequency edges still produce repeated HTTP callbacks. +- `packages/vm-agent/internal/acp/session_host_reporting.go:reportActivity()` snapshots status metadata and launches an independent goroutine per report. There is no ordering guarantee across concurrent POSTs. +- `packages/vm-agent/internal/acp/session_host_prompt.go:markPromptStarted()` and `markPromptDone()` are authoritative turn-level transitions and should remain immediate. +- Activity reports also carry normalized `runtimeWork*` fields. The coalescer must reduce user-visible activity churn without suppressing real runtime-work lease state changes. +- `apps/api/src/routes/projects/agent-activity-callback.ts` cancels sleep on `prompting` and on `idle` reports with active/settling runtime work; changing the VM-agent report cadence must preserve this contract. +- `apps/web/src/components/project-message-view/index.tsx` passes `working={lc.agentActivity !== 'idle'}` into `CompletionDock`; a client-side working→idle stabilization can hide residual message-batch races without changing the dock animation. +- Existing tests cover prompt re-report stopping, terminal retry budgets, activity payload contract fixtures, harness-work normalization, and CompletionDock behavior. New tests should extend those patterns rather than source-grep assertions. + +## Implementation checklist + +- [ ] Add a debounced, coalescing harness activity reporter in the VM agent. +- [ ] Ensure harness-originated reporting reads current mirrored status only when the debounce fires. +- [ ] Ensure only one harness-originated activity POST is in flight at a time. +- [ ] Track successful activity report snapshots so redundant coalesced reports are skipped while runtime-work state changes still propagate. +- [ ] Keep `markPromptStarted()` / `markPromptDone()` immediate and ensure successful authoritative reports update the coalescer's last-success state. +- [ ] Keep the 60s harness work re-report loop as the reliability backstop. +- [ ] Add a configurable VM-agent debounce interval with a default in the requested 500ms–1s range. +- [ ] Add Go regression tests for burst coalescing, stale prompting suppression after prompt done, successful-report dedupe, retry/no-success behavior, and runtime-work payload preservation. +- [ ] Add a client-side stabilized CompletionDock working signal that delays only working→idle propagation. +- [ ] Ensure idle→working remains immediate so Stop/Interrupt appears immediately. +- [ ] Add web unit tests for working→idle stabilization and reversal swallowing. +- [ ] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. +- [ ] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. +- [ ] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. +- [ ] Add the process fix to repository agent guidance. + +## Acceptance criteria + +- ACP tool-call edge bursts no longer produce one activity HTTP POST per edge. +- A late harness report cannot overwrite an authoritative prompt-done idle transition with stale prompting. +- Prompt start and prompt done activity transitions remain immediate. +- Lost changed-value harness reports self-heal through the existing periodic re-report loop. +- Runtime-work `active` / `settling` reports still reach the control plane when they are semantically new. +- CompletionDock shows Interrupt immediately when activity becomes working. +- CompletionDock does not flip to idle/Sleep for transient idle signals that reverse within the stabilization window. +- Existing CompletionDock visuals/animation timing are unchanged. +- Regression tests prove the race and stabilization behavior. + +## Post-mortem + +### What broke + +The CompletionDock lifecycle control flipped between Interrupt and Sleep and sometimes showed the wrong control because VM-agent activity reports arrived unordered and too frequently. + +### Root cause + +PR #1874 introduced per-edge ACP tool-call lifecycle reporting. The report path launched independent goroutines that read host status at send time and posted last-write-wins `activity` values to the control plane. At prompt boundaries, those goroutines could race with authoritative prompt-start/prompt-done reports. + +### Timeline + +- PR #1874 (`85d69a89b`) added ACP tool-call lifecycle reporting. +- PR #1881 made the CompletionDock morph directly reflect `agentActivity`, making the existing activity churn visible as button twitch. +- The issue was diagnosed on 2026-08-25 and fixed in this task. + +### Why it was not caught + +Existing tests proved harness lifecycle normalization and deadlock avoidance, but they did not assert report cadence, single-flight HTTP ordering, or UI stabilization across rapid working↔idle reversals. + +### Class of bug + +High-frequency runtime lifecycle signals converted directly into unordered cross-service writes and user-visible state, without a coalescing/order boundary. + +### Process fix + +Update VM-agent guidance so high-frequency callback streams must be debounced/coalesced, single-flight, and regression-tested for ordering/cadence before they can mutate control-plane state. From bb01c5b67c3bd1fa5bc262e49c4073771dc9c9e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 09:51:10 +0000 Subject: [PATCH 2/8] task: activate completiondock activity coalescing --- .../2026-08-25-completiondock-activity-coalescing.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename tasks/{backlog => active}/2026-08-25-completiondock-activity-coalescing.md (100%) diff --git a/tasks/backlog/2026-08-25-completiondock-activity-coalescing.md b/tasks/active/2026-08-25-completiondock-activity-coalescing.md similarity index 100% rename from tasks/backlog/2026-08-25-completiondock-activity-coalescing.md rename to tasks/active/2026-08-25-completiondock-activity-coalescing.md From e051bc793205643897a5f575fe8d7857d37483f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 10:10:13 +0000 Subject: [PATCH 3/8] fix: coalesce ACP harness activity reports --- .claude/rules/06-vm-agent-patterns.md | 22 ++ .claude/skills/env-reference/SKILL.md | 1 + .../project-message-view/CompletionDock.tsx | 2 +- .../components/project-message-view/index.tsx | 4 +- .../useCompletionDockWorking.ts | 43 ++++ .../useSessionLifecycle.ts | 7 +- .../useSessionLifecycle.types.ts | 2 + .../useCompletionDockWorking.test.ts | 74 +++++++ packages/vm-agent/internal/acp/gateway.go | 3 + .../vm-agent/internal/acp/gateway_test.go | 15 ++ .../vm-agent/internal/acp/session_host.go | 21 +- .../internal/acp/session_host_harness_work.go | 87 ++++++-- .../acp/session_host_harness_work_test.go | 198 +++++++++++++++++- .../internal/acp/session_host_reporting.go | 150 ++++++++++--- packages/vm-agent/internal/config/config.go | 6 + .../vm-agent/internal/config/config_load.go | 1 + .../vm-agent/internal/config/config_test.go | 8 +- packages/vm-agent/internal/config/helpers.go | 1 + packages/vm-agent/internal/server/server.go | 1 + ...8-25-completiondock-activity-coalescing.md | 22 +- 20 files changed, 596 insertions(+), 72 deletions(-) create mode 100644 apps/web/src/components/project-message-view/useCompletionDockWorking.ts create mode 100644 apps/web/tests/unit/components/useCompletionDockWorking.test.ts diff --git a/.claude/rules/06-vm-agent-patterns.md b/.claude/rules/06-vm-agent-patterns.md index 4379b38a5a..3e7a82897e 100644 --- a/.claude/rules/06-vm-agent-patterns.md +++ b/.claude/rules/06-vm-agent-patterns.md @@ -32,6 +32,28 @@ When a remote system (VM) is responsible for triggering its own cleanup: 3. **Both paths must use the same deletion logic** — reuse `deleteServer()`, `deleteDNSRecord()`, `cleanupWorkspaceDNSRecords()` 4. **Guard against duplicate execution** — Use DB status transitions (`running` -> `stopping`) as a lock. +## High-Frequency Callback Signals + +When the VM agent converts high-frequency runtime or harness lifecycle signals +into HTTP callbacks to the control plane, do not post one callback per edge. + +Required pattern: + +1. **Coalesce before crossing the network** — debounce bursts and send the latest + resolved state after a short quiet window. +2. **Serialize callback sends per signal source** — one source must not have + multiple in-flight POSTs that can arrive out of order and overwrite each + other. +3. **Read authoritative local state at send time** — edge handlers may record + intent, but they must not capture a status value that can become stale before + the POST lands. +4. **Dedupe only after success** — failed POST attempts must not update the + "last successfully reported" state; periodic re-report loops remain the + reliability backstop. +5. **Test cadence and ordering** — regression tests must prove bursts collapse, + turn-boundary stale reports are suppressed, and semantically new payloads + such as runtime-work lease changes still reach the control plane. + ## Modifying Cloud-Init 1. Edit `packages/cloud-init/src/template.ts` diff --git a/.claude/skills/env-reference/SKILL.md b/.claude/skills/env-reference/SKILL.md index dd32ab2e47..3f1ced8bd4 100644 --- a/.claude/skills/env-reference/SKILL.md +++ b/.claude/skills/env-reference/SKILL.md @@ -461,6 +461,7 @@ Generated deployments validate and pass these values through cloud-init to newly - `ACP_PROMPT_RETRY_INITIAL_BACKOFF` — Initial backoff before retrying transient provider prompt errors (default: 15s) - `ACP_PROMPT_RETRY_MAX_BACKOFF` — Max exponential backoff for transient provider prompt retries (default: 2m) - `ACTIVITY_REREPORT_INTERVAL` — Re-send `prompting` activity while a prompt is active (default: 60s) +- `ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE` — Debounce window for coalescing high-frequency ACP harness/tool-call activity reports before POSTing to the control plane (default: 750ms) - `ACP_CHECKPOINT_PREEMPT_GRACE` — Graceful ACP cancel/close wait before harness force-stop (default: 30s) - `ACP_CHECKPOINT_PREEMPT_MAX_GRACE` — Maximum caller-selected checkpoint rollover grace (default: 2m) - `ACP_CHECKPOINT_ROLLOVER_TIMEOUT` — Full checkpoint restart and strict LoadSession deadline (default: 2m) diff --git a/apps/web/src/components/project-message-view/CompletionDock.tsx b/apps/web/src/components/project-message-view/CompletionDock.tsx index c8c90e6b15..98a052a458 100644 --- a/apps/web/src/components/project-message-view/CompletionDock.tsx +++ b/apps/web/src/components/project-message-view/CompletionDock.tsx @@ -148,7 +148,7 @@ function Ring({ active, size }: { active: boolean; size: number }) { export type CompletionDockCenterAction = 'interrupt' | 'sleep' | 'archive'; export interface CompletionDockProps { - /** True while the agent is producing output (agentActivity !== 'idle'). */ + /** True while the agent is producing output; caller may stabilize brief idle reversals. */ working: boolean; /** Center lifecycle action. Defaults to legacy interrupt/archive derivation for compatibility. */ centerAction?: CompletionDockCenterAction; diff --git a/apps/web/src/components/project-message-view/index.tsx b/apps/web/src/components/project-message-view/index.tsx index 3cb5bf0297..74c095b89c 100644 --- a/apps/web/src/components/project-message-view/index.tsx +++ b/apps/web/src/components/project-message-view/index.tsx @@ -682,9 +682,9 @@ export const ProjectMessageView: FC = ({ is only the primary action after the reversible sleep boundary. */} {isActive && canWriteSession && - (lc.agentActivity !== 'idle' || canSleepSession || canArchiveSession) && ( + (lc.completionDockWorking || canSleepSession || canArchiveSession) && ( | null>(null); + + useEffect(() => { + if (activityWorking) { + if (idleTimerRef.current) { + clearTimeout(idleTimerRef.current); + idleTimerRef.current = null; + } + if (!completionDockWorkingRef.current) { + completionDockWorkingRef.current = true; + setCompletionDockWorking(true); + } + return; + } + + if (!completionDockWorkingRef.current || idleTimerRef.current) return; + + idleTimerRef.current = setTimeout(() => { + idleTimerRef.current = null; + completionDockWorkingRef.current = false; + setCompletionDockWorking(false); + }, COMPLETION_DOCK_IDLE_STABILIZE_MS); + + return () => { + if (idleTimerRef.current) { + clearTimeout(idleTimerRef.current); + idleTimerRef.current = null; + } + }; + }, [activityWorking]); + + return activityWorking || completionDockWorking; +} diff --git a/apps/web/src/components/project-message-view/useSessionLifecycle.ts b/apps/web/src/components/project-message-view/useSessionLifecycle.ts index fd6f805aa2..207db524c0 100644 --- a/apps/web/src/components/project-message-view/useSessionLifecycle.ts +++ b/apps/web/src/components/project-message-view/useSessionLifecycle.ts @@ -48,6 +48,7 @@ import { VIRTUAL_START, } from './types'; import { useActivityVerifyTimer } from './useActivityVerifyTimer'; +import { useCompletionDockWorking } from './useCompletionDockWorking'; import { useConnectionRecovery } from './useConnectionRecovery'; import { useSessionFileUpload } from './useSessionFileUpload'; import type { UseSessionLifecycleResult } from './useSessionLifecycle.types'; @@ -140,6 +141,7 @@ export function useSessionLifecycle( const [followUp, setFollowUp] = useState(''); const [sendingFollowUp, setSendingFollowUp] = useState(false); const [agentActivity, setAgentActivity] = useState('idle'); + const completionDockWorking = useCompletionDockWorking(agentActivity); const sleepingWakePendingRef = useRef(false); const [currentPlan, setCurrentPlan] = useState(null); const [promptStartedAt, setPromptStartedAt] = useState(null); @@ -649,7 +651,7 @@ export function useSessionLifecycle( // Cancel the current in-flight prompt via REST API const cancellingRef = useRef(false); const handleCancelPrompt = useCallback(() => { - if (agentActivity === 'idle' || cancellingRef.current) return; + if (!completionDockWorking || cancellingRef.current) return; cancellingRef.current = true; cancelAgentPrompt(projectId, sessionId) .then(() => { @@ -661,7 +663,7 @@ export function useSessionLifecycle( .finally(() => { cancellingRef.current = false; }); - }, [agentActivity, projectId, sessionId]); + }, [completionDockWorking, projectId, sessionId]); // Load more (pagination) const loadMore = async () => { @@ -763,6 +765,7 @@ export function useSessionLifecycle( showConnectionBanner: recovery.showConnectionBanner, retryWs, agentActivity, + completionDockWorking, /** True while a wake is in flight (hydrated from D1 or pushed over the socket). */ isWaking: wake.isWaking, /** Current wake phase, or null before the replacement runner reports a step. */ diff --git a/apps/web/src/components/project-message-view/useSessionLifecycle.types.ts b/apps/web/src/components/project-message-view/useSessionLifecycle.types.ts index 7c140e0b08..ff7f67f4f9 100644 --- a/apps/web/src/components/project-message-view/useSessionLifecycle.types.ts +++ b/apps/web/src/components/project-message-view/useSessionLifecycle.types.ts @@ -42,6 +42,8 @@ export interface UseSessionLifecycleResult { showConnectionBanner: boolean; retryWs: () => void; agentActivity: AgentActivityState; + /** Debounced working signal used only by CompletionDock to swallow brief idle reversals. */ + completionDockWorking: boolean; /** True while a wake is in flight (hydrated from D1 or pushed over the socket). */ isWaking: boolean; /** Current wake phase, or null before the replacement runner reports a step. */ diff --git a/apps/web/tests/unit/components/useCompletionDockWorking.test.ts b/apps/web/tests/unit/components/useCompletionDockWorking.test.ts new file mode 100644 index 0000000000..f2e51e1f24 --- /dev/null +++ b/apps/web/tests/unit/components/useCompletionDockWorking.test.ts @@ -0,0 +1,74 @@ +import { act, renderHook } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { + COMPLETION_DOCK_IDLE_STABILIZE_MS, + useCompletionDockWorking, +} from '../../../src/components/project-message-view/useCompletionDockWorking'; +import type { AgentActivityState } from '../../../src/components/project-message-view/types'; + +describe('useCompletionDockWorking', () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + + afterEach(() => { + vi.clearAllTimers(); + vi.useRealTimers(); + }); + + it('promotes idle to working immediately', () => { + const { result, rerender } = renderHook( + ({ activity }: { activity: AgentActivityState }) => useCompletionDockWorking(activity), + { initialProps: { activity: 'idle' } } + ); + + expect(result.current).toBe(false); + + rerender({ activity: 'prompting' }); + + expect(result.current).toBe(true); + }); + + it('delays working to idle transitions', async () => { + const { result, rerender } = renderHook( + ({ activity }: { activity: AgentActivityState }) => useCompletionDockWorking(activity), + { initialProps: { activity: 'responding' } } + ); + + expect(result.current).toBe(true); + + rerender({ activity: 'idle' }); + + expect(result.current).toBe(true); + await act(async () => { + await vi.advanceTimersByTimeAsync(COMPLETION_DOCK_IDLE_STABILIZE_MS - 1); + }); + expect(result.current).toBe(true); + + await act(async () => { + await vi.advanceTimersByTimeAsync(1); + }); + expect(result.current).toBe(false); + }); + + it('swallows an idle transition that reverses within the stabilization window', async () => { + const { result, rerender } = renderHook( + ({ activity }: { activity: AgentActivityState }) => useCompletionDockWorking(activity), + { initialProps: { activity: 'prompting' } } + ); + + rerender({ activity: 'idle' }); + await act(async () => { + await vi.advanceTimersByTimeAsync(COMPLETION_DOCK_IDLE_STABILIZE_MS / 2); + }); + expect(result.current).toBe(true); + + rerender({ activity: 'responding' }); + await act(async () => { + await vi.advanceTimersByTimeAsync(COMPLETION_DOCK_IDLE_STABILIZE_MS); + }); + + expect(result.current).toBe(true); + }); +}); diff --git a/packages/vm-agent/internal/acp/gateway.go b/packages/vm-agent/internal/acp/gateway.go index 0f49f581e5..fb3467a5cf 100644 --- a/packages/vm-agent/internal/acp/gateway.go +++ b/packages/vm-agent/internal/acp/gateway.go @@ -186,6 +186,9 @@ type GatewayConfig struct { // ActivityRereportInterval refreshes prompt activity while a prompt is active. // Zero disables the periodic re-report loop. ActivityRereportInterval time.Duration + // HarnessActivityReportDebounce coalesces high-frequency harness lifecycle + // edges before making activity callbacks. Zero uses the package default. + HarnessActivityReportDebounce time.Duration // Bounds on a single Claude harness lifecycle notification. Zero falls back // to the package defaults so existing constructions stay safe. diff --git a/packages/vm-agent/internal/acp/gateway_test.go b/packages/vm-agent/internal/acp/gateway_test.go index 149e549ee2..8c8e73ba85 100644 --- a/packages/vm-agent/internal/acp/gateway_test.go +++ b/packages/vm-agent/internal/acp/gateway_test.go @@ -10,6 +10,7 @@ import ( "time" "github.com/pelletier/go-toml/v2" + "github.com/workspace/vm-agent/internal/config" ) // Tests for OAuth support @@ -1952,6 +1953,20 @@ func TestActivityReportTimeoutPreservesLegacyDefaultAndOverride(t *testing.T) { } } +func TestHarnessActivityReportDebouncePreservesDefaultAndOverride(t *testing.T) { + t.Parallel() + + h := &SessionHost{config: SessionHostConfig{GatewayConfig: GatewayConfig{}}} + if got := h.harnessActivityReportDebounce(); got != config.DefaultACPHarnessActivityReportDebounce { + t.Fatalf("harnessActivityReportDebounce() = %v, want default %v", got, config.DefaultACPHarnessActivityReportDebounce) + } + + h.config.HarnessActivityReportDebounce = 875 * time.Millisecond + if got := h.harnessActivityReportDebounce(); got != 875*time.Millisecond { + t.Fatalf("harnessActivityReportDebounce() = %v, want configured 875ms", got) + } +} + func TestWriteAgentStartupConfigCodexMissingMcpTokenFailsClosed(t *testing.T) { for _, tc := range []struct { name string diff --git a/packages/vm-agent/internal/acp/session_host.go b/packages/vm-agent/internal/acp/session_host.go index c9a3159ee5..7fd2884a3c 100644 --- a/packages/vm-agent/internal/acp/session_host.go +++ b/packages/vm-agent/internal/acp/session_host.go @@ -309,14 +309,19 @@ type SessionHost struct { harnessWork harnessWorkStatus harnessTaskIDs map[string]struct{} harnessActivityCancel context.CancelFunc - // harnessReportPending single-flights the activity report triggered by a - // harness lifecycle notification. The ACP notification goroutine must never - // call reportActivity inline: reportActivity takes mu.RLock for its - // agentType/restartCount/statusErr snapshot, which is exactly the block the - // lock-free mirrors above exist to avoid. Handing the report to a short-lived - // goroutine keeps the notification worker unblocked, and coalescing stops a - // chatty harness from spawning one retried HTTP POST per message. - harnessReportPending atomic.Bool + // harnessReportMu owns the debounced, single-flight activity reporter + // triggered by harness lifecycle notifications. The ACP notification + // goroutine must never call reportActivity inline: reportActivity takes + // mu.RLock for its agentType/restartCount/statusErr snapshot, which is + // exactly the block the lock-free mirrors above exist to avoid. + harnessReportMu sync.Mutex + harnessReportTimer *time.Timer + harnessReportSequence uint64 + harnessReportRunning bool + harnessReportPending bool + lastActivityReportMu sync.Mutex + lastActivityReport activityReportSnapshot + lastActivityReportSet bool // activePromptID identifies the in-flight prompt associated with promptCancel. // Protected by promptCancelMu. activePromptID uint64 diff --git a/packages/vm-agent/internal/acp/session_host_harness_work.go b/packages/vm-agent/internal/acp/session_host_harness_work.go index 4e358c074f..ddce9f0c04 100644 --- a/packages/vm-agent/internal/acp/session_host_harness_work.go +++ b/packages/vm-agent/internal/acp/session_host_harness_work.go @@ -7,6 +7,7 @@ import ( "time" acpsdk "github.com/coder/acp-go-sdk" + "github.com/workspace/vm-agent/internal/config" ) const ( @@ -158,6 +159,7 @@ func (h *SessionHost) matchesHarnessSession(outerSessionID, innerSessionID strin } func (h *SessionHost) resetHarnessWorkForAgent(agentType string) { + h.stopHarnessActivityReportCoalescer() h.harnessWorkMu.Lock() h.stopHarnessWorkRereportLocked() progressAt := h.nextHarnessWorkProgressAtLocked() @@ -183,6 +185,7 @@ func harnessWorkSourceForAgent(agentType string) string { } func (h *SessionHost) clearHarnessWork() { + h.stopHarnessActivityReportCoalescer() h.harnessWorkMu.Lock() h.stopHarnessWorkRereportLocked() h.harnessTaskIDs = nil @@ -476,8 +479,8 @@ func (h *SessionHost) stopHarnessWorkRereportLocked() { } } -// nudgeHarnessActivityReport queues one activity report for the harness -// lifecycle change that just landed. +// nudgeHarnessActivityReport records that a harness lifecycle change landed and +// schedules one debounced activity report. // // It MUST be used instead of calling reportActivity inline from // HandleExtensionMethod. reportActivity takes h.mu.RLock to snapshot @@ -491,20 +494,80 @@ func (h *SessionHost) stopHarnessWorkRereportLocked() { // timeout fires — stalling the handshake and every later notification // (including session/update, i.e. the live stream) behind it. // -// The atomic single-flight also coalesces bursts: applyClaudeHarnessLifecycle +// The debounced single-flight also coalesces bursts: applyClaudeHarnessLifecycle // returns true for any recognized message, including repeat task_progress that -// mutates nothing, and each report otherwise spawns its own retried HTTP POST. -// Clearing the flag before reporting is deliberate — a nudge that races the -// in-flight report queues a trailing one, so the control plane always converges -// on the latest state. +// mutates nothing, and ACP tool-call streams can emit dozens of edges in one +// turn. The timer reads the current status mirror only after the stream has been +// quiet for the debounce window, so a pre-turn-end edge cannot post stale +// prompting after markPromptDone's authoritative idle report. func (h *SessionHost) nudgeHarnessActivityReport() { - if !h.harnessReportPending.CompareAndSwap(false, true) { + select { + case <-h.ctx.Done(): return + default: } - go func() { - h.harnessReportPending.Store(false) - h.reportActivity(h.activityForHarnessWork()) - }() + + h.harnessReportMu.Lock() + defer h.harnessReportMu.Unlock() + h.harnessReportPending = true + if h.harnessReportRunning { + return + } + h.scheduleHarnessActivityReportLocked(h.harnessActivityReportDebounce()) +} + +func (h *SessionHost) harnessActivityReportDebounce() time.Duration { + if h.config.HarnessActivityReportDebounce > 0 { + return h.config.HarnessActivityReportDebounce + } + return config.DefaultACPHarnessActivityReportDebounce +} + +func (h *SessionHost) scheduleHarnessActivityReportLocked(delay time.Duration) { + h.harnessReportSequence++ + sequence := h.harnessReportSequence + if h.harnessReportTimer != nil { + h.harnessReportTimer.Stop() + } + h.harnessReportTimer = time.AfterFunc(delay, func() { + h.flushHarnessActivityReport(sequence) + }) +} + +func (h *SessionHost) flushHarnessActivityReport(sequence uint64) { + h.harnessReportMu.Lock() + if sequence != h.harnessReportSequence { + h.harnessReportMu.Unlock() + return + } + h.harnessReportTimer = nil + if !h.harnessReportPending { + h.harnessReportMu.Unlock() + return + } + h.harnessReportPending = false + h.harnessReportRunning = true + h.harnessReportMu.Unlock() + + h.reportCoalescedHarnessActivity() + + h.harnessReportMu.Lock() + h.harnessReportRunning = false + if h.harnessReportPending { + h.scheduleHarnessActivityReportLocked(h.harnessActivityReportDebounce()) + } + h.harnessReportMu.Unlock() +} + +func (h *SessionHost) stopHarnessActivityReportCoalescer() { + h.harnessReportMu.Lock() + h.harnessReportSequence++ + h.harnessReportPending = false + if h.harnessReportTimer != nil { + h.harnessReportTimer.Stop() + h.harnessReportTimer = nil + } + h.harnessReportMu.Unlock() } // activityForHarnessWork is reachable from the ACP notification goroutine, so it diff --git a/packages/vm-agent/internal/acp/session_host_harness_work_test.go b/packages/vm-agent/internal/acp/session_host_harness_work_test.go index 4d776e25f3..31c507fb35 100644 --- a/packages/vm-agent/internal/acp/session_host_harness_work_test.go +++ b/packages/vm-agent/internal/acp/session_host_harness_work_test.go @@ -448,19 +448,191 @@ func TestACPToolCallActivityReportsOnlyNormalizedState(t *testing.T) { } } +func TestHarnessActivityReportCoalescesACPToolCallBursts(t *testing.T) { + t.Parallel() + + reports := newActivityReportCapture(t) + debounce := 20 * time.Millisecond + + _, client := newHarnessWorkTestClient(t, SessionHostConfig{GatewayConfig: GatewayConfig{ + ProjectID: "project", + NodeID: "node", + SessionID: "sam-session", + ControlPlaneURL: reports.server.URL, + CallbackToken: "token", + HTTPClient: reports.server.Client(), + HarnessActivityReportDebounce: debounce, + TerminalActivityReportAttempts: 1, + }}, "openai-codex", "acp-session", HostPrompting) + + notifyACPToolCall(t, client, "acp-session", "tool-1", "Read file", acpsdk.ToolCallStatusPending, nil) + for range 12 { + notifyACPToolCallUpdate(t, client, "acp-session", "tool-1", nil) + } + + waitFor(t, 300*time.Millisecond, func() bool { + return reports.count() == 1 + }) + time.Sleep(3 * debounce) + if got := reports.count(); got != 1 { + t.Fatalf("tool-call burst was not coalesced: reports=%d", got) + } + + _, payloads := reports.snapshot() + payload := payloads[0] + if payload.Activity != "prompting" { + t.Fatalf("coalesced activity = %q, want prompting", payload.Activity) + } + if payload.RuntimeWorkState != string(harnessWorkActive) || + payload.RuntimeWorkCount == nil || + *payload.RuntimeWorkCount != 1 || + payload.RuntimeWorkSource != acpToolCallWorkSource { + t.Fatalf("coalesced runtime-work payload = %#v", payload) + } +} + +func TestHarnessActivityReportDebounceReadsStatusAfterPromptDone(t *testing.T) { + t.Parallel() + + reports := newActivityReportCapture(t) + debounce := 40 * time.Millisecond + + host, client := newHarnessWorkTestClient(t, SessionHostConfig{GatewayConfig: GatewayConfig{ + ProjectID: "project", + NodeID: "node", + SessionID: "sam-session", + ControlPlaneURL: reports.server.URL, + CallbackToken: "token", + HTTPClient: reports.server.Client(), + HarnessActivityReportDebounce: debounce, + }}, "openai-codex", "acp-session", HostPrompting) + + notifyACPToolCall(t, client, "acp-session", "tool-orphan", "Run command", acpsdk.ToolCallStatusInProgress, nil) + time.Sleep(debounce / 4) + if got := reports.count(); got != 0 { + t.Fatalf("harness report fired before debounce elapsed: reports=%d", got) + } + + endPromptTurn(t, host, "end_turn", nil) + waitFor(t, 300*time.Millisecond, func() bool { + return reports.count() >= 1 + }) + time.Sleep(3 * debounce) + + _, payloads := reports.snapshot() + idleSeen := false + for _, payload := range payloads { + if payload.Activity == "prompting" { + t.Fatalf("stale prompting report escaped after prompt done: payloads=%#v", payloads) + } + if payload.Activity == "idle" { + idleSeen = true + } + } + if !idleSeen { + t.Fatalf("prompt done did not report idle: payloads=%#v", payloads) + } +} + +func TestPromptReportsUpdateHarnessCoalescerSuccessfulSnapshot(t *testing.T) { + t.Parallel() + + reports := newActivityReportCapture(t) + debounce := 15 * time.Millisecond + + host := NewSessionHost(SessionHostConfig{GatewayConfig: GatewayConfig{ + ProjectID: "project", + NodeID: "node", + SessionID: "sam-session", + ControlPlaneURL: reports.server.URL, + CallbackToken: "token", + HTTPClient: reports.server.Client(), + HarnessActivityReportDebounce: debounce, + }}) + t.Cleanup(host.Stop) + + host.markPromptStarted(acpsdk.SessionId("acp-session"), 1, "viewer-1") + waitForActivitySnapshot(t, host, "prompting") + host.nudgeHarnessActivityReport() + time.Sleep(3 * debounce) + if got := reports.count(); got != 1 { + t.Fatalf("coalescer re-sent successful prompt-start snapshot: reports=%d", got) + } + + host.markPromptDone() + waitForActivitySnapshot(t, host, "idle") + host.nudgeHarnessActivityReport() + time.Sleep(3 * debounce) + if got := reports.count(); got != 2 { + t.Fatalf("coalescer re-sent successful prompt-done snapshot: reports=%d", got) + } +} + +func TestFailedHarnessActivityReportDoesNotUpdateCoalescerSnapshot(t *testing.T) { + t.Parallel() + + var mu sync.Mutex + fail := true + reports := 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + mu.Lock() + reports++ + shouldFail := fail + mu.Unlock() + if shouldFail { + w.WriteHeader(http.StatusInternalServerError) + return + } + w.WriteHeader(http.StatusNoContent) + })) + defer server.Close() + + debounce := 10 * time.Millisecond + host := NewSessionHost(SessionHostConfig{GatewayConfig: GatewayConfig{ + ProjectID: "project", + NodeID: "node", + SessionID: "sam-session", + ControlPlaneURL: server.URL, + CallbackToken: "token", + HTTPClient: server.Client(), + HarnessActivityReportDebounce: debounce, + TerminalActivityReportAttempts: 1, + }}) + t.Cleanup(host.Stop) + host.setStatus(HostReady, "") + + host.nudgeHarnessActivityReport() + waitFor(t, 300*time.Millisecond, func() bool { + mu.Lock() + defer mu.Unlock() + return reports == 1 + }) + + mu.Lock() + fail = false + mu.Unlock() + host.nudgeHarnessActivityReport() + waitFor(t, 300*time.Millisecond, func() bool { + mu.Lock() + defer mu.Unlock() + return reports == 2 + }) +} + func TestHarnessActivityReportsOnlyNormalizedStateAndRereportsActiveWork(t *testing.T) { t.Parallel() reports := newActivityReportCapture(t) host, client := newHarnessWorkTestClient(t, SessionHostConfig{GatewayConfig: GatewayConfig{ - ProjectID: "project", - NodeID: "node", - SessionID: "sam-session", - ControlPlaneURL: reports.server.URL, - CallbackToken: "token", - HTTPClient: reports.server.Client(), - ActivityRereportInterval: 10 * time.Millisecond, + ProjectID: "project", + NodeID: "node", + SessionID: "sam-session", + ControlPlaneURL: reports.server.URL, + CallbackToken: "token", + HTTPClient: reports.server.Client(), + HarnessActivityReportDebounce: 10 * time.Millisecond, + ActivityRereportInterval: 10 * time.Millisecond, }}, "claude-code", "sdk-session", HostReady) notifyClaudeLifecycle(t, client, `{ @@ -592,6 +764,18 @@ func notifyClaudeLifecycle(t *testing.T, client *sessionHostClient, payload stri } } +func waitForActivitySnapshot(t *testing.T, host *SessionHost, activity string) { + t.Helper() + + waitFor(t, 300*time.Millisecond, func() bool { + request, ok := host.prepareActivityReport(activity) + if !ok { + return false + } + return host.successfulActivityReportMatches(request.payload) + }) +} + func newHarnessWorkTestClient(t *testing.T, config SessionHostConfig, agentType string, sessionID string, status SessionHostStatus) (*SessionHost, *sessionHostClient) { t.Helper() diff --git a/packages/vm-agent/internal/acp/session_host_reporting.go b/packages/vm-agent/internal/acp/session_host_reporting.go index 31cdd653d5..c09310bfb2 100644 --- a/packages/vm-agent/internal/acp/session_host_reporting.go +++ b/packages/vm-agent/internal/acp/session_host_reporting.go @@ -176,11 +176,82 @@ type activityPayload struct { RuntimeWorkProgressAt *int64 `json:"runtimeWorkProgressAt,omitempty"` } +type activityReportRequest struct { + activity string + url string + callbackToken string + payload activityPayload +} + +type activityReportSnapshot struct { + Activity string + NodeID string + PromptStartedAt int64 + HasPromptStartedAt bool + AgentType string + RestartCount int + StatusError string + HasStatusError bool + RuntimeWorkState string + RuntimeWorkCount int + HasRuntimeWorkCount bool + RuntimeWorkSource string + RuntimeWorkProgressAt int64 + HasRuntimeWorkProgressAt bool +} + +func activityReportSnapshotFromPayload(payload activityPayload) activityReportSnapshot { + snapshot := activityReportSnapshot{ + Activity: payload.Activity, + NodeID: payload.NodeID, + AgentType: payload.AgentType, + RestartCount: payload.RestartCount, + RuntimeWorkState: payload.RuntimeWorkState, + RuntimeWorkSource: payload.RuntimeWorkSource, + } + if payload.PromptStartedAt != nil { + snapshot.PromptStartedAt = *payload.PromptStartedAt + snapshot.HasPromptStartedAt = true + } + if payload.StatusError != nil { + snapshot.StatusError = *payload.StatusError + snapshot.HasStatusError = true + } + if payload.RuntimeWorkCount != nil { + snapshot.RuntimeWorkCount = *payload.RuntimeWorkCount + snapshot.HasRuntimeWorkCount = true + } + if payload.RuntimeWorkProgressAt != nil { + snapshot.RuntimeWorkProgressAt = *payload.RuntimeWorkProgressAt + snapshot.HasRuntimeWorkProgressAt = true + } + return snapshot +} + // reportActivity sends a durable activity signal to the control plane. // Prompting reports stay cheap because the periodic re-report loop self-heals // missed starts; terminal/error reports use a larger retry budget. // activity should be "prompting", "idle", "recovering", or "error". func (h *SessionHost) reportActivity(activity string) { + request, ok := h.prepareActivityReport(activity) + if !ok { + return + } + go h.sendActivityReport(request) +} + +func (h *SessionHost) reportCoalescedHarnessActivity() { + request, ok := h.prepareActivityReport(h.activityForHarnessWork()) + if !ok { + return + } + if h.successfulActivityReportMatches(request.payload) { + return + } + h.sendActivityReport(request) +} + +func (h *SessionHost) prepareActivityReport(activity string) (activityReportRequest, bool) { // h.config fields are immutable after construction — no lock needed. projectID := h.config.ProjectID nodeID := h.config.NodeID @@ -194,7 +265,7 @@ func (h *SessionHost) reportActivity(activity string) { "hasNodeID", nodeID != "", "hasControlPlaneURL", controlPlaneURL != "", "hasSessionID", sessionID != "") - return + return activityReportRequest{}, false } // Snapshot state under read lock for the enhanced payload. @@ -234,39 +305,62 @@ func (h *SessionHost) reportActivity(activity string) { payload.StatusError = &redactedStatusErr } - go func() { - url := strings.TrimRight(controlPlaneURL, "/") + - "/api/projects/" + projectID + "/acp-sessions/" + sessionID + "/activity" + return activityReportRequest{ + activity: activity, + url: strings.TrimRight(controlPlaneURL, "/") + + "/api/projects/" + projectID + "/acp-sessions/" + sessionID + "/activity", + callbackToken: callbackToken, + payload: payload, + }, true +} - body, err := json.Marshal(payload) - if err != nil { - slog.Warn("reportActivity: marshal failed", "error", err) - return - } +func (h *SessionHost) sendActivityReport(request activityReportRequest) bool { + body, err := json.Marshal(request.payload) + if err != nil { + slog.Warn("reportActivity: marshal failed", "error", err) + return false + } - maxAttempts, retryBackoff := h.activityReportRetryPolicy(activity) - for attempt := 1; attempt <= maxAttempts; attempt++ { - statusCode, doErr := h.doActivityRequest(url, body, callbackToken) - if doErr != nil { - if attempt < maxAttempts { - slog.Info("reportActivity: attempt failed, retrying", "attempt", attempt, "error", doErr) - time.Sleep(retryBackoff) - continue - } - slog.Warn("reportActivity: all attempts failed", "error", doErr) - return - } - if statusCode >= 500 && attempt < maxAttempts { - slog.Info("reportActivity: server error, retrying", "status", statusCode) + maxAttempts, retryBackoff := h.activityReportRetryPolicy(request.activity) + for attempt := 1; attempt <= maxAttempts; attempt++ { + statusCode, doErr := h.doActivityRequest(request.url, body, request.callbackToken) + if doErr != nil { + if attempt < maxAttempts { + slog.Info("reportActivity: attempt failed, retrying", "attempt", attempt, "error", doErr) time.Sleep(retryBackoff) continue } - if statusCode >= 400 { - slog.Warn("reportActivity: non-2xx response", "status", statusCode) - } - return + slog.Warn("reportActivity: all attempts failed", "error", doErr) + return false + } + if statusCode >= 500 && attempt < maxAttempts { + slog.Info("reportActivity: server error, retrying", "status", statusCode) + time.Sleep(retryBackoff) + continue } - }() + if statusCode >= 400 { + slog.Warn("reportActivity: non-2xx response", "status", statusCode) + return false + } + h.recordSuccessfulActivityReport(request.payload) + return true + } + return false +} + +func (h *SessionHost) successfulActivityReportMatches(payload activityPayload) bool { + snapshot := activityReportSnapshotFromPayload(payload) + h.lastActivityReportMu.Lock() + defer h.lastActivityReportMu.Unlock() + return h.lastActivityReportSet && h.lastActivityReport == snapshot +} + +func (h *SessionHost) recordSuccessfulActivityReport(payload activityPayload) { + snapshot := activityReportSnapshotFromPayload(payload) + h.lastActivityReportMu.Lock() + h.lastActivityReport = snapshot + h.lastActivityReportSet = true + h.lastActivityReportMu.Unlock() } func (h *SessionHost) activityReportRetryPolicy(activity string) (int, time.Duration) { diff --git a/packages/vm-agent/internal/config/config.go b/packages/vm-agent/internal/config/config.go index b3793327a5..d8cc345e7e 100644 --- a/packages/vm-agent/internal/config/config.go +++ b/packages/vm-agent/internal/config/config.go @@ -43,6 +43,11 @@ const ( // prompt is in flight. Override via ACTIVITY_REREPORT_INTERVAL. DefaultACPActivityRereportInterval = 60 * time.Second + // DefaultACPHarnessActivityReportDebounce coalesces high-frequency harness + // lifecycle edges before reporting durable activity. Override via + // ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE. + DefaultACPHarnessActivityReportDebounce = 750 * time.Millisecond + // DefaultClaudeHarnessLifecycleMaxBytes bounds a single `_claude/sdkMessage` // extension notification. Override via CLAUDE_HARNESS_LIFECYCLE_MAX_BYTES. DefaultClaudeHarnessLifecycleMaxBytes = 64 * 1024 @@ -254,6 +259,7 @@ type Config struct { ACPNotifSerializeTimeout time.Duration // Max wait for previous notification processing before delivering next (default: 5s) ACPHeartbeatInterval time.Duration // Interval for direct ACP session heartbeats to control plane (default: 60s, env: ACP_HEARTBEAT_INTERVAL) ACPActivityRereportInterval time.Duration // Re-report prompting while a prompt is active (default: 60s, env: ACTIVITY_REREPORT_INTERVAL) + ACPHarnessActivityReportDebounce time.Duration // Debounce high-frequency harness activity reports (default: 750ms, env: ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE) ClaudeHarnessLifecycleMaxBytes int // Max bytes per harness lifecycle notification (default: 65536, env: CLAUDE_HARNESS_LIFECYCLE_MAX_BYTES) ClaudeHarnessLifecycleMaxTasks int // Max tracked background tasks per snapshot (default: 256, env: CLAUDE_HARNESS_LIFECYCLE_MAX_TASKS) ClaudeHarnessLifecycleMaxIDBytes int // Max identifier length in a lifecycle notification (default: 256, env: CLAUDE_HARNESS_LIFECYCLE_MAX_ID_BYTES) diff --git a/packages/vm-agent/internal/config/config_load.go b/packages/vm-agent/internal/config/config_load.go index efdc77c7ed..bc2c1a875f 100644 --- a/packages/vm-agent/internal/config/config_load.go +++ b/packages/vm-agent/internal/config/config_load.go @@ -158,6 +158,7 @@ func Load() (*Config, error) { ACPNotifSerializeTimeout: getEnvDuration("ACP_NOTIF_SERIALIZE_TIMEOUT", 5*time.Second), ACPHeartbeatInterval: getEnvDuration("ACP_HEARTBEAT_INTERVAL", 60*time.Second), ACPActivityRereportInterval: getEnvDuration("ACTIVITY_REREPORT_INTERVAL", DefaultACPActivityRereportInterval), + ACPHarnessActivityReportDebounce: getEnvDuration("ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE", DefaultACPHarnessActivityReportDebounce), ClaudeHarnessLifecycleMaxBytes: getEnvInt("CLAUDE_HARNESS_LIFECYCLE_MAX_BYTES", DefaultClaudeHarnessLifecycleMaxBytes), ClaudeHarnessLifecycleMaxTasks: getEnvInt("CLAUDE_HARNESS_LIFECYCLE_MAX_TASKS", DefaultClaudeHarnessLifecycleMaxTasks), ClaudeHarnessLifecycleMaxIDBytes: getEnvInt("CLAUDE_HARNESS_LIFECYCLE_MAX_ID_BYTES", DefaultClaudeHarnessLifecycleMaxIDBytes), diff --git a/packages/vm-agent/internal/config/config_test.go b/packages/vm-agent/internal/config/config_test.go index 4985bac16b..66706b078f 100644 --- a/packages/vm-agent/internal/config/config_test.go +++ b/packages/vm-agent/internal/config/config_test.go @@ -379,6 +379,7 @@ func legacyOperationalTimeoutChecks(cfg *Config) []struct { {"JWKSFetchTimeout", cfg.JWKSFetchTimeout, 10 * time.Second}, {"ACPCredentialSyncTimeout", cfg.ACPCredentialSyncTimeout, 10 * time.Second}, {"ACPActivityReportTimeout", cfg.ACPActivityReportTimeout, 10 * time.Second}, + {"ACPHarnessActivityReportDebounce", cfg.ACPHarnessActivityReportDebounce, 750 * time.Millisecond}, {"DevcontainerCachePushTimeout", cfg.DevcontainerCachePushTimeout, 10 * time.Minute}, {"DeployPreflightCommandTimeout", cfg.DeployPreflightCommandTimeout, 15 * time.Second}, {"LogStreamPingWriteTimeout", cfg.LogStreamPingWriteTimeout, 10 * time.Second}, @@ -416,6 +417,7 @@ func TestOperationalTimeoutOverrides(t *testing.T) { t.Setenv("JWKS_FETCH_TIMEOUT", "14s") t.Setenv("ACP_CREDENTIAL_SYNC_TIMEOUT", "16s") t.Setenv("ACP_ACTIVITY_REPORT_TIMEOUT", "17s") + t.Setenv("ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE", "875ms") t.Setenv("WORKSPACE_READY_CALLBACK_TIMEOUT", "33s") t.Setenv("DEVCONTAINER_CACHE_PUSH_TIMEOUT", "11m") t.Setenv("DEPLOY_PREFLIGHT_COMMAND_TIMEOUT", "18s") @@ -441,6 +443,7 @@ func TestOperationalTimeoutOverrides(t *testing.T) { {"JWKSFetchTimeout", cfg.JWKSFetchTimeout, 14 * time.Second}, {"ACPCredentialSyncTimeout", cfg.ACPCredentialSyncTimeout, 16 * time.Second}, {"ACPActivityReportTimeout", cfg.ACPActivityReportTimeout, 17 * time.Second}, + {"ACPHarnessActivityReportDebounce", cfg.ACPHarnessActivityReportDebounce, 875 * time.Millisecond}, {"WorkspaceReadyCallbackTimeout", cfg.WorkspaceReadyCallbackTimeout, 33 * time.Second}, {"DevcontainerCachePushTimeout", cfg.DevcontainerCachePushTimeout, 11 * time.Minute}, {"DeployPreflightCommandTimeout", cfg.DeployPreflightCommandTimeout, 18 * time.Second}, @@ -461,7 +464,8 @@ func TestInvalidOperationalTimeoutParseFallsBackAndRedactsValue(t *testing.T) { "GRACEFUL_SHUTDOWN_TIMEOUT", "SYSTEM_PROVISIONING_TIMEOUT", "CF_IP_FETCH_TIMEOUT", "BOOT_LOG_HTTP_TIMEOUT", "MCP_SHORT_COMMAND_TIMEOUT", "MCP_DIFF_COMMAND_TIMEOUT", "MCP_BUILD_PREPARE_TIMEOUT", "JWKS_FETCH_TIMEOUT", "ACP_CREDENTIAL_SYNC_TIMEOUT", - "ACP_ACTIVITY_REPORT_TIMEOUT", "DEVCONTAINER_CACHE_PUSH_TIMEOUT", + "ACP_ACTIVITY_REPORT_TIMEOUT", "ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE", + "DEVCONTAINER_CACHE_PUSH_TIMEOUT", "DEPLOY_PREFLIGHT_COMMAND_TIMEOUT", "LOG_STREAM_PING_WRITE_TIMEOUT", "WORKSPACE_READY_CALLBACK_TIMEOUT", } @@ -907,6 +911,7 @@ func validConfig() *Config { JWKSFetchTimeout: DefaultJWKSFetchTimeout, ACPCredentialSyncTimeout: DefaultACPCredentialSyncTimeout, ACPActivityReportTimeout: DefaultACPActivityReportTimeout, + ACPHarnessActivityReportDebounce: DefaultACPHarnessActivityReportDebounce, WorkspaceReadyCallbackTimeout: DefaultWorkspaceReadyCallbackTimeout, ErrorReportResponseBytes: DefaultErrorReportResponseMaxBytes, ErrorReportStoredErrBytes: DefaultErrorReportStoredErrorBytes, @@ -947,6 +952,7 @@ func TestValidateOperationalTimeouts(t *testing.T) { {"jwks fetch", func(cfg *Config) { cfg.JWKSFetchTimeout = 0 }, "JWKS_FETCH_TIMEOUT"}, {"credential sync", func(cfg *Config) { cfg.ACPCredentialSyncTimeout = 0 }, "ACP_CREDENTIAL_SYNC_TIMEOUT"}, {"activity report", func(cfg *Config) { cfg.ACPActivityReportTimeout = 0 }, "ACP_ACTIVITY_REPORT_TIMEOUT"}, + {"harness activity debounce", func(cfg *Config) { cfg.ACPHarnessActivityReportDebounce = 0 }, "ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE"}, {"cache push", func(cfg *Config) { cfg.DevcontainerCachePushTimeout = 0 }, "DEVCONTAINER_CACHE_PUSH_TIMEOUT"}, {"deploy preflight", func(cfg *Config) { cfg.DeployPreflightCommandTimeout = 0 }, "DEPLOY_PREFLIGHT_COMMAND_TIMEOUT"}, {"log stream ping write", func(cfg *Config) { cfg.LogStreamPingWriteTimeout = 0 }, "LOG_STREAM_PING_WRITE_TIMEOUT"}, diff --git a/packages/vm-agent/internal/config/helpers.go b/packages/vm-agent/internal/config/helpers.go index c2a3e7969e..2582fc5598 100644 --- a/packages/vm-agent/internal/config/helpers.go +++ b/packages/vm-agent/internal/config/helpers.go @@ -291,6 +291,7 @@ func (c *Config) Validate() error { {"DEVCONTAINER_CACHE_PUSH_TIMEOUT", c.DevcontainerCachePushTimeout}, {"ACP_CREDENTIAL_SYNC_TIMEOUT", c.ACPCredentialSyncTimeout}, {"ACP_ACTIVITY_REPORT_TIMEOUT", c.ACPActivityReportTimeout}, + {"ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE", c.ACPHarnessActivityReportDebounce}, {"JWKS_FETCH_TIMEOUT", c.JWKSFetchTimeout}, } for _, timeout := range requiredTimeouts { diff --git a/packages/vm-agent/internal/server/server.go b/packages/vm-agent/internal/server/server.go index d33256b498..a4c458656e 100644 --- a/packages/vm-agent/internal/server/server.go +++ b/packages/vm-agent/internal/server/server.go @@ -420,6 +420,7 @@ func New(cfg *config.Config) (*Server, error) { PromptRetryInitialDelay: cfg.ACPPromptRetryInitial, PromptRetryMaxDelay: cfg.ACPPromptRetryMax, ActivityRereportInterval: cfg.ACPActivityRereportInterval, + HarnessActivityReportDebounce: cfg.ACPHarnessActivityReportDebounce, ClaudeHarnessLifecycleMaxBytes: cfg.ClaudeHarnessLifecycleMaxBytes, ClaudeHarnessLifecycleMaxTasks: cfg.ClaudeHarnessLifecycleMaxTasks, ClaudeHarnessLifecycleMaxIDBytes: cfg.ClaudeHarnessLifecycleMaxIDBytes, diff --git a/tasks/active/2026-08-25-completiondock-activity-coalescing.md b/tasks/active/2026-08-25-completiondock-activity-coalescing.md index efa0321c65..71d8522498 100644 --- a/tasks/active/2026-08-25-completiondock-activity-coalescing.md +++ b/tasks/active/2026-08-25-completiondock-activity-coalescing.md @@ -19,17 +19,17 @@ The visible failure is a twitchy CompletionDock center button: Stop/Interrupt an ## Implementation checklist -- [ ] Add a debounced, coalescing harness activity reporter in the VM agent. -- [ ] Ensure harness-originated reporting reads current mirrored status only when the debounce fires. -- [ ] Ensure only one harness-originated activity POST is in flight at a time. -- [ ] Track successful activity report snapshots so redundant coalesced reports are skipped while runtime-work state changes still propagate. -- [ ] Keep `markPromptStarted()` / `markPromptDone()` immediate and ensure successful authoritative reports update the coalescer's last-success state. -- [ ] Keep the 60s harness work re-report loop as the reliability backstop. -- [ ] Add a configurable VM-agent debounce interval with a default in the requested 500ms–1s range. -- [ ] Add Go regression tests for burst coalescing, stale prompting suppression after prompt done, successful-report dedupe, retry/no-success behavior, and runtime-work payload preservation. -- [ ] Add a client-side stabilized CompletionDock working signal that delays only working→idle propagation. -- [ ] Ensure idle→working remains immediate so Stop/Interrupt appears immediately. -- [ ] Add web unit tests for working→idle stabilization and reversal swallowing. +- [x] Add a debounced, coalescing harness activity reporter in the VM agent. +- [x] Ensure harness-originated reporting reads current mirrored status only when the debounce fires. +- [x] Ensure only one harness-originated activity POST is in flight at a time. +- [x] Track successful activity report snapshots so redundant coalesced reports are skipped while runtime-work state changes still propagate. +- [x] Keep `markPromptStarted()` / `markPromptDone()` immediate and ensure successful authoritative reports update the coalescer's last-success state. +- [x] Keep the 60s harness work re-report loop as the reliability backstop. +- [x] Add a configurable VM-agent debounce interval with a default in the requested 500ms–1s range. +- [x] Add Go regression tests for burst coalescing, stale prompting suppression after prompt done, successful-report dedupe, retry/no-success behavior, and runtime-work payload preservation. +- [x] Add a client-side stabilized CompletionDock working signal that delays only working→idle propagation. +- [x] Ensure idle→working remains immediate so Stop/Interrupt appears immediately. +- [x] Add web unit tests for working→idle stabilization and reversal swallowing. - [ ] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. - [ ] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. - [ ] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. From 3dc00891f68ce0275891803089d1bd6106fae41c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 10:32:29 +0000 Subject: [PATCH 4/8] chore: record completion dock validation --- .../tests/unit/components/useCompletionDockWorking.test.ts | 2 +- tasks/active/2026-08-25-completiondock-activity-coalescing.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/apps/web/tests/unit/components/useCompletionDockWorking.test.ts b/apps/web/tests/unit/components/useCompletionDockWorking.test.ts index f2e51e1f24..3c677676a4 100644 --- a/apps/web/tests/unit/components/useCompletionDockWorking.test.ts +++ b/apps/web/tests/unit/components/useCompletionDockWorking.test.ts @@ -1,11 +1,11 @@ import { act, renderHook } from '@testing-library/react'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { AgentActivityState } from '../../../src/components/project-message-view/types'; import { COMPLETION_DOCK_IDLE_STABILIZE_MS, useCompletionDockWorking, } from '../../../src/components/project-message-view/useCompletionDockWorking'; -import type { AgentActivityState } from '../../../src/components/project-message-view/types'; describe('useCompletionDockWorking', () => { beforeEach(() => { diff --git a/tasks/active/2026-08-25-completiondock-activity-coalescing.md b/tasks/active/2026-08-25-completiondock-activity-coalescing.md index 71d8522498..ff5de35fce 100644 --- a/tasks/active/2026-08-25-completiondock-activity-coalescing.md +++ b/tasks/active/2026-08-25-completiondock-activity-coalescing.md @@ -30,10 +30,10 @@ The visible failure is a twitchy CompletionDock center button: Stop/Interrupt an - [x] Add a client-side stabilized CompletionDock working signal that delays only working→idle propagation. - [x] Ensure idle→working remains immediate so Stop/Interrupt appears immediately. - [x] Add web unit tests for working→idle stabilization and reversal swallowing. -- [ ] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. +- [x] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. - [ ] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. - [ ] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. -- [ ] Add the process fix to repository agent guidance. +- [x] Add the process fix to repository agent guidance. ## Acceptance criteria From 15b86875ee89c637ffcb85a38e8efd12d86b4892 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 10:37:48 +0000 Subject: [PATCH 5/8] docs: document ACP harness activity debounce --- apps/www/src/content/docs/docs/reference/configuration.md | 1 + apps/www/src/content/docs/docs/reference/vm-agent.md | 1 + 2 files changed, 2 insertions(+) diff --git a/apps/www/src/content/docs/docs/reference/configuration.md b/apps/www/src/content/docs/docs/reference/configuration.md index dbce56963b..88365df05a 100644 --- a/apps/www/src/content/docs/docs/reference/configuration.md +++ b/apps/www/src/content/docs/docs/reference/configuration.md @@ -602,6 +602,7 @@ Webhook damping uses Cloudflare KV's eventually consistent read-update-write beh | `ACP_PROMPT_RETRY_INITIAL_BACKOFF` | `15s` | Initial backoff before retrying transient provider prompt errors | | `ACP_PROMPT_RETRY_MAX_BACKOFF` | `2m` | Max exponential backoff for transient provider prompt retries | | `ACTIVITY_REREPORT_INTERVAL` | `60s` | Re-send prompting activity while a prompt is active | +| `ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE` | `750ms` | Debounce ACP harness/tool-call activity reports before callbacks | | `ACP_CHECKPOINT_PREEMPT_GRACE` | `30s` | Graceful ACP cancel/close wait before harness force-stop | | `ACP_CHECKPOINT_PREEMPT_MAX_GRACE` | `2m` | Maximum caller-selected checkpoint rollover grace | | `ACP_CHECKPOINT_ROLLOVER_TIMEOUT` | `2m` | Full checkpoint restart and strict LoadSession deadline | diff --git a/apps/www/src/content/docs/docs/reference/vm-agent.md b/apps/www/src/content/docs/docs/reference/vm-agent.md index 96051f867c..808aa0ade7 100644 --- a/apps/www/src/content/docs/docs/reference/vm-agent.md +++ b/apps/www/src/content/docs/docs/reference/vm-agent.md @@ -256,6 +256,7 @@ Environment variables set by the cloud-init template: | `ACP_CHECKPOINT_PREEMPT_MAX_GRACE` | `2m` | Maximum `graceMs` accepted by the rollover endpoint | | `ACP_CHECKPOINT_ROLLOVER_TIMEOUT` | `2m` | Deadline for the complete stop, restart, and strict LoadSession operation | | `ACP_NOTIF_SERIALIZE_TIMEOUT` | `5s` | Timeout for ACP notification serialization | +| `ACP_HARNESS_ACTIVITY_REPORT_DEBOUNCE` | `750ms` | Debounce window for coalescing ACP harness/tool-call activity reports before POSTing activity callbacks | | `STANDALONE_CLONE_FILTER` | `blob:none` | Git partial-clone filter for standalone (Cloudflare Container) workspace clones, which run synchronously inside the control plane's create-workspace request (`cloneStandaloneRepository` in `internal/server/standalone_workspace.go`). Set `off` to force full clones. The control plane forwards `CF_CONTAINER_CLONE_FILTER` here. | | `GRACEFUL_SHUTDOWN_TIMEOUT` | `30s` | Max time to wait for VM-agent HTTP server shutdown after SIGTERM | | `SYSTEM_PROVISIONING_TIMEOUT` | `15m` | Max time for workspace host provisioning before bootstrap | From a7b661f70f619d613b22bc6f32fca51b05aeb261 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 10:38:17 +0000 Subject: [PATCH 6/8] chore: record completion dock review --- tasks/active/2026-08-25-completiondock-activity-coalescing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/active/2026-08-25-completiondock-activity-coalescing.md b/tasks/active/2026-08-25-completiondock-activity-coalescing.md index ff5de35fce..5606f79390 100644 --- a/tasks/active/2026-08-25-completiondock-activity-coalescing.md +++ b/tasks/active/2026-08-25-completiondock-activity-coalescing.md @@ -31,7 +31,7 @@ The visible failure is a twitchy CompletionDock center button: Stop/Interrupt an - [x] Ensure idle→working remains immediate so Stop/Interrupt appears immediately. - [x] Add web unit tests for working→idle stabilization and reversal swallowing. - [x] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. -- [ ] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. +- [x] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. - [ ] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. - [x] Add the process fix to repository agent guidance. From 092ced77b2f40e171f029e127d312dc412072cc1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 11:14:51 +0000 Subject: [PATCH 7/8] chore: record staging validation --- ...-08-25-completiondock-activity-coalescing.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tasks/active/2026-08-25-completiondock-activity-coalescing.md b/tasks/active/2026-08-25-completiondock-activity-coalescing.md index 5606f79390..dbb12868f7 100644 --- a/tasks/active/2026-08-25-completiondock-activity-coalescing.md +++ b/tasks/active/2026-08-25-completiondock-activity-coalescing.md @@ -32,7 +32,7 @@ The visible failure is a twitchy CompletionDock center button: Stop/Interrupt an - [x] Add web unit tests for working→idle stabilization and reversal swallowing. - [x] Run the required local quality checks and Playwright visual audit for the changed chat UI surface. - [x] Run specialist reviews: task-completion-validator, go-specialist, ui-ux-specialist, test-engineer, constitution-validator, and env-validator. -- [ ] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. +- [x] Deploy to staging, verify the live app, and provision a VM to verify vm-agent heartbeat/workspace access because `packages/vm-agent` changes. - [x] Add the process fix to repository agent guidance. ## Acceptance criteria @@ -74,3 +74,18 @@ High-frequency runtime lifecycle signals converted directly into unordered cross ### Process fix Update VM-agent guidance so high-frequency callback streams must be debounced/coalesced, single-flight, and regression-tested for ordering/cadence before they can mutate control-plane state. + +## Staging verification + +- Staging deployment run: `32838455761` — deploy and GitHub smoke-tests jobs passed. +- Fresh staging VM run: + - Task: `01M0W9A19ZKHNRTED6QKGGBYW1` + - Chat session: `f719fc22-dbe1-4d97-a852-292fef0a3ee0` + - Node: `01M0W9A6P4GC57Z33PBN6SZYE7` + - Workspace: `01M0W9KMAKSADEY31MM4BG8XN6` + - ACP session: `01M0W9MVKG17N9RPWWYDHN3Z7V` +- VM-agent system info on the fresh node reported branch build `a7b661f70f619d613b22bc6f32fca51b05aeb261` and Go `1.26.6`. +- During active Codex runtime work, session state reported `activity=prompting`, `runtimeWorkState=active`, and `runtimeWorkCount=1`; the live UI exposed the CompletionDock `Interrupt agent` control. +- After the prompt completed, state reported `activity=idle`, `runtimeWorkState=inactive`, and `runtimeWorkCount=0`; the live UI stabilized to `Sleep session`. +- No browser console errors were observed during the live UI check. +- Cleanup completed: `POST /sessions/:sessionId/stop` returned `workspaceDeleted=true`, and final staging `/api/nodes` plus `/api/workspaces` were both `[]`. From 90841fc69029d93e0d6b0636e22d4b488183d000 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Titsworth-Morin?= Date: Tue, 25 Aug 2026 11:21:18 +0000 Subject: [PATCH 8/8] test: avoid token-shaped vm-agent fixtures --- packages/vm-agent/internal/acp/gateway_test.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/vm-agent/internal/acp/gateway_test.go b/packages/vm-agent/internal/acp/gateway_test.go index 8c8e73ba85..927d5ac566 100644 --- a/packages/vm-agent/internal/acp/gateway_test.go +++ b/packages/vm-agent/internal/acp/gateway_test.go @@ -744,19 +744,19 @@ func TestProcessConfig_EnvVarInjection(t *testing.T) { name: "Mistral Vibe API key uses env var", agentType: "mistral-vibe", credential: &agentCredential{ - credential: "mistral-api-key-123", + credential: "mistral-placeholder", credentialKind: "api-key", }, - wantEnvVar: "MISTRAL_API_KEY=mistral-api-key-123", + wantEnvVar: "MISTRAL_API_KEY=mistral-placeholder", }, { name: "Amp API key uses env var", agentType: "amp", credential: &agentCredential{ - credential: "sgamp-api-key-123", + credential: "amp-placeholder", credentialKind: "api-key", }, - wantEnvVar: "AMP_API_KEY=sgamp-api-key-123", + wantEnvVar: "AMP_API_KEY=amp-placeholder", }, }