Skip to content

fix: normalize session idleness for ACP tool work - #1874

Draft
simple-agent-manager[bot] wants to merge 8 commits into
mainfrom
sam/implement-next-independently-shippable-3wpb6g
Draft

fix: normalize session idleness for ACP tool work#1874
simple-agent-manager[bot] wants to merge 8 commits into
mainfrom
sam/implement-next-independently-shippable-3wpb6g

Conversation

@simple-agent-manager

@simple-agent-manager simple-agent-manager Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a shared classifySessionIdleness() helper for prompt activity, normalized runtime-work lease/ceiling, child-task work, durable-wait signal input, and inconclusive evidence.
  • Converts session sleep eligibility plus the destructive sleep teardown gates (stateBefore, stateAfter, stateAtStop) to use the shared predicate.
  • Normalizes Codex/OpenCode ACP tool_call / tool_call_update lifecycles into the existing VM runtime_work_state lease with source acp_tool_call while preserving Claude _claude/sdkMessage source claude_sdk and cf-container activeWork semantics.
  • Adds D1 child-task signal handling for sleep: active children block sleep; child-query failure is unknown and also blocks sleep.
  • Keeps raw ACP tool inputs/outputs VM-local; activity reports carry only normalized state/count/source/progress.

Converted readers:

  • checkAutomaticSessionSleepEligibility()
  • sleepWorkspaceSession() state-before gate
  • sleepWorkspaceSession() state-after snapshot gate
  • sleepWorkspaceSession() state-at-stop artifact-verification gate

Deliberate gaps:

  • ProjectData idle cleanup and workspace idle-timeout candidate selectors are not converted yet; they still use schedule/workspace-activity candidate selection plus classifyTaskRuntimeLiveness() before terminalization.
  • Sleep consumes D1 child-task state but not ProjectData task_wait_subscriptions directly. The predicate now has activeWaitCount for that follow-up adapter.
  • ACP terminal methods remain stubbed; this PR only consumes ACP tool_call / tool_call_update.

Expected control-loop/candidate cost:

  • The new sleep child-work read is one parameterized D1 query per sleep eligibility/gate check, scoped by project_id + parent_task_id, with active status constants and LIMIT 1. Failure returns inconclusive instead of proving idleness.

Rollback:

  • Revert this PR. That restores the previous Claude-only VM runtime-work adapter and the previous local sleep idleness checks. No schema migration or new env var is introduced.

Validation

  • pnpm lint
  • pnpm typecheck
  • pnpm test
  • Additional validation run (if applicable)
  • If this PR changes candidate selection for a sweep/cron/alarm loop (WHERE clause, status set, join, or equivalent), expected candidate volume and worst-case per-candidate cost are stated in the summary or validation notes (see .claude/rules/47-control-loop-io-budget.md)

Additional validation:

  • CI on head 52a328e67b42fce46f59561b0324482e61306f76 — all visible PR checks passed, including Test, Durable Object Workers, VM Agent Integration/E2E/Test/Smoke, Lint, Type Check, SonarCloud, Preflight Evidence, Specialist Review Evidence, Build, Workspace Quality Surfaces, and Devcontainer Volume Mount.
  • pnpm --filter @simple-agent-manager/api test -- tests/unit/services/session-idleness.test.ts tests/unit/services/session-sleep.test.ts — 45/45 passed.
  • pnpm --filter @simple-agent-manager/api test — 580 files / 7,851 tests passed.
  • pnpm --filter @simple-agent-manager/api lint — passed.
  • pnpm --filter @simple-agent-manager/api typecheck — passed.
  • PATH=<task-local-go>/bin:$PATH go test ./internal/acp -run 'Test(ACPToolCallLifecycle|ACPToolCallsDoNotOverrideClaudeLifecycleSource|ACPToolCallActivityReportsOnlyNormalizedState|ClaudeHarnessLifecycle|HarnessActivityReportsOnlyNormalizedState|HarnessActivityStopsWhenCrashRestartFailsBeforeAttach)' -count=1 — passed.
  • PATH=<task-local-go>/bin:$PATH go test ./... in packages/vm-agent — passed.
  • pnpm format:check — passed at head 52a328e67b42fce46f59561b0324482e61306f76.
  • pnpm lint — passed at head with existing acp-client and apps/web warnings only.
  • pnpm typecheck — passed at head with the documented Astro template-validation baseline.

Staging Verification (REQUIRED for all code changes — merge-blocking)

Draft PR remains draft. Staging was executed only after the parent granted the exclusive slot for exact SHA 52a328e67b42fce46f59561b0324482e61306f76; the slot was released after cleanup proof was accepted. Do not mark ready, merge, redeploy, or perform further staging mutations without renewed parent instruction.

  • Staging deployment greenDeploy Staging workflow run 32455123148 succeeded for branch sam/implement-next-independently-shippable-3wpb6g at exact SHA 52a328e67b42fce46f59561b0324482e61306f76: https://github.com/raphaeltm/simple-agent-manager/actions/runs/32455123148
  • Live app verified via Playwright — logged into app.sammy.party staging with one reused primary storage state (/tmp/pr1874-staging-storage.json) to avoid token-login rate limits.
  • Existing workflows confirmed working — dashboard, projects, settings/cloud-provider, and settings/agents smoke paths loaded with zero captured console/page errors.
  • New feature/fix verified on staging — real Codex/OpenAI VM ACP tool-call runtime work, child/subtask blocking, sleep blocking, child sleep, and wake/recovery were exercised against staging; live gaps are explicitly labeled below.
  • Infrastructure verification completed — successful chain used one Hetzner small VM node (01M0HJR0DYP4HN4W0RK9AG7NNA, provider instance 162988618, runtime vm, credential attribution platform).
  • Mobile and desktop verification notes added for UI changes — N/A: no UI changes; desktop smoke screenshots captured at /tmp/pr1874-staging-screens/{dashboard,projects,settings,settings-agents}.png.

Staging Verification Evidence

Pre-mutation gates:

  • deploy-staging.yml was rechecked with an unfiltered workflow-run listing and had no active queued/in_progress staging deploy before mutation.
  • Branch head was confirmed equal to pinned SHA 52a328e67b42fce46f59561b0324482e61306f76 before the deploy; live PR metadata remains OPEN + draft at that SHA for this handoff update.
  • Staging capacity/state was queried before deployment: zero non-deleted nodes, zero non-sleeping/non-deleted workspaces, and an enabled Hetzner platform cloud credential existed: platform_credentials.id=01KNY6DC06C9QCYQM0389NAGNT, provider=hetzner, credential_type='cloud-provider', is_enabled=1, label Users Staging Hetzner. Missing user cloud credentials were not treated as blockers; the successful chain proved user→platform fallback through credential_attribution_source=platform.

Deployment/UI smoke:

  • Deploy run: 32455123148, conclusion=success, status=completed, headBranch=sam/implement-next-independently-shippable-3wpb6g, headSha=52a328e67b42fce46f59561b0324482e61306f76, createdAt=2026-08-21T06:37:06Z, updatedAt=2026-08-21T06:52:51Z.
  • Playwright smoke paths: dashboard, projects, settings/cloud-provider, settings/agents.
  • Captured console/page errors: 0.
  • Screenshot paths on the validation VM: /tmp/pr1874-staging-screens/dashboard.png, /tmp/pr1874-staging-screens/projects.png, /tmp/pr1874-staging-screens/settings.png, /tmp/pr1874-staging-screens/settings-agents.png.

Primary Codex attempt:

  • Primary profile 01M0HJ2VAPH0T09DEDWC6N5580 (PR1874 Codex 5.5 High Chat mt2lrj69) created and later deleted.
  • Primary task 01M0HJ2Y2ECQ8SP0AEJ92005X9, session 8e757c39-264c-422a-a3bc-c9702309043c, VM node 01M0HJ31P8JBNTVYAVCCK98WMH / provider instance 162987662, workspace 01M0HJ9VDZKWR1QBWSJZ9EE8R9.
  • VM reached ACP liveness, but the smoke user's Codex OAuth credential failed before useful tool validation: Authentication required: Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again. This is a live credential issue, not evidence against the PR behavior. The task/profile/workspace/node were cleaned before retry.

Successful secondary Codex VM chain:

  • Secondary project 01KP5SJ5XZFZFZCQM8YG5H847V (Complex Test Project), temporary profile 01M0HJQR2B6TZNHNX0NW51VCVK (PR1874 Secondary Codex 5.5 High VM mt2m673b), agentType=openai-codex, model=gpt-5.5, effort=high, runtime=vm, taskMode=conversation, workspaceProfile=lightweight, vmSizeOverride=small.
  • Parent task 01M0HJQWA557KWAR4Z68W6N3PM, session 1f2c862b-aeb7-422d-b04f-c26115c63d39, workspace 01M0HK0CW4WE349J28Q4WB1XZE.
  • Child task 01M0HKF3F7P1JBVEZH7JNCHGRE, session b25f0ccf-4668-482e-bac4-391d9c8900e0, workspace 01M0HKF7S4K2EGRCP1A1X59MKX, parent 01M0HJQWA557KWAR4Z68W6N3PM.
  • Recovery task 01M0HMCD728Z8E7366DRA1DP1Q, recovery_source_task_id=01M0HKF3F7P1JBVEZH7JNCHGRE, replacement workspace 01M0HMCG5HETXGYXJNDZVB39PZ.
  • One shared staging VM hosted parent, child, and recovery: node 01M0HJR0DYP4HN4W0RK9AG7NNA, provider instance 162988618, runtime vm, size small, Hetzner, credential_attribution_source=platform.

Live diagnostics and lifecycle evidence:

  1. ACP tool_call/tool_call_update in-flight detection, real diagnostics endpoint:

    • Time 2026-08-21T07:25:06.012Z, GET /api/admin/tasks/01M0HJQWA557KWAR4Z68W6N3PM/reconciliation-diagnostics.
    • Task status=in_progress, executionStep=running, workspaceId=01M0HK0CW4WE349J28Q4WB1XZE.
    • eligible=true, elapsedMs=310912, eligibilityThresholdMs=300000, decision=preserve_live_runtime.
    • liveness.live=true, liveness.conclusive=true, liveness.reason=task_acp_session_live, liveness.workspaceStatus=running, liveness.nodeId=01M0HJR0DYP4HN4W0RK9AG7NNA, liveness.activeAcpSessionId=01M0HK1E06G1AZRCYNVSDE1R7Z.
    • ProjectData session state: agentSessionId=01M0HK1E06G1AZRCYNVSDE1R7Z, agentType=openai-codex, activity=prompting, runtimeWorkState=active, runtimeWorkCount=1, runtimeWorkSource=acp_tool_call, runtimeWorkUpdatedAt=1787297103604, runtimeWorkProgressAt=1787297103423.
  2. Parent idle/awaiting-followup while child/subtask is active:

    • Time 2026-08-21T07:30:55.065Z, same diagnostics endpoint for parent task 01M0HJQWA557KWAR4Z68W6N3PM.
    • Parent task status=in_progress, execution_step=awaiting_followup, workspace_id=01M0HK0CW4WE349J28Q4WB1XZE.
    • Child task 01M0HKF3F7P1JBVEZH7JNCHGRE was status=in_progress, execution_step=running, workspace_id=01M0HKF7S4K2EGRCP1A1X59MKX, parent_task_id=01M0HJQWA557KWAR4Z68W6N3PM.
    • Diagnostics: eligible=true, elapsedMs=661500, eligibilityThresholdMs=300000, decision=preserve_live_runtime, liveness.reason=task_acp_session_live.
    • Parent session state was idle with no current runtime work: activity=idle, runtimeWorkState=inactive, runtimeWorkCount=0, runtimeWorkSource=acp_tool_call, runtimeWorkUpdatedAt=1787297250247, runtimeWorkProgressAt=1787297247112. This is the load-bearing evidence that absence of runtime-work signal alone is not evidence of idleness when child work exists.
  3. Sleep blocked by active child/subtask:

    • POST /api/workspaces/01M0HK0CW4WE349J28Q4WB1XZE/sleep at 2026-08-21T07:36:45.105Z returned HTTP 500 due route error masking (requestId=c350da61-7b02-4361-80cd-2275bf893de2).
    • D1 snapshot state exposed the authoritative internal predicate result: parent session 1f2c862b-aeb7-422d-b04f-c26115c63d39, workspace 01M0HK0CW4WE349J28Q4WB1XZE, status=available, degradation=none, sleep_status=failed, sleep_attempts=1, sleep_error=Child/subtask work is active, sleeping_at=null, updated_at=2026-08-21T07:36:47.950Z.
    • Workspaces remained running after the rejected sleep attempt.
  4. Child terminalization and follow-up behavior:

    • Child terminalized through the normal task status API at 2026-08-21T07:37:34.362Z: POST /api/projects/01KP5SJ5XZFZFZCQM8YG5H847V/tasks/01M0HKF3F7P1JBVEZH7JNCHGRE/status, HTTP 200, child status=completed, completedAt=2026-08-21T07:37:32.665Z.
    • Parent transcript had registered wait_for_subtasks for the child task. Manual user-status terminalization did not immediately resume the parent during the observed window; this is labeled as a live gap below because the low-latency wake hook is on task-runner/MCP terminal paths and ProjectData alarm remains fallback.
  5. Idle interval and parent close behavior:

    • Parent follow-up was accepted at 2026-08-21T07:39:46.249Z with deliveryId=01M0HM5XM65P5BSBCE1BBRPBE1.
    • Parent sleep after child terminalization failed with internal snapshot sleep_error=Workspace idle interval has not elapsed, updated_at=2026-08-21T07:42:06.864Z.
    • Parent close succeeded at 2026-08-21T07:42:42.547Z (closedAt=2026-08-21T07:42:39.808Z); a later explicit sleep call returned 404 Workspace not found because close cleanup had already removed the parent workspace.
  6. Child automatic sleep:

    • Child session b25f0ccf-4668-482e-bac4-391d9c8900e0, workspace 01M0HKF7S4K2EGRCP1A1X59MKX, snapshot: status=available, degradation=none, sleep_status=sleeping, sleep_attempts=1, sleeping_at=2026-08-21T07:40:57.094Z, updated_at=2026-08-21T07:40:57.094Z.
  7. Sleep/wake recovery:

    • Wake prompt accepted for child session at 2026-08-21T07:43:16.696Z: POST /api/projects/01KP5SJ5XZFZFZCQM8YG5H847V/sessions/b25f0ccf-4668-482e-bac4-391d9c8900e0/prompt, HTTP 202, deliveryId=01M0HMCB4HMW03207JAX99FHB7.
    • Recovery created task 01M0HMCD728Z8E7366DRA1DP1Q from source task 01M0HKF3F7P1JBVEZH7JNCHGRE, replacement workspace 01M0HMCG5HETXGYXJNDZVB39PZ, same node 01M0HJR0DYP4HN4W0RK9AG7NNA, session_snapshots.recovery_attempts=1.
    • At 2026-08-21T07:44:46.551Z, recovery task was in_progress/running, workspace 01M0HMCG5HETXGYXJNDZVB39PZ was running, and the transcript contained PR1874_CHILD_WAKE_DONE.

Cleanup proof:

  • Cleanup actions completed via normal APIs between 2026-08-21T07:45:26Z and 2026-08-21T07:45:45Z for recovery/child/parent tasks, profiles, workspaces, and node resources.
  • Accepted final proof at 2026-08-21T07:46:49.281Z: {"t":"2026-08-21T07:46:49.281Z","nonDeletedNodes":[],"activeWorkspaces":[],"prTasks":[],"prProfiles":[]}.
  • Staging slot was released after this proof. No further staging mutations should be performed for this PR until a new explicit grant.

Clearly labeled live gaps:

  • Detached-process completion was not executed live. The parent did not resume immediately after manual child terminalization through the user-status route, and cleanup proceeded after the required evidence was captured. Detached long-lived process completion therefore remains deterministic-test evidence in this draft handoff.
  • The 30-minute absolute ceiling was not waited live. Staging observed renewable active ACP tool lease fields and live preservation past the 5-minute eligibility threshold; the default 30-minute ceiling/expiry remains deterministic-test evidence to avoid holding the exclusive staging slot unnecessarily.
  • Parent sleep/recovery chain did not complete live. The manual follow-up reset the parent idle interval, then close cleanup removed the parent workspace before a second explicit sleep could succeed. Child sleep/wake/recovery completed successfully and is the live recovery evidence for this run.
  • Manual child terminalization through the user status API did not immediately resume the parent during the observed window. This is a route/hook observation to preserve in follow-up work, not a reason to infer idleness from missing runtime-work state.
  • Primary smoke-user Codex OAuth was stale. The successful validation used a secondary Codex user with primary superadmin auth only for admin diagnostics; this does not block the PR behavior but should not be hidden.

UI Compliance Checklist (Required for UI changes)

N/A: no UI changes.

End-to-End Verification (Required for multi-component changes)

  • Data flow traced from user input to final outcome with code path citations (see .claude/rules/10-e2e-verification.md)
  • Capability test exercises the complete happy path across system boundaries
  • All spec/doc assumptions about existing behavior verified against code (not just "read the code")
  • If any gap exists between automated test coverage and full E2E, manual verification steps documented below

Data Flow Trace

  • ACP peer emits session/update with tool_call / tool_call_updatepackages/vm-agent/internal/acp/session_host_client.go:SessionUpdate.
  • SessionUpdate invokes normalized lifecycle handling → packages/vm-agent/internal/acp/session_host_harness_work.go:applyACPToolCallLifecycle / applyACPToolCallStatus.
  • VM Agent reports normalized runtime work, not raw tool payloads → packages/vm-agent/internal/acp/session_host_reporting.go:reportActivity via nudgeHarnessActivityReport().
  • API activity callback persists runtime-work fields in ProjectData → apps/api/src/routes/projects/agent-activity-callback.ts and apps/api/src/durable-objects/project-data/session-state.ts.
  • Sleep eligibility/gates classify idleness using one predicate → apps/api/src/services/session-idleness.ts:classifySessionIdleness from apps/api/src/services/session-sleep.ts.
  • Sleep blocks while runtime work lease, active child task, durable-wait input, prompt activity, or unknown child-work evidence says non-idle.

Remaining Live Gaps

  • Detached-process completion remains deterministic-test evidence only for this draft: the live staging run did not reach the detached-process completion phase before cleanup.
  • The default 30-minute absolute ceiling remains deterministic-test evidence only; live staging observed renewable ACP tool-call lease state but did not wait through the full ceiling.
  • Parent sleep/recovery was not completed live because the parent follow-up reset the idle interval and close cleanup removed the parent workspace. Child sleep/wake/recovery was completed live and is documented in the staging section.
  • ProjectData idle cleanup/workspace idle-timeout selector migration is intentionally deferred; current automated coverage preserves existing runtime-liveness terminalization behavior.
  • Direct ProjectData task_wait_subscriptions reader is not wired into sleep yet; the predicate input is present for the follow-up adapter.

Post-Mortem (Required for bug fix PRs)

What broke

Codex/OpenCode ACP tool calls could be invisible to session idleness, so a session could appear idle after the prompt turn even while an agent-initiated tool/unit was expected to return.

Root cause

PR #1845 added normalized VM runtime-work state for Claude _claude/sdkMessage lifecycle notifications, but standard ACP tool_call / tool_call_update notifications were only persisted as chat metadata and did not feed runtime_work_state.

Class of bug

Cross-runtime lifecycle signal gap / idleness predicate split-brain.

Why it wasn't caught

Coverage existed for Claude background tasks and sleep lease math, but not for harness-agnostic ACP tool-call lifecycle signals or one shared idleness predicate.

Process fix included in this PR

The active task file records the reconciled research, implemented slice, validation, specialist review, and deliberate gaps: tasks/active/2026-08-21-session-idleness-work-lease-slice.md.

Post-mortem file

tasks/active/2026-08-21-session-idleness-work-lease-slice.md

Specialist Review Evidence (Required for agent-authored PRs)

  • All local reviewers completed and findings addressed before merge
  • If any reviewer did NOT complete: needs-human-review label added and merge deferred to human
Reviewer Status Outcome
test-engineer PASS Deterministic coverage added for ACP tool lifecycle, raw payload minimization, shared predicate, child/wait signals, sleep gates, lease expiry/ceiling, and inconclusive evidence.
go-specialist PASS ACP handler stays on existing notification path, uses session mirror before harnessWorkMu, avoids h.mu, and preserves Claude source exclusivity.
constitution-validator PASS No new env vars; lease/ceiling remain configurable; new active status set is a domain constant for child-task liveness.
cloudflare-specialist PASS New D1 child-task query is parameterized, project/parent scoped, LIMIT 1, and failure is inconclusive.
doc-sync-validator PASS No public env/config/API contract changes requiring docs updates; task/PR document the slice and gaps.
security-auditor PASS Raw ACP tool input/output stays VM-local; activity report tests assert secret marker and tool ID do not cross the boundary.
task-completion-validator PASS Checklist matches diff; deferred ProjectData cleanup selector and direct wait-subscription adapter gaps are documented.

Exceptions (If any)

  • Scope: PR remains draft after live staging; do not mark ready, merge, redeploy, or mutate staging without renewed parent instruction.
  • Live gaps: detached-process completion and the 30-minute absolute ceiling were not exercised live and remain deterministic-test evidence; parent sleep/recovery did not complete live, while child sleep/wake/recovery did.
  • Cleanup: staging cleanup proof accepted at 2026-08-21T07:46:49.281Z with zero non-deleted nodes, active workspaces, PR1874 tasks, and PR1874 profiles.
  • Expiration: parent review / renewed merge or staging instruction.

Agent Preflight (Required)

  • Preflight completed before code changes

Classification

  • external-api-change
  • cross-component-change
  • business-logic-change
  • public-surface-change
  • docs-sync-change
  • security-sensitive-change
  • ui-change
  • infra-change

External References

Codebase Impact Analysis

  • packages/vm-agent/internal/acp: ACP tool-call lifecycle signal detection and tests.
  • apps/api/src/services: shared session idleness predicate and sleep-reader conversion.
  • apps/api/tests/unit/services: deterministic idleness/sleep tests.
  • tasks/active: task record/review evidence.

Documentation & Specs

  • Updated tasks/active/2026-08-21-session-idleness-work-lease-slice.md.
  • N/A for public docs: no env var, schema, route, or user-facing config contract changed.

Constitution & Risk Check

  • Principle XI/no hardcoded values checked: no new operator-tunable constants introduced; existing lease/ceiling env controls retained.
  • Primary risks are false sleep due missing signal and indefinite compute pinning. This PR fails closed on unknown child-work evidence, keeps finite lease/absolute ceiling, and documents remaining cleanup-selector gaps.

@codspeed-hq

codspeed-hq Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 6 untouched benchmarks


Comparing sam/implement-next-independently-shippable-3wpb6g (52a328e) with main (35e74d8)

Open in CodSpeed

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant