fix: preserve VM agent releases and explain placement

.agents/skills/env-reference/SKILL.md

@@ -17,7 +17,7 @@ The reference covers:
   - Resource limits (MAX_NODES_PER_USER, MAX_AGENT_SESSIONS_PER_WORKSPACE, etc.)
   - Pagination settings
   - Timeouts (heartbeat, Hetzner API, Cloudflare API, Node Agent)
-  - VM agent rollout requirement (`VM_AGENT_REQUIRED_VERSION`)
+  - Deployment-owned VM agent release metadata (`VM_AGENT_REQUIRED_VERSION`, `VM_AGENT_BUILD_FINGERPRINT`)
   - Audio/Transcription settings
   - Client error reporting settings
   - Generic webhook trigger limits, rate damping, and audit retention

.claude/rules/54-vm-agent-rollout-compatibility.md

@@ -4,16 +4,20 @@ When a change affects VM-agent behavior required for scheduling new work, the ro
 
 Required pattern:
 
-1. Build vm-agent binaries with the deployment commit SHA as the agent version.
+1. Build vm-agent binaries with the publishing deployment commit SHA as the agent version.
 2. Upload the matching binaries/version metadata before deploying Worker code that requires that version.
-3. Generate `VM_AGENT_REQUIRED_VERSION` from the deployment commit SHA; do not hardcode rollout-specific SHAs or ask operators to maintain a manual required version.
-4. If a deployment intentionally skips agent artifacts (`skip_agent`), do not advance the required version.
-5. VM-agent `/ready` and heartbeat callbacks must report the build identity additively so old agents remain protocol-compatible.
-6. Every reusable VM placement path must reject nodes whose reported build differs from the required build: preferred nodes, warm nodes, capacity selectors, TaskRunner readiness/health checks, trial reuse, and manual workspace creation.
-7. Busy incompatible managed VMs must keep active work and receive no new work. Cleanup may retire them only after active work drains.
-8. Cloudflare Instant/cf-container sessions are not reusable VM-pool nodes; do not conflate their baked container image lifecycle with VM node scheduling.
-9. Destructive rollout cleanup must treat an active task's provisioning claim as active work even before a workspace row exists. A node referenced by `tasks.auto_provisioned_node_id` for a queued/delegated/in-progress task is not idle.
-10. Missing build metadata is the normal pre-heartbeat state for a freshly booting VM. Cleanup must preserve a configurable boot grace before retiring an unversioned, unclaimed node.
-11. A state machine waiting on a claimed node must distinguish missing/deleted state from "still booting" and terminalize promptly without returning the gone node to a reusable pool.
+3. Compute a deterministic build-input fingerprint over `packages/vm-agent/**` (including Go dependency/toolchain and Makefile inputs) plus `scripts/deploy/vm-agent-compatibility-version.txt`. Carry the last actually published `VM_AGENT_REQUIRED_VERSION` when the fingerprint is unchanged; advance it to the publishing deployment SHA only after changed binaries are uploaded.
+4. Bump the explicit compatibility marker when a control-plane/agent protocol change requires a new exact build even if agent source is unchanged. Do not hardcode rollout-specific SHAs or ask operators to maintain either release metadata value manually.
+5. If a deployment intentionally skips agent artifacts (`skip_agent`), preserve the prior exact requirement. Reject the deploy when there is no prior published release or when build inputs changed/cannot be proven compatible; never clear the requirement.
+6. VM-agent `/ready` and heartbeat callbacks must report the build identity additively so old agents remain protocol-compatible.
+7. Every reusable VM placement path must reject nodes whose reported build differs from the required build: preferred nodes, warm nodes, capacity selectors, TaskRunner readiness/health checks, trial reuse, and manual workspace creation.
+8. Busy incompatible managed VMs must keep active work and receive no new work. Cleanup may retire them only after active work drains.
+9. Cloudflare Instant/cf-container sessions are not reusable VM-pool nodes; do not conflate their baked container image lifecycle with VM node scheduling.
+10. Destructive rollout cleanup must treat an active task's provisioning claim as active work even before a workspace row exists. A node referenced by `tasks.auto_provisioned_node_id` for a queued/delegated/in-progress task is not idle.
+11. Missing build metadata is the normal pre-heartbeat state for a freshly booting VM. Cleanup must preserve a configurable boot grace before retiring an unversioned, unclaimed node.
+12. A state machine waiting on a claimed node must distinguish missing/deleted state from "still booting" and terminalize promptly without returning the gone node to a reusable pool.
+13. Persist the versioned, allowlisted placement explanation immediately after reusable selection, append typed provisioning/readiness failures, and copy the final record to the workspace. Trials must persist before workspace creation because they have no task row.
+14. Placement APIs, MCP tools, logs, and UI may expose only the shared explanation contract. Never copy raw agent versions, raw metrics JSON, provider errors, credentials, prompts, repository data, environment values, or secrets into placement evidence.
+15. Preserve a real node identifier only for the selected node. Persist every rejected or eligible-but-unselected candidate as a stable `candidate-N` alias so placement evidence cannot disclose another tenant's host identifiers.
 
 Tests for scheduling-affecting VM-agent changes should include a stale-but-otherwise-better candidate losing to a compatible node, preferred/warm stale-node rejection, current fresh-node readiness, active stale-node preservation, idle stale-node retirement, and the pre-heartbeat interleaving where an active task owns an unversioned node before any workspace exists.

.claude/skills/env-reference/SKILL.md

@@ -190,7 +190,7 @@ by the read-only cron-liveness check.
 - `AGENT_SETTINGS_VALIDATION_LIMITS` — Optional JSON object overriding
   agent-settings validation bounds for model IDs, tool lists, additional env
   entries, provider display names, and OpenCode base URLs. See
-  `apps/api/.env.example` and `apps/www/src/content/docs/docs/guides/self-hosting.md` for supported keys
+  `apps/api/.env.example` and `apps/www/src/content/docs/docs/guides/self-hosting.mdx` for supported keys
   and defaults.
 
 ### Pagination
@@ -215,7 +215,8 @@ by the read-only cron-liveness check.
 - `NODE_HEARTBEAT_STALE_SECONDS` — Staleness threshold for node health
 - `NODE_AGENT_READY_TIMEOUT_MS` — Max wait for freshly provisioned node-agent health
 - `NODE_AGENT_READY_POLL_INTERVAL_MS` — Polling interval for fresh-node readiness checks
-- `VM_AGENT_REQUIRED_VERSION` — Deployment-generated required vm-agent build for reusable VM nodes. Official deploys set this from the Git commit SHA after publishing matching binaries; unset disables rollout gating for local/manual or skip-agent deploys.
+- `VM_AGENT_REQUIRED_VERSION` — Deployment-generated exact vm-agent build for reusable VM nodes. Official deploys carry the last published Git SHA when build inputs are unchanged and advance it only after publishing a changed release; unset disables rollout gating only for local/manual development.
+- `VM_AGENT_BUILD_FINGERPRINT` — Deployment-generated fingerprint of vm-agent source, Go dependency/toolchain inputs, build scripts, and the explicit compatibility marker. It is release metadata, not an operator-maintained value.
 - `HETZNER_API_TIMEOUT_MS` — Timeout for Hetzner Cloud API calls (default: 30000)
 - `CF_API_TIMEOUT_MS` — Timeout for Cloudflare DNS API calls (default: 30000)
 - `NODE_AGENT_REQUEST_TIMEOUT_MS` — Timeout for Node Agent HTTP requests (default: 30000)

.github/workflows/deploy-reusable.yml

@@ -22,7 +22,6 @@ on:
 
 env:
   NODE_VERSION: '22'
-  GO_VERSION: '1.25'
 
 jobs:
   validate:
@@ -123,24 +122,21 @@ jobs:
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.target_commit_sha || github.sha }}
+          # First-rollout release resolution compares the deployed VM-agent SHA
+          # with this commit. Full history keeps that fail-closed inference local.
+          fetch-depth: 0
 
       - name: Resolve and Verify Deployment SHA
         id: deploy-sha
         env:
           EXPECTED_DEPLOY_SHA: ${{ inputs.target_commit_sha || github.sha }}
-          SKIP_AGENT: ${{ inputs.skip_agent }}
         run: |
           ACTUAL_DEPLOY_SHA=$(git rev-parse HEAD)
           if [ "$ACTUAL_DEPLOY_SHA" != "$EXPECTED_DEPLOY_SHA" ]; then
             echo "::error::Checked out SHA $ACTUAL_DEPLOY_SHA does not match verified deployment SHA $EXPECTED_DEPLOY_SHA"
             exit 1
           fi
           echo "value=$ACTUAL_DEPLOY_SHA" >> "$GITHUB_OUTPUT"
-          if [ "$SKIP_AGENT" = "true" ]; then
-            echo "agent_version=" >> "$GITHUB_OUTPUT"
-          else
-            echo "agent_version=$ACTUAL_DEPLOY_SHA" >> "$GITHUB_OUTPUT"
-          fi
 
       - name: Setup pnpm
         uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9
@@ -367,6 +363,19 @@ jobs:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }}
           CLOUDFLARE_API_USER_SERVICE_KEY: ${{ secrets.CF_ORIGIN_CA_KEY }}
 
+      - name: Resolve VM Agent Release
+        id: vm-agent-release
+        if: ${{ inputs.dry_run != true }}
+        run: pnpm tsx scripts/deploy/resolve-vm-agent-release.ts
+        env:
+          DEPLOY_SHA: ${{ steps.deploy-sha.outputs.value }}
+          PULUMI_STACK: ${{ steps.pulumi-select.outputs.stack_name }}
+          BASE_DOMAIN: ${{ vars.BASE_DOMAIN }}
+          RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }}
+          CF_ACCOUNT_ID: ${{ secrets.CF_ACCOUNT_ID }}
+          CF_API_TOKEN: ${{ secrets.CF_API_TOKEN }}
+          SKIP_AGENT: ${{ inputs.skip_agent }}
+
       # ========================================
       # Phase 2: Configuration
       # ========================================
@@ -382,7 +391,8 @@ jobs:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }}
           BASE_DOMAIN: ${{ vars.BASE_DOMAIN }}
           RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }}
-          VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }}
+          VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}
+          VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }}
           REQUIRE_APPROVAL: ${{ vars.REQUIRE_APPROVAL }}
           CRON_SWEEPS_ENABLED_KV_KEY: ${{ vars.CRON_SWEEPS_ENABLED_KV_KEY }}
           DO_ALARMS_ENABLED_KV_KEY: ${{ vars.DO_ALARMS_ENABLED_KV_KEY }}
@@ -589,7 +599,9 @@ jobs:
         if: ${{ inputs.dry_run != true }}
         uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
         with:
-          go-version: ${{ env.GO_VERSION }}
+          # go.mod is part of the reusable-VM fingerprint, so toolchain changes
+          # advance the published release deterministically.
+          go-version-file: packages/vm-agent/go.mod
 
       - name: Prepare Versioned VM Agent Container Artifact
         if: ${{ inputs.dry_run != true }}
@@ -624,22 +636,19 @@ jobs:
           D1_MIGRATION_CHURNING_TABLES: ${{ vars.D1_MIGRATION_CHURNING_TABLES }}
           D1_MIGRATION_CHURNING_TABLE_MAX_DECREASE_PERCENT: ${{ vars.D1_MIGRATION_CHURNING_TABLE_MAX_DECREASE_PERCENT }}
 
-      # Publish VM-agent binaries before deploying Worker code that may require
-      # this exact build via VM_AGENT_REQUIRED_VERSION. When skip_agent is true,
-      # the sync step leaves VM_AGENT_REQUIRED_VERSION empty so existing nodes are
-      # not drained for binaries this deployment intentionally did not publish.
+      # Publish VM-agent binaries before deploying Worker code that requires the
+      # resolved exact build. Unchanged inputs carry the last published version;
+      # changed or unproven inputs publish first and advance fail-closed.
       - name: Build VM Agent
-        if: ${{ inputs.dry_run != true && inputs.skip_agent != true }}
-        # Pin VERSION to the deploy commit so R2 binaries report the same
-        # version string as the container-baked binary (prepare-container above).
+        if: ${{ inputs.dry_run != true && steps.vm-agent-release.outputs.build_agent == 'true' }}
         run: |
           BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-          make -C packages/vm-agent build-all VERSION="$DEPLOY_SHA" BUILD_DATE="$BUILD_DATE"
+          make -C packages/vm-agent build-all VERSION="$AGENT_VERSION" BUILD_DATE="$BUILD_DATE"
         env:
-          DEPLOY_SHA: ${{ steps.deploy-sha.outputs.value }}
+          AGENT_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}
 
       - name: Upload VM Agent Binaries
-        if: ${{ inputs.dry_run != true && inputs.skip_agent != true }}
+        if: ${{ inputs.dry_run != true && steps.vm-agent-release.outputs.build_agent == 'true' }}
         working-directory: infra
         run: |
           R2_BUCKET=$(pulumi stack output r2Name)
@@ -702,7 +711,8 @@ jobs:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }}
           BASE_DOMAIN: ${{ vars.BASE_DOMAIN }}
           RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }}
-          VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }}
+          VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}
+          VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }}
           REQUIRE_APPROVAL: ${{ vars.REQUIRE_APPROVAL }}
           CRON_SWEEPS_ENABLED_KV_KEY: ${{ vars.CRON_SWEEPS_ENABLED_KV_KEY }}
           DO_ALARMS_ENABLED_KV_KEY: ${{ vars.DO_ALARMS_ENABLED_KV_KEY }}
@@ -1024,8 +1034,10 @@ jobs:
             echo "- Tail Worker (log streaming)" >> $GITHUB_STEP_SUMMARY
             echo "- Web UI (Pages)" >> $GITHUB_STEP_SUMMARY
             echo "- Database migrations" >> $GITHUB_STEP_SUMMARY
-            if [ "${{ inputs.skip_agent }}" != "true" ]; then
+            if [ "${{ steps.vm-agent-release.outputs.build_agent }}" = "true" ]; then
               echo "- VM Agent binaries" >> $GITHUB_STEP_SUMMARY
+            else
+              echo "- VM Agent release carried forward (${{ steps.vm-agent-release.outputs.reason }})" >> $GITHUB_STEP_SUMMARY
             fi
           else
             echo "## Deployment Failed (${{ inputs.environment }})" >> $GITHUB_STEP_SUMMARY

apps/api/.env.example

@@ -257,10 +257,11 @@ BASE_DOMAIN=workspaces.example.com
 # Fresh-node readiness wait before first workspace provisioning (default: 600000ms / 10min)
 # NODE_AGENT_READY_TIMEOUT_MS=600000
 # NODE_AGENT_READY_POLL_INTERVAL_MS=5000
-# Deployment-generated required vm-agent build for reusable VM nodes. Official
-# deploys derive this from the Git commit SHA after uploading matching binaries.
-# Leave unset for local/manual development and skip-agent deploys.
+# Deployment-generated release metadata for reusable VM nodes. Official deploys
+# carry the last published SHA across unrelated changes and advance only after
+# uploading changed binaries. Leave unset only for local/manual development.
 # VM_AGENT_REQUIRED_VERSION=
+# VM_AGENT_BUILD_FINGERPRINT=
 
 # AI task title generation (Workers AI via AI Gateway)
 # TASK_TITLE_MODEL=@cf/zai-org/glm-5.2