diff --git a/.agents/skills/env-reference/SKILL.md b/.agents/skills/env-reference/SKILL.md index 26d73b0ed1..5d00e71bf1 100644 --- a/.agents/skills/env-reference/SKILL.md +++ b/.agents/skills/env-reference/SKILL.md @@ -17,7 +17,7 @@ The reference covers: - Resource limits (MAX_NODES_PER_USER, MAX_AGENT_SESSIONS_PER_WORKSPACE, etc.) - Pagination settings - Timeouts (heartbeat, Hetzner API, Cloudflare API, Node Agent) - - VM agent rollout requirement (`VM_AGENT_REQUIRED_VERSION`) + - Deployment-owned VM agent release metadata (`VM_AGENT_REQUIRED_VERSION`, `VM_AGENT_BUILD_FINGERPRINT`) - Audio/Transcription settings - Client error reporting settings - Generic webhook trigger limits, rate damping, and audit retention diff --git a/.claude/rules/54-vm-agent-rollout-compatibility.md b/.claude/rules/54-vm-agent-rollout-compatibility.md index 94dc17399e..10ae94bea1 100644 --- a/.claude/rules/54-vm-agent-rollout-compatibility.md +++ b/.claude/rules/54-vm-agent-rollout-compatibility.md @@ -4,16 +4,20 @@ When a change affects VM-agent behavior required for scheduling new work, the ro Required pattern: -1. Build vm-agent binaries with the deployment commit SHA as the agent version. +1. Build vm-agent binaries with the publishing deployment commit SHA as the agent version. 2. Upload the matching binaries/version metadata before deploying Worker code that requires that version. -3. Generate `VM_AGENT_REQUIRED_VERSION` from the deployment commit SHA; do not hardcode rollout-specific SHAs or ask operators to maintain a manual required version. -4. If a deployment intentionally skips agent artifacts (`skip_agent`), do not advance the required version. -5. VM-agent `/ready` and heartbeat callbacks must report the build identity additively so old agents remain protocol-compatible. -6. Every reusable VM placement path must reject nodes whose reported build differs from the required build: preferred nodes, warm nodes, capacity selectors, TaskRunner readiness/health checks, trial reuse, and manual workspace creation. -7. Busy incompatible managed VMs must keep active work and receive no new work. Cleanup may retire them only after active work drains. -8. Cloudflare Instant/cf-container sessions are not reusable VM-pool nodes; do not conflate their baked container image lifecycle with VM node scheduling. -9. Destructive rollout cleanup must treat an active task's provisioning claim as active work even before a workspace row exists. A node referenced by `tasks.auto_provisioned_node_id` for a queued/delegated/in-progress task is not idle. -10. Missing build metadata is the normal pre-heartbeat state for a freshly booting VM. Cleanup must preserve a configurable boot grace before retiring an unversioned, unclaimed node. -11. A state machine waiting on a claimed node must distinguish missing/deleted state from "still booting" and terminalize promptly without returning the gone node to a reusable pool. +3. Compute a deterministic build-input fingerprint over `packages/vm-agent/**` (including Go dependency/toolchain and Makefile inputs) plus `scripts/deploy/vm-agent-compatibility-version.txt`. Carry the last actually published `VM_AGENT_REQUIRED_VERSION` when the fingerprint is unchanged; advance it to the publishing deployment SHA only after changed binaries are uploaded. +4. Bump the explicit compatibility marker when a control-plane/agent protocol change requires a new exact build even if agent source is unchanged. Do not hardcode rollout-specific SHAs or ask operators to maintain either release metadata value manually. +5. If a deployment intentionally skips agent artifacts (`skip_agent`), preserve the prior exact requirement. Reject the deploy when there is no prior published release or when build inputs changed/cannot be proven compatible; never clear the requirement. +6. VM-agent `/ready` and heartbeat callbacks must report the build identity additively so old agents remain protocol-compatible. +7. Every reusable VM placement path must reject nodes whose reported build differs from the required build: preferred nodes, warm nodes, capacity selectors, TaskRunner readiness/health checks, trial reuse, and manual workspace creation. +8. Busy incompatible managed VMs must keep active work and receive no new work. Cleanup may retire them only after active work drains. +9. Cloudflare Instant/cf-container sessions are not reusable VM-pool nodes; do not conflate their baked container image lifecycle with VM node scheduling. +10. Destructive rollout cleanup must treat an active task's provisioning claim as active work even before a workspace row exists. A node referenced by `tasks.auto_provisioned_node_id` for a queued/delegated/in-progress task is not idle. +11. Missing build metadata is the normal pre-heartbeat state for a freshly booting VM. Cleanup must preserve a configurable boot grace before retiring an unversioned, unclaimed node. +12. A state machine waiting on a claimed node must distinguish missing/deleted state from "still booting" and terminalize promptly without returning the gone node to a reusable pool. +13. Persist the versioned, allowlisted placement explanation immediately after reusable selection, append typed provisioning/readiness failures, and copy the final record to the workspace. Trials must persist before workspace creation because they have no task row. +14. Placement APIs, MCP tools, logs, and UI may expose only the shared explanation contract. Never copy raw agent versions, raw metrics JSON, provider errors, credentials, prompts, repository data, environment values, or secrets into placement evidence. +15. Preserve a real node identifier only for the selected node. Persist every rejected or eligible-but-unselected candidate as a stable `candidate-N` alias so placement evidence cannot disclose another tenant's host identifiers. Tests for scheduling-affecting VM-agent changes should include a stale-but-otherwise-better candidate losing to a compatible node, preferred/warm stale-node rejection, current fresh-node readiness, active stale-node preservation, idle stale-node retirement, and the pre-heartbeat interleaving where an active task owns an unversioned node before any workspace exists. diff --git a/.claude/skills/env-reference/SKILL.md b/.claude/skills/env-reference/SKILL.md index bce6d35c72..f8bef73e61 100644 --- a/.claude/skills/env-reference/SKILL.md +++ b/.claude/skills/env-reference/SKILL.md @@ -190,7 +190,7 @@ by the read-only cron-liveness check. - `AGENT_SETTINGS_VALIDATION_LIMITS` — Optional JSON object overriding agent-settings validation bounds for model IDs, tool lists, additional env entries, provider display names, and OpenCode base URLs. See - `apps/api/.env.example` and `apps/www/src/content/docs/docs/guides/self-hosting.md` for supported keys + `apps/api/.env.example` and `apps/www/src/content/docs/docs/guides/self-hosting.mdx` for supported keys and defaults. ### Pagination @@ -215,7 +215,8 @@ by the read-only cron-liveness check. - `NODE_HEARTBEAT_STALE_SECONDS` — Staleness threshold for node health - `NODE_AGENT_READY_TIMEOUT_MS` — Max wait for freshly provisioned node-agent health - `NODE_AGENT_READY_POLL_INTERVAL_MS` — Polling interval for fresh-node readiness checks -- `VM_AGENT_REQUIRED_VERSION` — Deployment-generated required vm-agent build for reusable VM nodes. Official deploys set this from the Git commit SHA after publishing matching binaries; unset disables rollout gating for local/manual or skip-agent deploys. +- `VM_AGENT_REQUIRED_VERSION` — Deployment-generated exact vm-agent build for reusable VM nodes. Official deploys carry the last published Git SHA when build inputs are unchanged and advance it only after publishing a changed release; unset disables rollout gating only for local/manual development. +- `VM_AGENT_BUILD_FINGERPRINT` — Deployment-generated fingerprint of vm-agent source, Go dependency/toolchain inputs, build scripts, and the explicit compatibility marker. It is release metadata, not an operator-maintained value. - `HETZNER_API_TIMEOUT_MS` — Timeout for Hetzner Cloud API calls (default: 30000) - `CF_API_TIMEOUT_MS` — Timeout for Cloudflare DNS API calls (default: 30000) - `NODE_AGENT_REQUEST_TIMEOUT_MS` — Timeout for Node Agent HTTP requests (default: 30000) diff --git a/.github/workflows/deploy-reusable.yml b/.github/workflows/deploy-reusable.yml index 9d600ae6cf..f7cc154ef7 100644 --- a/.github/workflows/deploy-reusable.yml +++ b/.github/workflows/deploy-reusable.yml @@ -22,7 +22,6 @@ on: env: NODE_VERSION: '22' - GO_VERSION: '1.25' jobs: validate: @@ -123,12 +122,14 @@ jobs: uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.target_commit_sha || github.sha }} + # First-rollout release resolution compares the deployed VM-agent SHA + # with this commit. Full history keeps that fail-closed inference local. + fetch-depth: 0 - name: Resolve and Verify Deployment SHA id: deploy-sha env: EXPECTED_DEPLOY_SHA: ${{ inputs.target_commit_sha || github.sha }} - SKIP_AGENT: ${{ inputs.skip_agent }} run: | ACTUAL_DEPLOY_SHA=$(git rev-parse HEAD) if [ "$ACTUAL_DEPLOY_SHA" != "$EXPECTED_DEPLOY_SHA" ]; then @@ -136,11 +137,6 @@ jobs: exit 1 fi echo "value=$ACTUAL_DEPLOY_SHA" >> "$GITHUB_OUTPUT" - if [ "$SKIP_AGENT" = "true" ]; then - echo "agent_version=" >> "$GITHUB_OUTPUT" - else - echo "agent_version=$ACTUAL_DEPLOY_SHA" >> "$GITHUB_OUTPUT" - fi - name: Setup pnpm uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 @@ -367,6 +363,19 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }} CLOUDFLARE_API_USER_SERVICE_KEY: ${{ secrets.CF_ORIGIN_CA_KEY }} + - name: Resolve VM Agent Release + id: vm-agent-release + if: ${{ inputs.dry_run != true }} + run: pnpm tsx scripts/deploy/resolve-vm-agent-release.ts + env: + DEPLOY_SHA: ${{ steps.deploy-sha.outputs.value }} + PULUMI_STACK: ${{ steps.pulumi-select.outputs.stack_name }} + BASE_DOMAIN: ${{ vars.BASE_DOMAIN }} + RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }} + CF_ACCOUNT_ID: ${{ secrets.CF_ACCOUNT_ID }} + CF_API_TOKEN: ${{ secrets.CF_API_TOKEN }} + SKIP_AGENT: ${{ inputs.skip_agent }} + # ======================================== # Phase 2: Configuration # ======================================== @@ -382,7 +391,8 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }} BASE_DOMAIN: ${{ vars.BASE_DOMAIN }} RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }} - VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }} + VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }} + VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }} REQUIRE_APPROVAL: ${{ vars.REQUIRE_APPROVAL }} CRON_SWEEPS_ENABLED_KV_KEY: ${{ vars.CRON_SWEEPS_ENABLED_KV_KEY }} DO_ALARMS_ENABLED_KV_KEY: ${{ vars.DO_ALARMS_ENABLED_KV_KEY }} @@ -589,7 +599,9 @@ jobs: if: ${{ inputs.dry_run != true }} uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: - go-version: ${{ env.GO_VERSION }} + # go.mod is part of the reusable-VM fingerprint, so toolchain changes + # advance the published release deterministically. + go-version-file: packages/vm-agent/go.mod - name: Prepare Versioned VM Agent Container Artifact if: ${{ inputs.dry_run != true }} @@ -624,22 +636,19 @@ jobs: D1_MIGRATION_CHURNING_TABLES: ${{ vars.D1_MIGRATION_CHURNING_TABLES }} D1_MIGRATION_CHURNING_TABLE_MAX_DECREASE_PERCENT: ${{ vars.D1_MIGRATION_CHURNING_TABLE_MAX_DECREASE_PERCENT }} - # Publish VM-agent binaries before deploying Worker code that may require - # this exact build via VM_AGENT_REQUIRED_VERSION. When skip_agent is true, - # the sync step leaves VM_AGENT_REQUIRED_VERSION empty so existing nodes are - # not drained for binaries this deployment intentionally did not publish. + # Publish VM-agent binaries before deploying Worker code that requires the + # resolved exact build. Unchanged inputs carry the last published version; + # changed or unproven inputs publish first and advance fail-closed. - name: Build VM Agent - if: ${{ inputs.dry_run != true && inputs.skip_agent != true }} - # Pin VERSION to the deploy commit so R2 binaries report the same - # version string as the container-baked binary (prepare-container above). + if: ${{ inputs.dry_run != true && steps.vm-agent-release.outputs.build_agent == 'true' }} run: | BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) - make -C packages/vm-agent build-all VERSION="$DEPLOY_SHA" BUILD_DATE="$BUILD_DATE" + make -C packages/vm-agent build-all VERSION="$AGENT_VERSION" BUILD_DATE="$BUILD_DATE" env: - DEPLOY_SHA: ${{ steps.deploy-sha.outputs.value }} + AGENT_VERSION: ${{ steps.vm-agent-release.outputs.required_version }} - name: Upload VM Agent Binaries - if: ${{ inputs.dry_run != true && inputs.skip_agent != true }} + if: ${{ inputs.dry_run != true && steps.vm-agent-release.outputs.build_agent == 'true' }} working-directory: infra run: | R2_BUCKET=$(pulumi stack output r2Name) @@ -702,7 +711,8 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }} BASE_DOMAIN: ${{ vars.BASE_DOMAIN }} RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }} - VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }} + VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }} + VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }} REQUIRE_APPROVAL: ${{ vars.REQUIRE_APPROVAL }} CRON_SWEEPS_ENABLED_KV_KEY: ${{ vars.CRON_SWEEPS_ENABLED_KV_KEY }} DO_ALARMS_ENABLED_KV_KEY: ${{ vars.DO_ALARMS_ENABLED_KV_KEY }} @@ -1024,8 +1034,10 @@ jobs: echo "- Tail Worker (log streaming)" >> $GITHUB_STEP_SUMMARY echo "- Web UI (Pages)" >> $GITHUB_STEP_SUMMARY echo "- Database migrations" >> $GITHUB_STEP_SUMMARY - if [ "${{ inputs.skip_agent }}" != "true" ]; then + if [ "${{ steps.vm-agent-release.outputs.build_agent }}" = "true" ]; then echo "- VM Agent binaries" >> $GITHUB_STEP_SUMMARY + else + echo "- VM Agent release carried forward (${{ steps.vm-agent-release.outputs.reason }})" >> $GITHUB_STEP_SUMMARY fi else echo "## Deployment Failed (${{ inputs.environment }})" >> $GITHUB_STEP_SUMMARY diff --git a/apps/api/.env.example b/apps/api/.env.example index 66ac5f3c6c..a84ec6e57e 100644 --- a/apps/api/.env.example +++ b/apps/api/.env.example @@ -257,10 +257,11 @@ BASE_DOMAIN=workspaces.example.com # Fresh-node readiness wait before first workspace provisioning (default: 600000ms / 10min) # NODE_AGENT_READY_TIMEOUT_MS=600000 # NODE_AGENT_READY_POLL_INTERVAL_MS=5000 -# Deployment-generated required vm-agent build for reusable VM nodes. Official -# deploys derive this from the Git commit SHA after uploading matching binaries. -# Leave unset for local/manual development and skip-agent deploys. +# Deployment-generated release metadata for reusable VM nodes. Official deploys +# carry the last published SHA across unrelated changes and advance only after +# uploading changed binaries. Leave unset only for local/manual development. # VM_AGENT_REQUIRED_VERSION= +# VM_AGENT_BUILD_FINGERPRINT= # AI task title generation (Workers AI via AI Gateway) # TASK_TITLE_MODEL=@cf/zai-org/glm-5.2 diff --git a/apps/api/openapi/sam-cli.openapi.json b/apps/api/openapi/sam-cli.openapi.json index badc96e8b8..fbc97c1ce6 100644 --- a/apps/api/openapi/sam-cli.openapi.json +++ b/apps/api/openapi/sam-cli.openapi.json @@ -1090,6 +1090,380 @@ ], "additionalProperties": false }, + "PlacementRequestSnapshot": { + "type": "object", + "properties": { + "runtime": { + "type": "string", + "enum": [ + "vm" + ] + }, + "vmSize": { + "type": "string", + "enum": [ + "small", + "medium", + "large" + ] + }, + "vmLocation": { + "type": "string", + "description": "Configured placement location; never a credential value." + }, + "maxWorkspacesPerNode": { + "type": "integer" + }, + "cpuThresholdPercent": { + "type": "integer" + }, + "memoryThresholdPercent": { + "type": "integer" + }, + "heartbeatStaleSeconds": { + "type": "integer" + } + }, + "required": [ + "runtime", + "vmSize", + "vmLocation", + "maxWorkspacesPerNode", + "cpuThresholdPercent", + "memoryThresholdPercent", + "heartbeatStaleSeconds" + ], + "additionalProperties": false + }, + "PlacementNodeSnapshot": { + "type": "object", + "properties": { + "runtime": { + "type": "string", + "enum": [ + "vm", + "other" + ] + }, + "vmSize": { + "type": "string" + }, + "vmLocation": { + "type": "string" + }, + "healthStatus": { + "type": "string", + "enum": [ + "healthy", + "stale", + "unhealthy", + "unknown" + ] + }, + "agentVersionCompatible": { + "type": "boolean" + }, + "heartbeatAgeSeconds": { + "type": "number", + "nullable": true + }, + "activeWorkspaceCount": { + "type": "integer" + }, + "cpuLoadAvg1": { + "type": "number", + "nullable": true + }, + "memoryPercent": { + "type": "number", + "nullable": true + } + }, + "required": [ + "runtime", + "vmSize", + "vmLocation", + "healthStatus", + "agentVersionCompatible", + "heartbeatAgeSeconds", + "activeWorkspaceCount", + "cpuLoadAvg1", + "memoryPercent" + ], + "additionalProperties": false + }, + "PlacementNodeEvaluation": { + "type": "object", + "properties": { + "nodeId": { + "type": "string", + "description": "Selected node ID, or a stable candidate-N alias for an unselected node." + }, + "path": { + "type": "string", + "enum": [ + "preferred", + "warm", + "capacity", + "trial", + "manual" + ] + }, + "accepted": { + "type": "boolean" + }, + "rejectionReasons": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "node-not-found", + "not-running", + "wrong-runtime", + "unhealthy", + "heartbeat-missing", + "heartbeat-stale", + "agent-not-ready", + "agent-version-mismatch", + "undersized", + "workspace-limit", + "cpu-threshold", + "memory-threshold", + "not-warm", + "warm-claim-lost" + ] + } + }, + "snapshot": { + "$ref": "#/components/schemas/PlacementNodeSnapshot" + } + }, + "required": [ + "nodeId", + "path", + "accepted", + "rejectionReasons", + "snapshot" + ], + "additionalProperties": false + }, + "PlacementProvisioningAttempt": { + "type": "object", + "properties": { + "vmSize": { + "type": "string", + "enum": [ + "small", + "medium", + "large" + ] + }, + "vmLocation": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "started", + "succeeded", + "capacity-rejected", + "failed" + ] + }, + "failureReason": { + "type": "string", + "enum": [ + "capacity-unavailable", + "node-limit", + "quota-exceeded", + "credentials-unavailable", + "provider-failed", + "provisioning-timeout", + "readiness-timeout", + "node-unavailable" + ] + } + }, + "required": [ + "vmSize", + "vmLocation", + "outcome" + ], + "additionalProperties": false + }, + "PlacementExplanationV2": { + "type": "object", + "properties": { + "schemaVersion": { + "type": "integer", + "enum": [ + 2 + ] + }, + "outcome": { + "type": "string", + "enum": [ + "reused", + "provisioned", + "failed" + ] + }, + "selectionPath": { + "type": "string", + "enum": [ + "preferred", + "warm", + "capacity", + "trial", + "manual", + "provisioning" + ] + }, + "selectedNodeId": { + "type": "string", + "nullable": true + }, + "summary": { + "type": "string" + }, + "request": { + "$ref": "#/components/schemas/PlacementRequestSnapshot" + }, + "evaluatedNodes": { + "type": "array", + "items": { + "$ref": "#/components/schemas/PlacementNodeEvaluation" + } + }, + "provisioningAttempts": { + "type": "array", + "items": { + "$ref": "#/components/schemas/PlacementProvisioningAttempt" + } + }, + "decidedAt": { + "type": "string", + "format": "date-time" + }, + "updatedAt": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "schemaVersion", + "outcome", + "selectionPath", + "selectedNodeId", + "summary", + "request", + "evaluatedNodes", + "provisioningAttempts", + "decidedAt", + "updatedAt" + ], + "additionalProperties": false + }, + "LegacyPlacementExplanation": { + "type": "object", + "properties": { + "selectedVmSize": { + "type": "string", + "enum": [ + "small", + "medium", + "large" + ] + }, + "vmSizeSource": { + "type": "string", + "enum": [ + "task", + "trigger", + "skill", + "agent-profile", + "project", + "user", + "platform", + "explicit" + ] + }, + "reservation": { + "type": "object", + "properties": { + "cpuMillis": { + "type": "integer" + }, + "memoryMb": { + "type": "integer" + }, + "diskMb": { + "type": "integer" + }, + "exclusiveNode": { + "type": "boolean" + }, + "maxCoTenants": { + "type": "integer" + }, + "source": { + "type": "string", + "enum": [ + "task", + "trigger", + "skill", + "agent-profile", + "project", + "user", + "platform" + ] + }, + "sourceId": { + "type": "string" + }, + "version": { + "type": "integer" + } + }, + "required": [ + "cpuMillis", + "memoryMb", + "diskMb", + "exclusiveNode", + "maxCoTenants", + "source", + "sourceId", + "version" + ], + "additionalProperties": false + }, + "reason": { + "type": "string" + }, + "decidedAt": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "selectedVmSize", + "vmSizeSource", + "reservation", + "reason", + "decidedAt" + ], + "additionalProperties": false + }, + "PlacementExplanation": { + "oneOf": [ + { + "$ref": "#/components/schemas/PlacementExplanationV2" + }, + { + "$ref": "#/components/schemas/LegacyPlacementExplanation" + } + ] + }, "Project": { "type": "object", "properties": { @@ -1552,6 +1926,15 @@ "additionalProperties": true, "nullable": true }, + "placementExplanationJson": { + "type": "string", + "description": "Raw persisted placement JSON retained for backward compatibility.", + "nullable": true + }, + "placementExplanation": { + "$ref": "#/components/schemas/PlacementExplanation", + "nullable": true + }, "errorMessage": { "type": "string", "nullable": true @@ -1643,6 +2026,15 @@ "additionalProperties": true, "nullable": true }, + "placementExplanationJson": { + "type": "string", + "description": "Raw persisted placement JSON retained for backward compatibility.", + "nullable": true + }, + "placementExplanation": { + "$ref": "#/components/schemas/PlacementExplanation", + "nullable": true + }, "errorMessage": { "type": "string", "nullable": true @@ -2348,6 +2740,10 @@ "branch": { "type": "string" }, + "placementExplanation": { + "$ref": "#/components/schemas/PlacementExplanation", + "nullable": true + }, "createdAt": { "type": "string", "format": "date-time" diff --git a/apps/api/src/db/migrations/0110_trial_placement_explanation.sql b/apps/api/src/db/migrations/0110_trial_placement_explanation.sql new file mode 100644 index 0000000000..7bbebb038d --- /dev/null +++ b/apps/api/src/db/migrations/0110_trial_placement_explanation.sql @@ -0,0 +1,2 @@ +-- Persist node placement evidence even when a trial fails before creating a workspace. +ALTER TABLE trials ADD COLUMN placement_explanation_json TEXT; diff --git a/apps/api/src/db/schema.ts b/apps/api/src/db/schema.ts index 16291a466e..669ebf2549 100644 --- a/apps/api/src/db/schema.ts +++ b/apps/api/src/db/schema.ts @@ -2256,6 +2256,8 @@ export const trials = sqliteTable( claimedAt: integer('claimed_at'), // epoch ms, nullable errorCode: text('error_code'), errorMessage: text('error_message'), + /** Versioned, non-sensitive node placement audit record. */ + placementExplanationJson: text('placement_explanation_json'), }, (table) => ({ fingerprintIdx: index('idx_trials_fingerprint').on(table.fingerprint, table.createdAt), diff --git a/apps/api/src/durable-objects/task-runner/claimed-node-availability.ts b/apps/api/src/durable-objects/task-runner/claimed-node-availability.ts index 06768eb044..7e2a0fd99c 100644 --- a/apps/api/src/durable-objects/task-runner/claimed-node-availability.ts +++ b/apps/api/src/durable-objects/task-runner/claimed-node-availability.ts @@ -5,6 +5,7 @@ * the resource is already missing/deleted and must not re-enter the warm pool. */ import { log } from '../../lib/logger'; +import { recordPlacementFailure } from './placement'; import type { TaskRunnerContext, TaskRunnerState } from './types'; export async function assertClaimedNodeAvailable( @@ -22,6 +23,7 @@ export async function assertClaimedNodeAvailable( state.stepResults.autoProvisioned = false; await rc.ctx.storage.put('state', state); + await recordPlacementFailure(state, rc, 'node-unavailable'); log.error('task_runner_do.claimed_node_unavailable', { taskId: state.taskId, diff --git a/apps/api/src/durable-objects/task-runner/node-selection.ts b/apps/api/src/durable-objects/task-runner/node-selection.ts index 65a37e4e37..ea8c6332fe 100644 --- a/apps/api/src/durable-objects/task-runner/node-selection.ts +++ b/apps/api/src/durable-objects/task-runner/node-selection.ts @@ -1,27 +1,10 @@ -/** - * Reusable-node health, warm-pool claim, and capacity-selection helpers. - * - * Kept separate from the node step handlers so provisioning and placement - * policy remain independently reviewable. See rule 18. - */ -import { - canSatisfyVmSize, - DEFAULT_MAX_WORKSPACES_PER_NODE, - DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, -} from '@simple-agent-manager/shared'; - -import { log } from '../../lib/logger'; +/** Reusable-node heartbeat verification retained for readiness/recovery checks. */ import { isNodeAgentVersionCompatible } from '../../services/node-agent-compatibility'; -import type { NodeLifecycle } from '../node-lifecycle'; -import { parseEnvInt } from './helpers'; -import type { TaskRunnerContext, TaskRunnerState } from './types'; +import type { TaskRunnerContext } from './types'; /** - * Verify that the VM agent on a node is actually healthy by checking D1 - * heartbeat records. We cannot fetch the VM directly because Cloudflare - * same-zone routing intercepts Worker subrequests to vm-* hostnames, - * routing them back to this API Worker instead of the VM agent. + * Verify that the VM agent on a node is healthy using D1 heartbeat records. + * Reusable placement itself is centralized in services/node-selector.ts. */ export async function verifyNodeAgentHealthy( nodeId: string, @@ -49,211 +32,10 @@ export async function verifyNodeAgentHealthy( return false; } - // Consider node healthy if heartbeat is within the stale threshold - const staleSeconds = parseInt(rc.env.NODE_HEARTBEAT_STALE_SECONDS || '180', 10); + const staleSeconds = Number.parseInt(rc.env.NODE_HEARTBEAT_STALE_SECONDS || '180', 10); const heartbeatAge = (Date.now() - new Date(node.last_heartbeat_at).getTime()) / 1000; return heartbeatAge < staleSeconds; } catch { return false; } } - -export async function tryClaimWarmNode( - state: TaskRunnerState, - rc: TaskRunnerContext -): Promise { - if (!rc.env.NODE_LIFECYCLE) return null; - - const warmNodes = await rc.env.DATABASE.prepare( - `SELECT id, vm_size, vm_location, agent_version FROM nodes - WHERE user_id = ? AND status = 'running' AND warm_since IS NOT NULL AND node_role = 'workspace' - AND (runtime IS NULL OR runtime != 'cf-container')` - ) - .bind(state.userId) - .all<{ id: string; vm_size: string; vm_location: string; agent_version: string | null }>(); - - if (!warmNodes.results.length) return null; - - // Sort nodes that can satisfy the requested size, preferring exact size/location. - const sorted = warmNodes.results - .filter((node) => - isNodeAgentVersionCompatible(node.agent_version, rc.env.VM_AGENT_REQUIRED_VERSION) - ) - .filter((node) => canSatisfyVmSize(node.vm_size, state.config.vmSize)) - .sort((a, b) => { - const aSizeMatch = a.vm_size === state.config.vmSize ? 1 : 0; - const bSizeMatch = b.vm_size === state.config.vmSize ? 1 : 0; - if (aSizeMatch !== bSizeMatch) return bSizeMatch - aSizeMatch; - const aLocMatch = a.vm_location === state.config.vmLocation ? 1 : 0; - const bLocMatch = b.vm_location === state.config.vmLocation ? 1 : 0; - return bLocMatch - aLocMatch; - }); - - for (const warmNode of sorted) { - try { - // Re-check freshness - const fresh = await rc.env.DATABASE.prepare( - `SELECT status, warm_since, agent_version FROM nodes WHERE id = ? AND status = 'running' AND warm_since IS NOT NULL` - ) - .bind(warmNode.id) - .first<{ status: string; warm_since: string | null; agent_version: string | null }>(); - - if ( - !fresh || - !isNodeAgentVersionCompatible(fresh.agent_version, rc.env.VM_AGENT_REQUIRED_VERSION) - ) { - continue; - } - - // Try to claim via NodeLifecycle DO - const doId = rc.env.NODE_LIFECYCLE.idFromName(warmNode.id); - const stub = rc.env.NODE_LIFECYCLE.get(doId) as DurableObjectStub; - const result = (await stub.tryClaim(state.taskId)) as { claimed: boolean }; - - if (result.claimed) { - // Defense-in-depth: verify workspace count even for warm nodes - const wsCount = await rc.env.DATABASE.prepare( - `SELECT COUNT(*) as c FROM workspaces WHERE node_id = ? AND status IN ('running', 'creating', 'recovery')` - ) - .bind(warmNode.id) - .first<{ c: number }>(); - const warmMaxWs = - state.config.projectScaling?.maxWorkspacesPerNode ?? - parseEnvInt(rc.env.MAX_WORKSPACES_PER_NODE, DEFAULT_MAX_WORKSPACES_PER_NODE); - if ((wsCount?.c ?? 0) >= warmMaxWs) { - continue; // At capacity despite being warm — skip - } - log.info('task_runner_do.warm_node_claimed', { - taskId: state.taskId, - nodeId: warmNode.id, - }); - return warmNode.id; - } - } catch { - // Claim failed — try next - } - } - - return null; -} - -export async function findNodeWithCapacity( - state: TaskRunnerState, - rc: TaskRunnerContext -): Promise { - const scaling = state.config.projectScaling; - const cpuThreshold = - scaling?.nodeCpuThresholdPercent ?? - parseEnvInt( - rc.env.TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT - ); - const memThreshold = - scaling?.nodeMemoryThresholdPercent ?? - parseEnvInt( - rc.env.TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT - ); - const maxWorkspaces = - scaling?.maxWorkspacesPerNode ?? - parseEnvInt(rc.env.MAX_WORKSPACES_PER_NODE, DEFAULT_MAX_WORKSPACES_PER_NODE); - - const nodes = await rc.env.DATABASE.prepare( - `SELECT id, vm_size, vm_location, health_status, last_metrics, agent_version FROM nodes - WHERE user_id = ? AND status = 'running' AND health_status != 'unhealthy' AND node_role = 'workspace' - AND (runtime IS NULL OR runtime != 'cf-container')` - ) - .bind(state.userId) - .all<{ - id: string; - vm_size: string; - vm_location: string; - health_status: string; - last_metrics: string | null; - agent_version: string | null; - }>(); - - if (!nodes.results.length) return null; - - // Batch workspace count query to avoid N+1 D1 round-trips - const nodeIds = nodes.results.map((n) => n.id); - const placeholders = nodeIds.map(() => '?').join(','); - const wsCounts = await rc.env.DATABASE.prepare( - `SELECT node_id, COUNT(*) as c FROM workspaces - WHERE node_id IN (${placeholders}) - AND status IN ('running', 'creating', 'recovery') - GROUP BY node_id` - ) - .bind(...nodeIds) - .all<{ node_id: string; c: number }>(); - const countByNode = new Map((wsCounts.results ?? []).map((r) => [r.node_id, r.c])); - - type ScoredNode = { - id: string; - vmSize: string; - vmLocation: string; - score: number | null; - }; - - const candidates: ScoredNode[] = []; - - for (const node of nodes.results) { - if (!isNodeAgentVersionCompatible(node.agent_version, rc.env.VM_AGENT_REQUIRED_VERSION)) { - continue; - } - if (!canSatisfyVmSize(node.vm_size, state.config.vmSize)) continue; - - // Hard workspace count limit — reject node regardless of CPU/memory metrics - if ((countByNode.get(node.id) ?? 0) >= maxWorkspaces) continue; - let metrics: { cpuLoadAvg1?: number; memoryPercent?: number } | null = null; - if (node.last_metrics) { - try { - metrics = JSON.parse(node.last_metrics); - } catch { - /* ignore */ - } - } - - if (metrics) { - const cpu = metrics.cpuLoadAvg1 ?? 0; - const mem = metrics.memoryPercent ?? 0; - if (cpu >= cpuThreshold || mem >= memThreshold) continue; - candidates.push({ - id: node.id, - vmSize: node.vm_size, - vmLocation: node.vm_location, - score: cpu * 0.4 + mem * 0.6, - }); - } else { - candidates.push({ - id: node.id, - vmSize: node.vm_size, - vmLocation: node.vm_location, - score: null, - }); - } - } - - if (!candidates.length) return null; - - // Sort: prefer matching location/size, then lowest load - candidates.sort((a, b) => { - const aLoc = a.vmLocation === state.config.vmLocation ? 1 : 0; - const bLoc = b.vmLocation === state.config.vmLocation ? 1 : 0; - if (aLoc !== bLoc) return bLoc - aLoc; - const aSize = a.vmSize === state.config.vmSize ? 1 : 0; - const bSize = b.vmSize === state.config.vmSize ? 1 : 0; - if (aSize !== bSize) return bSize - aSize; - if (a.score === null && b.score === null) return 0; - if (a.score === null) return 1; - if (b.score === null) return -1; - return a.score - b.score; - }); - - const best = candidates[0]; - if (!best) { - // candidates.length was already checked above — this should never happen. - return null; - } - return best.id; -} diff --git a/apps/api/src/durable-objects/task-runner/node-steps.ts b/apps/api/src/durable-objects/task-runner/node-steps.ts index 3449f4a62f..58ada93903 100644 --- a/apps/api/src/durable-objects/task-runner/node-steps.ts +++ b/apps/api/src/durable-objects/task-runner/node-steps.ts @@ -6,13 +6,19 @@ */ import { isTransientCapacityError, ProviderError } from '@simple-agent-manager/providers'; import type { VMSize } from '@simple-agent-manager/shared'; -import { canSatisfyVmSize, vmSizeFallbackChain } from '@simple-agent-manager/shared'; +import { vmSizeFallbackChain } from '@simple-agent-manager/shared'; +import { drizzle } from 'drizzle-orm/d1'; +import * as schema from '../../db/schema'; import { log } from '../../lib/logger'; -import { isNodeAgentVersionCompatible } from '../../services/node-agent-compatibility'; +import { selectNodeWithExplanation } from '../../services/node-selector'; import { assertClaimedNodeAvailable } from './claimed-node-availability'; -import { parseEnvInt } from './helpers'; -import { findNodeWithCapacity, tryClaimWarmNode, verifyNodeAgentHealthy } from './node-selection'; +import { + persistTaskPlacement, + recordPlacementFailure, + recordProvisioningAttempt, +} from './placement'; +import { assertTaskNodeProvisioningAllowed } from './provisioning-guards'; import { isNodeAgentReadyForWorkspaceDispatch } from './readiness'; import type { TaskRunnerContext, TaskRunnerState } from './types'; @@ -33,72 +39,49 @@ export async function handleNodeSelection( preferredNodeId: state.config.preferredNodeId, }); - if (state.config.preferredNodeId) { - // Validate the preferred node - const node = await rc.env.DATABASE.prepare( - `SELECT id, status, vm_size, agent_version FROM nodes WHERE id = ? AND user_id = ?` - ) - .bind(state.config.preferredNodeId, state.userId) - .first<{ id: string; status: string; vm_size: string; agent_version: string | null }>(); - - if (!node || node.status !== 'running') { - throw Object.assign(new Error('Specified node is not available'), { permanent: true }); - } - if (!canSatisfyVmSize(node.vm_size, state.config.vmSize)) { - throw Object.assign(new Error('Specified node is smaller than the requested VM size'), { - permanent: true, - }); - } - if (!isNodeAgentVersionCompatible(node.agent_version, rc.env.VM_AGENT_REQUIRED_VERSION)) { - throw Object.assign(new Error('Specified node is running an incompatible VM agent build'), { - permanent: true, - }); - } - - // Verify the VM agent is actually reachable before reusing - if (await verifyNodeAgentHealthy(node.id, rc)) { - state.stepResults.nodeId = node.id; - await rc.advanceToStep(state, 'workspace_creation'); - return; - } - log.warn('task_runner_do.preferred_node_unhealthy', { + const scaling = state.config.projectScaling; + const placement = await selectNodeWithExplanation( + drizzle(rc.env.DATABASE, { schema }), + state.userId, + rc.env, + { + vmSize: state.config.vmSize, + vmLocation: state.config.vmLocation, taskId: state.taskId, - nodeId: node.id, - }); - throw Object.assign(new Error('Specified node is not reachable'), { permanent: true }); - } - - // Try warm pool first - const nodeId = await tryClaimWarmNode(state, rc); - if (nodeId) { - if (await verifyNodeAgentHealthy(nodeId, rc)) { - state.stepResults.nodeId = nodeId; - await rc.advanceToStep(state, 'workspace_creation'); - return; + preferredNodeId: state.config.preferredNodeId ?? undefined, + preferredOnly: Boolean(state.config.preferredNodeId), + limits: { + maxWorkspacesPerNode: scaling?.maxWorkspacesPerNode ?? undefined, + cpuThresholdPercent: scaling?.nodeCpuThresholdPercent ?? undefined, + memoryThresholdPercent: scaling?.nodeMemoryThresholdPercent ?? undefined, + }, } - // Warm node agent not healthy — fall through to try other options - log.warn('task_runner_do.warm_node_unhealthy', { - taskId: state.taskId, - nodeId, - }); - } + ); + await persistTaskPlacement(state, rc, placement.explanation); - // Try existing running nodes with capacity - const existingNodeId = await findNodeWithCapacity(state, rc); - if (existingNodeId) { - if (await verifyNodeAgentHealthy(existingNodeId, rc)) { - state.stepResults.nodeId = existingNodeId; - await rc.advanceToStep(state, 'workspace_creation'); - return; - } - // Existing node agent not healthy — fall through to provision - log.warn('task_runner_do.existing_node_unhealthy', { - taskId: state.taskId, - nodeId: existingNodeId, - }); + if (placement.node) { + state.stepResults.nodeId = placement.node.id; + await rc.advanceToStep(state, 'workspace_creation'); + return; + } + if (state.config.preferredNodeId) { + // Preferred placement evaluates exactly one candidate. Its persisted ID is + // intentionally aliased when rejected, so inspect that single typed result. + const reasons = placement.explanation.evaluatedNodes[0]?.rejectionReasons; + const message = reasons?.includes('undersized') + ? 'Specified node is smaller than the requested VM size' + : reasons?.includes('agent-version-mismatch') + ? 'Specified node is running an incompatible VM agent build' + : reasons?.some((reason) => + ['unhealthy', 'heartbeat-missing', 'heartbeat-stale', 'agent-not-ready'].includes( + reason + ) + ) + ? 'Specified node is not reachable' + : 'Specified node is not available'; + throw Object.assign(new Error(message), { permanent: true }); } - // No node found — need to provision await rc.advanceToStep(state, 'node_provisioning'); } @@ -183,6 +166,7 @@ export async function handleNodeProvisioning( const elapsed = Date.now() - state.provisioningStartedAt; if (elapsed > timeoutMs) { const minutes = Math.round(timeoutMs / 60_000); + await recordPlacementFailure(state, rc, 'provisioning-timeout'); throw Object.assign( new Error(`Node provisioning timed out after ${minutes} minute${minutes === 1 ? '' : 's'}`), { permanent: true } @@ -190,6 +174,19 @@ export async function handleNodeProvisioning( } if (node?.status === 'running') { + const latestAttempt = state.placementExplanation?.provisioningAttempts.at(-1); + if (latestAttempt?.outcome === 'started') { + await recordProvisioningAttempt( + state, + rc, + { + vmSize: state.config.vmSize, + vmLocation: state.config.vmLocation, + outcome: 'succeeded', + }, + state.stepResults.nodeId + ); + } // Already provisioned — advance await rc.advanceToStep(state, 'node_agent_ready'); return; @@ -204,64 +201,7 @@ export async function handleNodeProvisioning( return; } - // Check user node limit. User-owned (BYO) nodes are excluded — they cost SAM nothing to run, so - // they must not consume an auto-provisioning slot or block cloud provisioning (critique #8). - const maxNodes = parseEnvInt(rc.env.MAX_NODES_PER_USER, 10); - const countResult = await rc.env.DATABASE.prepare( - `SELECT COUNT(*) as c FROM nodes WHERE user_id = ? AND status IN ('running', 'creating', 'recovery') AND node_role = 'workspace' AND node_class != 'user-owned'` - ) - .bind(state.userId) - .first<{ c: number }>(); - - if ((countResult?.c ?? 0) >= maxNodes) { - throw Object.assign(new Error(`Maximum ${maxNodes} nodes allowed. Cannot auto-provision.`), { - permanent: true, - }); - } - - // Re-check quota before provisioning (hard gate for platform compute). - // Resolves credential source for the target provider — not just whether the user - // has ANY cloud credential. A user with a Hetzner credential who provisions on - // Scaleway (platform) must still be quota-enforced. - const quotaEnforcementEnabled = rc.env.COMPUTE_QUOTA_ENFORCEMENT_ENABLED !== 'false'; - if (quotaEnforcementEnabled) { - const { drizzle } = await import('drizzle-orm/d1'); - const drizzleSchema = await import('../../db/schema'); - const db = drizzle(rc.env.DATABASE, { schema: drizzleSchema }); - const { resolveCredentialSource } = await import('../../services/provider-credentials'); - const attributionProjectId = - state.config.credentialAttributionSource === 'project' - ? state.config.credentialAttributionProjectId - : null; - const credResult = await resolveCredentialSource( - db, - state.config.credentialAttributionUserId, - (state.config.cloudProvider as import('@simple-agent-manager/shared').CredentialProvider) ?? - undefined, - attributionProjectId - ); - - if (!credResult) { - throw Object.assign(new Error('No cloud provider credentials available for provisioning.'), { - permanent: true, - }); - } - - if (credResult.credentialSource === 'platform') { - const { checkQuotaForUser } = await import('../../services/compute-quotas'); - const quotaCheck = await checkQuotaForUser(db, state.userId); - - if (!quotaCheck.allowed) { - throw Object.assign( - new Error( - `Monthly compute quota exceeded: ${quotaCheck.used} of ${quotaCheck.limit} vCPU-hours used. ` + - 'Add your own cloud provider credentials or contact your admin.' - ), - { permanent: true } - ); - } - } - } + await assertTaskNodeProvisioningAllowed(state, rc); // Import and call node creation services // We import dynamically to avoid circular dependency issues and @@ -305,6 +245,17 @@ export async function handleNodeProvisioning( .bind(createdNode.id, new Date().toISOString(), state.taskId) .run(); + await recordProvisioningAttempt( + state, + rc, + { + vmSize: size, + vmLocation: state.config.vmLocation, + outcome: 'started', + }, + createdNode.id + ); + log.info('task_runner_do.step.node_provisioning', { taskId: state.taskId, nodeId: createdNode.id, @@ -336,6 +287,17 @@ export async function handleNodeProvisioning( // Any non-capacity provider failure fails fast — never descend on // invalid_config / quota_exceeded / auth_error / rate_limited / unknown. if (!isCapacityFailure) { + await recordProvisioningAttempt( + state, + rc, + { + vmSize: size, + vmLocation: state.config.vmLocation, + outcome: 'failed', + failureReason: 'provider-failed', + }, + createdNode.id + ); const message = err instanceof Error ? err.message : 'Node provisioning failed'; throw Object.assign(new Error(message), { permanent: true }); } @@ -343,6 +305,17 @@ export async function handleNodeProvisioning( // transient_capacity: descend to the next-smaller size if one remains. // The failed node row was already deleted inside provisionNode (decision #1). if (!isLastSize) { + await recordProvisioningAttempt( + state, + rc, + { + vmSize: size, + vmLocation: state.config.vmLocation, + outcome: 'capacity-rejected', + failureReason: 'capacity-unavailable', + }, + null + ); const nextSize = chain[i + 1]; if (nextSize === undefined) { throw Object.assign( @@ -365,6 +338,17 @@ export async function handleNodeProvisioning( chain.length === 1 ? `There were no ${requestedSize} machines available.` : `No capacity for any available VM size (tried ${chain.join(', ')}).`; + await recordProvisioningAttempt( + state, + rc, + { + vmSize: size, + vmLocation: state.config.vmLocation, + outcome: 'failed', + failureReason: 'capacity-unavailable', + }, + null + ); throw Object.assign(new Error(terminalMessage), { permanent: true }); } @@ -376,6 +360,16 @@ export async function handleNodeProvisioning( // provisioned (relevant when we descended below the requested size). state.config.vmSize = size; await rc.ctx.storage.put('state', state); + await recordProvisioningAttempt( + state, + rc, + { + vmSize: size, + vmLocation: state.config.vmLocation, + outcome: 'succeeded', + }, + createdNode.id + ); if (size !== requestedSize) { // Persist the downgraded size on the task so the UI can surface it. @@ -451,6 +445,7 @@ export async function handleNodeAgentReady( const timeoutMs = rc.getAgentReadyTimeoutMs(); const elapsed = Date.now() - agentReadyStartedAt; if (elapsed > timeoutMs) { + await recordPlacementFailure(state, rc, 'readiness-timeout'); throw Object.assign(new Error(`Node agent not ready within ${timeoutMs}ms`), { permanent: true, }); diff --git a/apps/api/src/durable-objects/task-runner/placement.ts b/apps/api/src/durable-objects/task-runner/placement.ts new file mode 100644 index 0000000000..abb5359a61 --- /dev/null +++ b/apps/api/src/durable-objects/task-runner/placement.ts @@ -0,0 +1,51 @@ +import type { + PlacementExplanation, + PlacementProvisioningAttempt, + PlacementProvisioningFailureReason, +} from '@simple-agent-manager/shared'; + +import { log } from '../../lib/logger'; +import { appendProvisioningAttempt, failPlacement } from '../../services/placement-explanation'; +import type { TaskRunnerContext, TaskRunnerState } from './types'; + +export async function persistTaskPlacement( + state: TaskRunnerState, + rc: TaskRunnerContext, + explanation: PlacementExplanation +): Promise { + state.placementExplanation = explanation; + const now = new Date().toISOString(); + await rc.env.DATABASE.prepare( + `UPDATE tasks SET placement_explanation_json = ?, updated_at = ? WHERE id = ?` + ) + .bind(JSON.stringify(explanation), now, state.taskId) + .run(); + await rc.ctx.storage.put('state', state); + log.info('task_runner_do.placement_decided', { + taskId: state.taskId, + placement: explanation, + }); +} + +export async function recordProvisioningAttempt( + state: TaskRunnerState, + rc: TaskRunnerContext, + attempt: PlacementProvisioningAttempt, + selectedNodeId?: string | null +): Promise { + if (!state.placementExplanation) return; + await persistTaskPlacement( + state, + rc, + appendProvisioningAttempt(state.placementExplanation, attempt, selectedNodeId) + ); +} + +export async function recordPlacementFailure( + state: TaskRunnerState, + rc: TaskRunnerContext, + reason: PlacementProvisioningFailureReason +): Promise { + if (!state.placementExplanation || state.placementExplanation.outcome === 'failed') return; + await persistTaskPlacement(state, rc, failPlacement(state.placementExplanation, reason)); +} diff --git a/apps/api/src/durable-objects/task-runner/provisioning-guards.ts b/apps/api/src/durable-objects/task-runner/provisioning-guards.ts new file mode 100644 index 0000000000..5353f821aa --- /dev/null +++ b/apps/api/src/durable-objects/task-runner/provisioning-guards.ts @@ -0,0 +1,61 @@ +import { DEFAULT_MAX_NODES_PER_USER } from '@simple-agent-manager/shared'; +import { drizzle } from 'drizzle-orm/d1'; + +import * as schema from '../../db/schema'; +import { checkQuotaForUser } from '../../services/compute-quotas'; +import { resolveCredentialSource } from '../../services/provider-credentials'; +import { parseEnvInt } from './helpers'; +import { recordPlacementFailure } from './placement'; +import type { TaskRunnerContext, TaskRunnerState } from './types'; + +export async function assertTaskNodeProvisioningAllowed( + state: TaskRunnerState, + rc: TaskRunnerContext +): Promise { + const maxNodes = parseEnvInt(rc.env.MAX_NODES_PER_USER, DEFAULT_MAX_NODES_PER_USER); + const countResult = await rc.env.DATABASE.prepare( + `SELECT COUNT(*) as c FROM nodes WHERE user_id = ? AND status IN ('running', 'creating', 'recovery') AND node_role = 'workspace' AND node_class != 'user-owned'` + ) + .bind(state.userId) + .first<{ c: number }>(); + + if ((countResult?.c ?? 0) >= maxNodes) { + await recordPlacementFailure(state, rc, 'node-limit'); + throw Object.assign(new Error(`Maximum ${maxNodes} nodes allowed. Cannot auto-provision.`), { + permanent: true, + }); + } + + if (rc.env.COMPUTE_QUOTA_ENFORCEMENT_ENABLED === 'false') return; + const db = drizzle(rc.env.DATABASE, { schema }); + const attributionProjectId = + state.config.credentialAttributionSource === 'project' + ? state.config.credentialAttributionProjectId + : null; + const credResult = await resolveCredentialSource( + db, + state.config.credentialAttributionUserId, + state.config.cloudProvider ?? undefined, + attributionProjectId + ); + + if (!credResult) { + await recordPlacementFailure(state, rc, 'credentials-unavailable'); + throw Object.assign(new Error('No cloud provider credentials available for provisioning.'), { + permanent: true, + }); + } + if (credResult.credentialSource === 'platform') { + const quotaCheck = await checkQuotaForUser(db, state.userId); + if (!quotaCheck.allowed) { + await recordPlacementFailure(state, rc, 'quota-exceeded'); + throw Object.assign( + new Error( + `Monthly compute quota exceeded: ${quotaCheck.used} of ${quotaCheck.limit} vCPU-hours used. ` + + 'Add your own cloud provider credentials or contact your admin.' + ), + { permanent: true } + ); + } + } +} diff --git a/apps/api/src/durable-objects/task-runner/state-machine.ts b/apps/api/src/durable-objects/task-runner/state-machine.ts index 07711c2216..c0b154873b 100644 --- a/apps/api/src/durable-objects/task-runner/state-machine.ts +++ b/apps/api/src/durable-objects/task-runner/state-machine.ts @@ -7,6 +7,7 @@ import { log } from '../../lib/logger'; import { persistError, redactSensitiveData } from '../../services/observability'; import { syncTriggerExecutionStatus } from '../../services/trigger-execution-sync'; +import { recordPlacementFailure } from './placement'; import type { TaskRunnerContext, TaskRunnerState } from './types'; // ========================================================================= @@ -208,6 +209,12 @@ export async function failTask( ): Promise { const now = new Date().toISOString(); + if (state.currentStep === 'node_provisioning') { + await recordPlacementFailure(state, rc, 'provider-failed'); + } else if (state.currentStep === 'node_agent_ready') { + await recordPlacementFailure(state, rc, 'readiness-timeout'); + } + log.error('task_runner_do.task_failed', { taskId: state.taskId, step: state.currentStep, diff --git a/apps/api/src/durable-objects/task-runner/types.ts b/apps/api/src/durable-objects/task-runner/types.ts index cdbafd9a20..5e3ff71842 100644 --- a/apps/api/src/durable-objects/task-runner/types.ts +++ b/apps/api/src/durable-objects/task-runner/types.ts @@ -7,6 +7,7 @@ import type { AgentEffort, CredentialProvider, CredentialSource, + PlacementExplanation, ResolvedResourceReservation, ResourceRequirements, ResourceRequirementsSource, @@ -113,6 +114,8 @@ export interface TaskRunnerState { currentStep: TaskExecutionStep; stepResults: StepResults; config: TaskRunConfig; + /** Versioned, non-sensitive placement audit record persisted to D1. */ + placementExplanation?: PlacementExplanation; retryCount: number; workspaceReadyReceived: boolean; workspaceReadyStatus: 'running' | 'recovery' | 'error' | null; diff --git a/apps/api/src/durable-objects/task-runner/workspace-steps.ts b/apps/api/src/durable-objects/task-runner/workspace-steps.ts index 4ebb43a4c2..52b48c157e 100644 --- a/apps/api/src/durable-objects/task-runner/workspace-steps.ts +++ b/apps/api/src/durable-objects/task-runner/workspace-steps.ts @@ -158,6 +158,9 @@ async function createAndProvisionWorkspace( workspaceProfile: state.config.workspaceProfile ?? DEFAULT_WORKSPACE_PROFILE, devcontainerConfigName: state.config.devcontainerConfigName ?? null, agentProfileHint: state.config.agentProfileHint ?? null, + placementExplanationJson: state.placementExplanation + ? JSON.stringify(state.placementExplanation) + : null, createdAt: now, updatedAt: now, }); diff --git a/apps/api/src/durable-objects/trial-orchestrator/index.ts b/apps/api/src/durable-objects/trial-orchestrator/index.ts index 5a96c355c5..cf642dbadc 100644 --- a/apps/api/src/durable-objects/trial-orchestrator/index.ts +++ b/apps/api/src/durable-objects/trial-orchestrator/index.ts @@ -42,6 +42,7 @@ import type { Env } from '../../env'; import { log } from '../../lib/logger'; import { deferAlarmWhenDisabled } from '../../services/operational-kill-switch'; import { computeBackoffMs, isTransientError, parseEnvInt, safeEmitTrialEvent } from './helpers'; +import { recordTrialPlacementFailure } from './placement'; import { handleDiscoveryAgentStart, handleNodeAgentReady, @@ -233,7 +234,7 @@ export class TrialOrchestrator extends DurableObject { const backoff = computeBackoffMs( state.retryCount, this.getRetryBaseDelayMs(), - this.getRetryMaxDelayMs(), + this.getRetryMaxDelayMs() ); await this.ctx.storage.setAlarm(Date.now() + backoff); log.info('trial_orchestrator_do.step_retry_scheduled', { @@ -255,8 +256,17 @@ export class TrialOrchestrator extends DurableObject { private async failTrial( state: TrialOrchestratorState, reason: string, - errorCode: string, + errorCode: string ): Promise { + const failureReason = + state.currentStep === 'node_agent_ready' + ? 'readiness-timeout' + : state.currentStep === 'node_provisioning' + ? 'provider-failed' + : null; + if (failureReason) { + await recordTrialPlacementFailure(state, this.buildContext(), failureReason); + } // Revoke MCP token BEFORE marking state as failed so a leaked token from a // botched/timed-out trial cannot continue hitting MCP endpoints for the // remainder of its 4-hour TTL (DEFAULT_MCP_TOKEN_TTL_SECONDS). Mirrors the @@ -343,63 +353,63 @@ export class TrialOrchestrator extends DurableObject { private getOverallTimeoutMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_OVERALL_TIMEOUT_MS, - DEFAULT_TRIAL_ORCHESTRATOR_OVERALL_TIMEOUT_MS, + DEFAULT_TRIAL_ORCHESTRATOR_OVERALL_TIMEOUT_MS ); } private getRetryBaseDelayMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_RETRY_BASE_DELAY_MS, - DEFAULT_TRIAL_ORCHESTRATOR_RETRY_BASE_DELAY_MS, + DEFAULT_TRIAL_ORCHESTRATOR_RETRY_BASE_DELAY_MS ); } private getRetryMaxDelayMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_RETRY_MAX_DELAY_MS, - DEFAULT_TRIAL_ORCHESTRATOR_RETRY_MAX_DELAY_MS, + DEFAULT_TRIAL_ORCHESTRATOR_RETRY_MAX_DELAY_MS ); } private getMaxRetries(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_STEP_MAX_RETRIES, - DEFAULT_TRIAL_ORCHESTRATOR_STEP_MAX_RETRIES, + DEFAULT_TRIAL_ORCHESTRATOR_STEP_MAX_RETRIES ); } private getWorkspaceReadyTimeoutMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_WORKSPACE_READY_TIMEOUT_MS, - DEFAULT_TRIAL_ORCHESTRATOR_WORKSPACE_READY_TIMEOUT_MS, + DEFAULT_TRIAL_ORCHESTRATOR_WORKSPACE_READY_TIMEOUT_MS ); } private getWorkspaceReadyPollIntervalMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_WORKSPACE_READY_POLL_INTERVAL_MS, - DEFAULT_TRIAL_ORCHESTRATOR_WORKSPACE_READY_POLL_INTERVAL_MS, + DEFAULT_TRIAL_ORCHESTRATOR_WORKSPACE_READY_POLL_INTERVAL_MS ); } private getNodeReadyTimeoutMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_NODE_READY_TIMEOUT_MS, - DEFAULT_TRIAL_ORCHESTRATOR_NODE_READY_TIMEOUT_MS, + DEFAULT_TRIAL_ORCHESTRATOR_NODE_READY_TIMEOUT_MS ); } private getAgentReadyTimeoutMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_AGENT_READY_TIMEOUT_MS, - DEFAULT_TRIAL_ORCHESTRATOR_AGENT_READY_TIMEOUT_MS, + DEFAULT_TRIAL_ORCHESTRATOR_AGENT_READY_TIMEOUT_MS ); } private getHeartbeatSkewMs(): number { return parseEnvInt( this.env.TRIAL_ORCHESTRATOR_HEARTBEAT_SKEW_MS, - DEFAULT_TRIAL_ORCHESTRATOR_HEARTBEAT_SKEW_MS, + DEFAULT_TRIAL_ORCHESTRATOR_HEARTBEAT_SKEW_MS ); } } diff --git a/apps/api/src/durable-objects/trial-orchestrator/placement.ts b/apps/api/src/durable-objects/trial-orchestrator/placement.ts new file mode 100644 index 0000000000..250d0b354a --- /dev/null +++ b/apps/api/src/durable-objects/trial-orchestrator/placement.ts @@ -0,0 +1,48 @@ +import type { + PlacementExplanation, + PlacementProvisioningAttempt, + PlacementProvisioningFailureReason, +} from '@simple-agent-manager/shared'; + +import { log } from '../../lib/logger'; +import { appendProvisioningAttempt, failPlacement } from '../../services/placement-explanation'; +import type { TrialOrchestratorContext, TrialOrchestratorState } from './types'; + +export async function persistTrialPlacement( + state: TrialOrchestratorState, + rc: TrialOrchestratorContext, + explanation: PlacementExplanation +): Promise { + state.placementExplanation = explanation; + await rc.env.DATABASE.prepare(`UPDATE trials SET placement_explanation_json = ? WHERE id = ?`) + .bind(JSON.stringify(explanation), state.trialId) + .run(); + await rc.ctx.storage.put('state', state); + log.info('trial_orchestrator_do.placement_decided', { + trialId: state.trialId, + placement: explanation, + }); +} + +export async function recordTrialProvisioningAttempt( + state: TrialOrchestratorState, + rc: TrialOrchestratorContext, + attempt: PlacementProvisioningAttempt, + nodeId?: string | null +): Promise { + if (!state.placementExplanation) return; + await persistTrialPlacement( + state, + rc, + appendProvisioningAttempt(state.placementExplanation, attempt, nodeId) + ); +} + +export async function recordTrialPlacementFailure( + state: TrialOrchestratorState, + rc: TrialOrchestratorContext, + reason: PlacementProvisioningFailureReason +): Promise { + if (!state.placementExplanation || state.placementExplanation.outcome === 'failed') return; + await persistTrialPlacement(state, rc, failPlacement(state.placementExplanation, reason)); +} diff --git a/apps/api/src/durable-objects/trial-orchestrator/steps.ts b/apps/api/src/durable-objects/trial-orchestrator/steps.ts index 00ae094450..37155b9966 100644 --- a/apps/api/src/durable-objects/trial-orchestrator/steps.ts +++ b/apps/api/src/durable-objects/trial-orchestrator/steps.ts @@ -38,6 +38,7 @@ import { createWorkspaceOnNode, startAgentSessionOnNode, } from '../../services/node-agent'; +import { selectNodeWithExplanation } from '../../services/node-selector'; import { createNodeRecord, provisionNode } from '../../services/nodes'; import * as projectDataService from '../../services/project-data'; import { DISCOVERY_PROMPT } from '../../services/trial/discovery-prompt'; @@ -52,10 +53,12 @@ import { resolveAnonymousUserId, safeEmitTrialEvent, } from './helpers'; -import type { - TrialOrchestratorContext, - TrialOrchestratorState, -} from './types'; +import { + persistTrialPlacement, + recordTrialPlacementFailure, + recordTrialProvisioningAttempt, +} from './placement'; +import type { TrialOrchestratorContext, TrialOrchestratorState } from './types'; /** Default trial workspace profile when TRIAL_DEFAULT_WORKSPACE_PROFILE is unset. */ const DEFAULT_TRIAL_WORKSPACE_PROFILE = 'lightweight'; @@ -83,7 +86,9 @@ async function syncTrialRecord( SET project_id = ? WHERE id = ? AND project_id IS NULL` - ).bind(patch.projectId, state.trialId).run(); + ) + .bind(patch.projectId, state.trialId) + .run(); } catch (err) { log.warn('trial_orchestrator.trial_d1_project_sync_failed', { trialId: state.trialId, @@ -122,11 +127,11 @@ async function syncTrialRecord( async function fetchDefaultBranch( owner: string, repo: string, - env: TrialOrchestratorContext['env'], + env: TrialOrchestratorContext['env'] ): Promise { const timeoutMs = parseEnvInt( env.TRIAL_GITHUB_TIMEOUT_MS, - DEFAULT_TRIAL_KNOWLEDGE_GITHUB_TIMEOUT_MS, + DEFAULT_TRIAL_KNOWLEDGE_GITHUB_TIMEOUT_MS ); const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), timeoutMs); @@ -192,9 +197,9 @@ export async function handleProjectCreation( // Idempotency — a retry after partial progress should pick up the existing row. if (state.projectId) { - const existing = await rc.env.DATABASE.prepare( - `SELECT id FROM projects WHERE id = ?` - ).bind(state.projectId).first<{ id: string }>(); + const existing = await rc.env.DATABASE.prepare(`SELECT id FROM projects WHERE id = ?`) + .bind(state.projectId) + .first<{ id: string }>(); if (existing) { await rc.advanceToStep(state, 'node_selection'); return; @@ -216,8 +221,8 @@ export async function handleProjectCreation( // Probe GitHub for the real default branch — falls back to 'main' on any // failure so master-default repos (e.g. octocat/Hello-World) clone correctly // without breaking main-default repos when GitHub is unreachable. - const defaultBranch = state.defaultBranch - ?? (await fetchDefaultBranch(state.repoOwner, state.repoName, rc.env)); + const defaultBranch = + state.defaultBranch ?? (await fetchDefaultBranch(state.repoOwner, state.repoName, rc.env)); // Persist the resolved branch BEFORE the D1 insert so a crash between here // and the project persist (line ~222) does not cause a retry to re-probe // GitHub and potentially resolve a different value than what is about to @@ -282,7 +287,12 @@ export async function handleNodeSelection( // skip ahead. If a preferred-node survived but isn't healthy anymore we // fall back to provisioning (trials never pin a specific node). if (state.nodeId) { - if (await verifyNodeAgentHealthy(state.nodeId, rc as unknown as import('../task-runner/types').TaskRunnerContext)) { + if ( + await verifyNodeAgentHealthy( + state.nodeId, + rc as unknown as import('../task-runner/types').TaskRunnerContext + ) + ) { await rc.advanceToStep(state, 'workspace_creation'); return; } @@ -294,26 +304,22 @@ export async function handleNodeSelection( await rc.ctx.storage.put('state', state); } - // Find any running node owned by the sentinel user with capacity. The - // sentinel user never has user-level Hetzner credentials, so in practice - // trial fleets always auto-provision on platform credentials. We still try - // reuse first because warm/multi-trial scenarios benefit from it. const userId = resolveAnonymousUserId(rc.env); - const requiredAgentVersion = rc.env.VM_AGENT_REQUIRED_VERSION || null; - const existing = await rc.env.DATABASE.prepare( - `SELECT id FROM nodes - WHERE user_id = ? AND status = 'running' AND health_status = 'healthy' - AND (? IS NULL OR agent_version = ?) - LIMIT 1` - ).bind(userId, requiredAgentVersion, requiredAgentVersion).first<{ id: string }>(); - - if (existing?.id) { - if (await verifyNodeAgentHealthy(existing.id, rc as unknown as import('../task-runner/types').TaskRunnerContext)) { - state.nodeId = existing.id; - await rc.ctx.storage.put('state', state); - await rc.advanceToStep(state, 'workspace_creation'); - return; - } + const vmSize = (rc.env.TRIAL_VM_SIZE as never) ?? DEFAULT_VM_SIZE; + const vmLocation = rc.env.TRIAL_VM_LOCATION ?? DEFAULT_VM_LOCATION; + const placement = await selectNodeWithExplanation( + drizzle(rc.env.DATABASE, { schema }), + userId, + rc.env, + { vmSize, vmLocation, selectionPath: 'trial' } + ); + await persistTrialPlacement(state, rc, placement.explanation); + + if (placement.node) { + state.nodeId = placement.node.id; + await rc.ctx.storage.put('state', state); + await rc.advanceToStep(state, 'workspace_creation'); + return; } await rc.advanceToStep(state, 'node_provisioning'); @@ -338,17 +344,32 @@ export async function handleNodeProvisioning( if (state.nodeId) { const node = await rc.env.DATABASE.prepare( `SELECT status, error_message FROM nodes WHERE id = ?` - ).bind(state.nodeId).first<{ status: string; error_message: string | null }>(); + ) + .bind(state.nodeId) + .first<{ status: string; error_message: string | null }>(); if (node?.status === 'running') { + const latestAttempt = state.placementExplanation?.provisioningAttempts.at(-1); + if (latestAttempt?.outcome === 'started') { + await recordTrialProvisioningAttempt( + state, + rc, + { + vmSize: (rc.env.TRIAL_VM_SIZE as never) ?? DEFAULT_VM_SIZE, + vmLocation: rc.env.TRIAL_VM_LOCATION ?? DEFAULT_VM_LOCATION, + outcome: 'succeeded', + }, + state.nodeId + ); + } await rc.advanceToStep(state, 'node_agent_ready'); return; } if (node?.status === 'error' || node?.status === 'stopped') { - throw Object.assign( - new Error(node.error_message || 'Trial node provisioning failed'), - { permanent: true }, - ); + await recordTrialPlacementFailure(state, rc, 'provider-failed'); + throw Object.assign(new Error(node.error_message || 'Trial node provisioning failed'), { + permanent: true, + }); } // Still creating — retry via backoff (caller's alarm loop handles the delay). throw new Error('Node still provisioning — will retry'); @@ -371,6 +392,12 @@ export async function handleNodeProvisioning( state.nodeId = createdNode.id; state.autoProvisionedNode = true; await rc.ctx.storage.put('state', state); + await recordTrialProvisioningAttempt( + state, + rc, + { vmSize, vmLocation, outcome: 'started' }, + createdNode.id + ); log.info('trial_orchestrator.step.node_provisioning_started', { trialId: state.trialId, @@ -402,10 +429,7 @@ export async function handleNodeAgentReady( rc: TrialOrchestratorContext ): Promise { if (!state.nodeId) { - throw Object.assign( - new Error('node_agent_ready entered without nodeId'), - { permanent: true }, - ); + throw Object.assign(new Error('node_agent_ready entered without nodeId'), { permanent: true }); } if (!state.nodeAgentReadyStartedAt) { @@ -416,21 +440,23 @@ export async function handleNodeAgentReady( const timeoutMs = rc.getNodeReadyTimeoutMs(); const elapsed = Date.now() - state.nodeAgentReadyStartedAt; if (elapsed > timeoutMs) { - throw Object.assign( - new Error(`Trial node agent not ready within ${timeoutMs}ms`), - { permanent: true }, - ); + await recordTrialPlacementFailure(state, rc, 'readiness-timeout'); + throw Object.assign(new Error(`Trial node agent not ready within ${timeoutMs}ms`), { + permanent: true, + }); } const node = await rc.env.DATABASE.prepare( `SELECT status, health_status, last_heartbeat_at, agent_ready_at, agent_version FROM nodes WHERE id = ?` - ).bind(state.nodeId).first<{ - status: string | null; - health_status: string | null; - last_heartbeat_at: string | null; - agent_ready_at: string | null; - agent_version: string | null; - }>(); + ) + .bind(state.nodeId) + .first<{ + status: string | null; + health_status: string | null; + last_heartbeat_at: string | null; + agent_ready_at: string | null; + agent_version: string | null; + }>(); if ( isNodeAgentReadyForWorkspaceDispatch( @@ -440,6 +466,19 @@ export async function handleNodeAgentReady( rc.env.VM_AGENT_REQUIRED_VERSION ) ) { + const latestAttempt = state.placementExplanation?.provisioningAttempts.at(-1); + if (latestAttempt?.outcome === 'started') { + await recordTrialProvisioningAttempt( + state, + rc, + { + vmSize: latestAttempt.vmSize, + vmLocation: latestAttempt.vmLocation, + outcome: 'succeeded', + }, + state.nodeId + ); + } await rc.advanceToStep(state, 'workspace_creation'); return; } @@ -468,17 +507,16 @@ export async function handleWorkspaceCreation( }); if (!state.projectId || !state.nodeId) { - throw Object.assign( - new Error('workspace_creation requires projectId and nodeId'), - { permanent: true }, - ); + throw Object.assign(new Error('workspace_creation requires projectId and nodeId'), { + permanent: true, + }); } // Idempotency — if workspace row already exists, just move on. if (state.workspaceId) { - const existing = await rc.env.DATABASE.prepare( - `SELECT id FROM workspaces WHERE id = ?` - ).bind(state.workspaceId).first<{ id: string }>(); + const existing = await rc.env.DATABASE.prepare(`SELECT id FROM workspaces WHERE id = ?`) + .bind(state.workspaceId) + .first<{ id: string }>(); if (existing) { await rc.advanceToStep(state, 'workspace_ready'); return; @@ -516,6 +554,9 @@ export async function handleWorkspaceCreation( vmSize, vmLocation, workspaceProfile: profile, + placementExplanationJson: state.placementExplanation + ? JSON.stringify(state.placementExplanation) + : null, createdAt: now, updatedAt: now, }); @@ -560,10 +601,7 @@ export async function handleWorkspaceReady( }); if (!state.workspaceId) { - throw Object.assign( - new Error('workspace_ready without workspaceId'), - { permanent: true }, - ); + throw Object.assign(new Error('workspace_ready without workspaceId'), { permanent: true }); } if (!state.workspaceReadyStartedAt) { @@ -573,26 +611,26 @@ export async function handleWorkspaceReady( const ws = await rc.env.DATABASE.prepare( `SELECT status, error_message FROM workspaces WHERE id = ?` - ).bind(state.workspaceId).first<{ status: string; error_message: string | null }>(); + ) + .bind(state.workspaceId) + .first<{ status: string; error_message: string | null }>(); if (ws?.status === 'running' || ws?.status === 'recovery') { await rc.advanceToStep(state, 'discovery_agent_start'); return; } if (ws?.status === 'error') { - throw Object.assign( - new Error(ws.error_message || 'Trial workspace creation failed'), - { permanent: true }, - ); + throw Object.assign(new Error(ws.error_message || 'Trial workspace creation failed'), { + permanent: true, + }); } const timeoutMs = rc.getWorkspaceReadyTimeoutMs(); const elapsed = Date.now() - state.workspaceReadyStartedAt; if (elapsed > timeoutMs) { - throw Object.assign( - new Error(`Trial workspace did not become ready within ${timeoutMs}ms`), - { permanent: true }, - ); + throw Object.assign(new Error(`Trial workspace did not become ready within ${timeoutMs}ms`), { + permanent: true, + }); } const pollIntervalMs = rc.getWorkspaceReadyPollIntervalMs(); @@ -618,7 +656,7 @@ export async function handleDiscoveryAgentStart( if (!state.projectId || !state.workspaceId || !state.nodeId) { throw Object.assign( new Error('discovery_agent_start requires projectId, workspaceId, and nodeId'), - { permanent: true }, + { permanent: true } ); } @@ -649,12 +687,14 @@ export async function handleDiscoveryAgentStart( try { await rc.env.DATABASE.prepare( `UPDATE workspaces SET chat_session_id = ?, updated_at = ? WHERE id = ?` - ).bind(chatSessionId, new Date().toISOString(), workspaceId).run(); + ) + .bind(chatSessionId, new Date().toISOString(), workspaceId) + .run(); await projectDataService.linkSessionToWorkspace( rc.env, projectId, chatSessionId, - workspaceId, + workspaceId ); } catch (err) { log.warn('trial_orchestrator.session_link_failed', { @@ -684,10 +724,9 @@ export async function handleDiscoveryAgentStart( } if (!acpSessionId || !chatSessionId) { - throw Object.assign( - new Error('discovery_agent_start lost session ids after creation'), - { permanent: true }, - ); + throw Object.assign(new Error('discovery_agent_start lost session ids after creation'), { + permanent: true, + }); } const resolvedAcpSessionId: string = acpSessionId; const resolvedChatSessionId: string = chatSessionId; @@ -702,7 +741,7 @@ export async function handleDiscoveryAgentStart( rc.env, userId, resolvedChatSessionId, - projectId, + projectId ); state.agentSessionCreatedOnVm = true; await rc.ctx.storage.put('state', state); @@ -732,7 +771,7 @@ export async function handleDiscoveryAgentStart( agentSessionId: resolvedAcpSessionId, createdAt: new Date().toISOString(), }, - rc.env, + rc.env ); state.mcpToken = token; await rc.ctx.storage.put('state', state); @@ -760,7 +799,7 @@ export async function handleDiscoveryAgentStart( initialPrompt, rc.env, userId, - { url: mcpServerUrl, token: state.mcpToken }, + { url: mcpServerUrl, token: state.mcpToken } ); state.agentStartedOnVm = true; await rc.ctx.storage.put('state', state); @@ -789,7 +828,7 @@ export async function handleDiscoveryAgentStart( reason: 'trial_orchestrator.agent_subprocess_started', workspaceId, nodeId, - }, + } ); state.acpAssignedOnVm = true; await rc.ctx.storage.put('state', state); @@ -815,7 +854,7 @@ export async function handleDiscoveryAgentStart( reason: 'trial_orchestrator.agent_subprocess_running', workspaceId, nodeId, - }, + } ); state.acpRunningOnVm = true; await rc.ctx.storage.put('state', state); diff --git a/apps/api/src/durable-objects/trial-orchestrator/types.ts b/apps/api/src/durable-objects/trial-orchestrator/types.ts index 12db0f0abf..2e815c9d69 100644 --- a/apps/api/src/durable-objects/trial-orchestrator/types.ts +++ b/apps/api/src/durable-objects/trial-orchestrator/types.ts @@ -4,6 +4,8 @@ * One DO instance per trialId, alarm-driven, mirrors TaskRunner's pattern. * See `apps/api/src/durable-objects/trial-orchestrator/index.ts` for lifecycle. */ +import type { PlacementExplanation } from '@simple-agent-manager/shared'; + import type { Env } from '../../env'; /** @@ -39,6 +41,8 @@ export interface TrialOrchestratorState { projectId: string | null; nodeId: string | null; autoProvisionedNode: boolean; + /** Versioned, non-sensitive placement audit record persisted to D1. */ + placementExplanation?: PlacementExplanation; workspaceId: string | null; chatSessionId: string | null; acpSessionId: string | null; diff --git a/apps/api/src/env.ts b/apps/api/src/env.ts index dc6f739d90..791d16e994 100644 --- a/apps/api/src/env.ts +++ b/apps/api/src/env.ts @@ -260,7 +260,8 @@ export interface Env extends WebhookTriggerEnv, TaskRecoveryEnv { NODE_HEARTBEAT_STALE_SECONDS?: string; NODE_AGENT_READY_TIMEOUT_MS?: string; NODE_AGENT_READY_POLL_INTERVAL_MS?: string; - VM_AGENT_REQUIRED_VERSION?: string; // Deployment commit SHA required for reusable VM nodes; unset disables rollout gating for local/manual dev + VM_AGENT_REQUIRED_VERSION?: string; // Last published build SHA required for reusable VM nodes; unset disables rollout gating only for local/manual dev + VM_AGENT_BUILD_FINGERPRINT?: string; // Deploy-owned fingerprint for deciding whether the published VM-agent release can be carried forward // Task run configuration (autonomous execution) TASK_RUN_NODE_CPU_THRESHOLD_PERCENT?: string; TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT?: string; diff --git a/apps/api/src/lib/mappers.ts b/apps/api/src/lib/mappers.ts index d4ea0cca8f..68d55f7ae9 100644 --- a/apps/api/src/lib/mappers.ts +++ b/apps/api/src/lib/mappers.ts @@ -24,6 +24,7 @@ import { DEFAULT_WORKSPACE_PROFILE, isTaskExecutionStep, parseCompletionEvidenceJson, + parsePlacementExplanationJson, VALID_PERMISSION_MODES, } from '@simple-agent-manager/shared'; import * as v from 'valibot'; @@ -90,6 +91,7 @@ export function toWorkspaceResponse(ws: schema.Workspace, baseDomain: string): W updatedAt: ws.updatedAt, url: getWorkspaceUrl(ws.id, baseDomain), chatSessionId: ws.chatSessionId ?? null, + placementExplanation: parsePlacementExplanationJson(ws.placementExplanationJson), }; } @@ -199,6 +201,7 @@ export function toTaskResponse( (task.resourceRequirementsSource as Task['resourceRequirementsSource']) ?? null, resolvedReservationJson: task.resolvedReservationJson ?? null, placementExplanationJson: task.placementExplanationJson ?? null, + placementExplanation: parsePlacementExplanationJson(task.placementExplanationJson), startedAt: task.startedAt, completedAt: task.completedAt, errorMessage: task.errorMessage, diff --git a/apps/api/src/openapi/sam-cli.ts b/apps/api/src/openapi/sam-cli.ts index 40dec8bc7a..77de50706f 100644 --- a/apps/api/src/openapi/sam-cli.ts +++ b/apps/api/src/openapi/sam-cli.ts @@ -7,12 +7,13 @@ type SchemaObject = { type?: string | string[]; format?: string; description?: string; - enum?: string[]; + enum?: Array; items?: SchemaObject | ReferenceObject; properties?: Record; required?: string[]; additionalProperties?: boolean | SchemaObject | ReferenceObject; nullable?: boolean; + oneOf?: Array; }; type ReferenceObject = { @@ -176,6 +177,10 @@ const taskBaseFields: Record = { outputPrUrl: nullable(stringSchema()), outputSummary: nullable(stringSchema()), completionEvidence: nullable(objectSchema({}, [], true)), + placementExplanationJson: nullable( + stringSchema('Raw persisted placement JSON retained for backward compatibility.') + ), + placementExplanation: nullable(ref('PlacementExplanation')), errorMessage: nullable(stringSchema()), finalizedAt: nullable(dateTimeSchema()), createdAt: dateTimeSchema(), @@ -427,6 +432,188 @@ export const samCliOpenApiDocument: OpenApiDocument = { 'interval', ] ), + PlacementRequestSnapshot: objectSchema( + { + runtime: { type: 'string', enum: ['vm'] }, + vmSize: { type: 'string', enum: ['small', 'medium', 'large'] }, + vmLocation: stringSchema('Configured placement location; never a credential value.'), + maxWorkspacesPerNode: integerSchema(), + cpuThresholdPercent: integerSchema(), + memoryThresholdPercent: integerSchema(), + heartbeatStaleSeconds: integerSchema(), + }, + [ + 'runtime', + 'vmSize', + 'vmLocation', + 'maxWorkspacesPerNode', + 'cpuThresholdPercent', + 'memoryThresholdPercent', + 'heartbeatStaleSeconds', + ] + ), + PlacementNodeSnapshot: objectSchema( + { + runtime: { type: 'string', enum: ['vm', 'other'] }, + vmSize: stringSchema(), + vmLocation: stringSchema(), + healthStatus: { + type: 'string', + enum: ['healthy', 'stale', 'unhealthy', 'unknown'], + }, + agentVersionCompatible: booleanSchema(), + heartbeatAgeSeconds: nullable(numberSchema()), + activeWorkspaceCount: integerSchema(), + cpuLoadAvg1: nullable(numberSchema()), + memoryPercent: nullable(numberSchema()), + }, + [ + 'runtime', + 'vmSize', + 'vmLocation', + 'healthStatus', + 'agentVersionCompatible', + 'heartbeatAgeSeconds', + 'activeWorkspaceCount', + 'cpuLoadAvg1', + 'memoryPercent', + ] + ), + PlacementNodeEvaluation: objectSchema( + { + nodeId: stringSchema( + 'Selected node ID, or a stable candidate-N alias for an unselected node.' + ), + path: { + type: 'string', + enum: ['preferred', 'warm', 'capacity', 'trial', 'manual'], + }, + accepted: booleanSchema(), + rejectionReasons: arrayOf({ + type: 'string', + enum: [ + 'node-not-found', + 'not-running', + 'wrong-runtime', + 'unhealthy', + 'heartbeat-missing', + 'heartbeat-stale', + 'agent-not-ready', + 'agent-version-mismatch', + 'undersized', + 'workspace-limit', + 'cpu-threshold', + 'memory-threshold', + 'not-warm', + 'warm-claim-lost', + ], + }), + snapshot: ref('PlacementNodeSnapshot'), + }, + ['nodeId', 'path', 'accepted', 'rejectionReasons', 'snapshot'] + ), + PlacementProvisioningAttempt: objectSchema( + { + vmSize: { type: 'string', enum: ['small', 'medium', 'large'] }, + vmLocation: stringSchema(), + outcome: { + type: 'string', + enum: ['started', 'succeeded', 'capacity-rejected', 'failed'], + }, + failureReason: { + type: 'string', + enum: [ + 'capacity-unavailable', + 'node-limit', + 'quota-exceeded', + 'credentials-unavailable', + 'provider-failed', + 'provisioning-timeout', + 'readiness-timeout', + 'node-unavailable', + ], + }, + }, + ['vmSize', 'vmLocation', 'outcome'] + ), + PlacementExplanationV2: objectSchema( + { + schemaVersion: { type: 'integer', enum: [2] }, + outcome: { type: 'string', enum: ['reused', 'provisioned', 'failed'] }, + selectionPath: { + type: 'string', + enum: ['preferred', 'warm', 'capacity', 'trial', 'manual', 'provisioning'], + }, + selectedNodeId: nullable(stringSchema()), + summary: stringSchema(), + request: ref('PlacementRequestSnapshot'), + evaluatedNodes: arrayOf(ref('PlacementNodeEvaluation')), + provisioningAttempts: arrayOf(ref('PlacementProvisioningAttempt')), + decidedAt: dateTimeSchema(), + updatedAt: dateTimeSchema(), + }, + [ + 'schemaVersion', + 'outcome', + 'selectionPath', + 'selectedNodeId', + 'summary', + 'request', + 'evaluatedNodes', + 'provisioningAttempts', + 'decidedAt', + 'updatedAt', + ] + ), + LegacyPlacementExplanation: objectSchema( + { + selectedVmSize: { type: 'string', enum: ['small', 'medium', 'large'] }, + vmSizeSource: { + type: 'string', + enum: [ + 'task', + 'trigger', + 'skill', + 'agent-profile', + 'project', + 'user', + 'platform', + 'explicit', + ], + }, + reservation: objectSchema( + { + cpuMillis: integerSchema(), + memoryMb: integerSchema(), + diskMb: integerSchema(), + exclusiveNode: booleanSchema(), + maxCoTenants: integerSchema(), + source: { + type: 'string', + enum: ['task', 'trigger', 'skill', 'agent-profile', 'project', 'user', 'platform'], + }, + sourceId: stringSchema(), + version: integerSchema(), + }, + [ + 'cpuMillis', + 'memoryMb', + 'diskMb', + 'exclusiveNode', + 'maxCoTenants', + 'source', + 'sourceId', + 'version', + ] + ), + reason: stringSchema(), + decidedAt: dateTimeSchema(), + }, + ['selectedVmSize', 'vmSizeSource', 'reservation', 'reason', 'decidedAt'] + ), + PlacementExplanation: { + oneOf: [ref('PlacementExplanationV2'), ref('LegacyPlacementExplanation')], + }, Project: objectSchema({ ...projectBaseFields }, ['id', 'name']), // Mirrors packages/shared/src/types/project.ts's ProjectSummary — the // actual shape GET /api/projects returns (toProjectSummaryResponse() in @@ -790,6 +977,7 @@ export const samCliOpenApiDocument: OpenApiDocument = { url: stringSchema(), repository: stringSchema(), branch: stringSchema(), + placementExplanation: nullable(ref('PlacementExplanation')), createdAt: dateTimeSchema(), updatedAt: dateTimeSchema(), }, diff --git a/apps/api/src/routes/mcp/tool-definitions-workspace-tools.ts b/apps/api/src/routes/mcp/tool-definitions-workspace-tools.ts index a9cadea2a5..e8e6c67b18 100644 --- a/apps/api/src/routes/mcp/tool-definitions-workspace-tools.ts +++ b/apps/api/src/routes/mcp/tool-definitions-workspace-tools.ts @@ -7,7 +7,7 @@ export const WORKSPACE_TOOLS = [ { name: 'get_workspace_info', description: - 'Get consolidated workspace metadata: ID, node, project, branch, mode, VM size, URL, uptime. Use this for orientation at the start of a session.', + 'Get consolidated workspace metadata: ID, node, project, branch, mode, VM size, URL, uptime, and the non-sensitive node-placement explanation. Use this for orientation at the start of a session.', inputSchema: { type: 'object' as const, properties: {}, diff --git a/apps/api/src/routes/mcp/workspace-tools.ts b/apps/api/src/routes/mcp/workspace-tools.ts index 77ba15f119..46dc5a3f65 100644 --- a/apps/api/src/routes/mcp/workspace-tools.ts +++ b/apps/api/src/routes/mcp/workspace-tools.ts @@ -8,6 +8,10 @@ * Category A (direct D1/API) and Category C (Worker-side DNS) handlers are in * workspace-tools-direct.ts. */ +import { + isPlacementExplanationV2, + parsePlacementExplanationJson, +} from '@simple-agent-manager/shared'; import { and, eq } from 'drizzle-orm'; import { drizzle } from 'drizzle-orm/d1'; @@ -54,6 +58,7 @@ export interface WorkspaceForVmAgent { status: string; nodeId: string; projectId: string; + placementExplanationJson: string | null; } export async function lookupWorkspaceForVmAgent( @@ -70,6 +75,7 @@ export async function lookupWorkspaceForVmAgent( status: schema.workspaces.status, nodeId: schema.workspaces.nodeId, projectId: schema.workspaces.projectId, + placementExplanationJson: schema.workspaces.placementExplanationJson, }) .from(schema.workspaces) .where(and(eq(schema.workspaces.id, workspaceId), eq(schema.workspaces.projectId, projectId))) @@ -90,6 +96,7 @@ export async function lookupWorkspaceForVmAgent( status: workspace.status, nodeId: workspace.nodeId, projectId: workspace.projectId ?? projectId, + placementExplanationJson: workspace.placementExplanationJson, }; } @@ -154,9 +161,11 @@ export async function proxyToVmAgent( toolPath: VmAgentToolPath, method: 'GET' | 'POST' = 'GET', body?: unknown, - timeoutOverrideMs?: number + timeoutOverrideMs?: number, + workspaceOverride?: WorkspaceForVmAgent ): Promise { - const workspace = await lookupWorkspaceForVmAgent(env, workspaceId, projectId); + const workspace = + workspaceOverride ?? (await lookupWorkspaceForVmAgent(env, workspaceId, projectId)); // Generate workspace token for VM agent auth const { token } = await signTerminalToken(userId, workspaceId, env); @@ -250,6 +259,22 @@ export function requireWorkspace( return null; } +export function enrichWorkspaceInfoWithPlacement( + result: unknown, + placementExplanationJson: string | null +): Record { + const explanation = parsePlacementExplanationJson(placementExplanationJson); + const placement = explanation + ? { + summary: isPlacementExplanationV2(explanation) ? explanation.summary : explanation.reason, + detail: explanation, + } + : null; + return typeof result === 'object' && result !== null && !Array.isArray(result) + ? { ...result, placement } + : { workspaceInfo: result, placement }; +} + // ─── Category B: Proxied to VM agent ──────────────────────────────────────── export async function handleGetWorkspaceInfo( @@ -260,15 +285,25 @@ export async function handleGetWorkspaceInfo( const err = requireWorkspace(requestId, tokenData); if (err) return err; try { + const workspace = await lookupWorkspaceForVmAgent( + env, + tokenData.workspaceId, + tokenData.projectId + ); const result = await proxyToVmAgent( env, tokenData.workspaceId, tokenData.userId, tokenData.projectId, - 'workspace-info' + 'workspace-info', + 'GET', + undefined, + undefined, + workspace ); + const enriched = enrichWorkspaceInfoWithPlacement(result, workspace.placementExplanationJson); return jsonRpcSuccess(requestId, { - content: [{ type: 'text', text: JSON.stringify(result, null, 2) }], + content: [{ type: 'text', text: JSON.stringify(enriched, null, 2) }], }); } catch (e) { return jsonRpcError( diff --git a/apps/api/src/routes/workspaces/crud.ts b/apps/api/src/routes/workspaces/crud.ts index 07ef28c13c..0aa247ca2b 100644 --- a/apps/api/src/routes/workspaces/crud.ts +++ b/apps/api/src/routes/workspaces/crud.ts @@ -1,4 +1,4 @@ -import { DEFAULT_VM_LOCATION,DEFAULT_VM_SIZE } from '@simple-agent-manager/shared'; +import { DEFAULT_VM_LOCATION, DEFAULT_VM_SIZE } from '@simple-agent-manager/shared'; import { and, count, desc, eq, inArray, ne } from 'drizzle-orm'; import { drizzle } from 'drizzle-orm/d1'; import { Hono } from 'hono'; @@ -8,25 +8,29 @@ import type { Env } from '../../env'; import { log } from '../../lib/logger'; import { toWorkspaceResponse } from '../../lib/mappers'; import { ulid } from '../../lib/ulid'; -import { getAuth, getUserId, requireApproved,requireAuth } from '../../middleware/auth'; +import { getAuth, getUserId, requireApproved, requireAuth } from '../../middleware/auth'; import { errors } from '../../middleware/error'; import { requireProjectCapability } from '../../middleware/project-auth'; -import { CreateWorkspaceSchema,jsonValidator, UpdateWorkspacePortsPublicSchema, UpdateWorkspaceSchema } from '../../schemas'; +import { + CreateWorkspaceSchema, + jsonValidator, + UpdateWorkspacePortsPublicSchema, + UpdateWorkspaceSchema, +} from '../../schemas'; import { startComputeTracking } from '../../services/compute-usage'; import { signPortAccessToken } from '../../services/jwt'; import { getRuntimeLimits } from '../../services/limits'; -import { - getWorkspacePortsOnNode, - waitForNodeAgentReady, -} from '../../services/node-agent'; -import { isNodeAgentVersionCompatible } from '../../services/node-agent-compatibility'; -import { createNodeRecord, provisionNode } from '../../services/nodes'; +import { getWorkspacePortsOnNode } from '../../services/node-agent'; import * as projectDataService from '../../services/project-data'; import { recordNodeRoutingMetric } from '../../services/telemetry'; import { cleanupWorkspaceForDeletion } from '../../services/workspace-cleanup'; import { resolveUniqueWorkspaceDisplayName } from '../../services/workspace-names'; import { requireRepositoryUserAccess } from '../projects/_helpers'; -import { getOwnedNode, getOwnedWorkspace, scheduleWorkspaceCreateOnNode } from './_helpers'; +import { getOwnedWorkspace } from './_helpers'; +import { + completeManualWorkspacePlacement, + resolveManualWorkspacePlacement, +} from './manual-placement'; const crudRoutes = new Hono<{ Bindings: Env }>(); @@ -125,359 +129,355 @@ crudRoutes.get('/:id/ports', requireAuth(), requireApproved(), async (c) => { return c.json(result); }); -crudRoutes.patch('/:id/ports-public', requireAuth(), requireApproved(), jsonValidator(UpdateWorkspacePortsPublicSchema), async (c) => { - const userId = getUserId(c); - const workspaceId = c.req.param('id'); - const body = c.req.valid('json'); - const db = drizzle(c.env.DATABASE, { schema }); - - await getOwnedWorkspace(db, workspaceId, userId); +crudRoutes.patch( + '/:id/ports-public', + requireAuth(), + requireApproved(), + jsonValidator(UpdateWorkspacePortsPublicSchema), + async (c) => { + const userId = getUserId(c); + const workspaceId = c.req.param('id'); + const body = c.req.valid('json'); + const db = drizzle(c.env.DATABASE, { schema }); - const [updated] = await db - .update(schema.workspaces) - .set({ - portsPublicEnabled: body.enabled, - updatedAt: new Date().toISOString(), - }) - .where(and(eq(schema.workspaces.id, workspaceId), eq(schema.workspaces.userId, userId))) - .returning(); - - if (!updated) { - throw errors.notFound('Workspace not found'); - } + await getOwnedWorkspace(db, workspaceId, userId); - return c.json(toWorkspaceResponse(updated, c.env.BASE_DOMAIN)); -}); + const [updated] = await db + .update(schema.workspaces) + .set({ + portsPublicEnabled: body.enabled, + updatedAt: new Date().toISOString(), + }) + .where(and(eq(schema.workspaces.id, workspaceId), eq(schema.workspaces.userId, userId))) + .returning(); -crudRoutes.patch('/:id', requireAuth(), requireApproved(), jsonValidator(UpdateWorkspaceSchema), async (c) => { - const userId = getUserId(c); - const workspaceId = c.req.param('id'); - const db = drizzle(c.env.DATABASE, { schema }); - const body = c.req.valid('json'); + if (!updated) { + throw errors.notFound('Workspace not found'); + } - if (!body.displayName?.trim()) { - throw errors.badRequest('displayName is required'); + return c.json(toWorkspaceResponse(updated, c.env.BASE_DOMAIN)); } +); + +crudRoutes.patch( + '/:id', + requireAuth(), + requireApproved(), + jsonValidator(UpdateWorkspaceSchema), + async (c) => { + const userId = getUserId(c); + const workspaceId = c.req.param('id'); + const db = drizzle(c.env.DATABASE, { schema }); + const body = c.req.valid('json'); + + if (!body.displayName?.trim()) { + throw errors.badRequest('displayName is required'); + } - const workspace = await getOwnedWorkspace(db, workspaceId, userId); - const nodeScopeId = workspace.nodeId ?? workspace.id; - const uniqueName = await resolveUniqueWorkspaceDisplayName( - db, - nodeScopeId, - body.displayName, - workspace.id - ); - - await db - .update(schema.workspaces) - .set({ - nodeId: nodeScopeId, - displayName: uniqueName.displayName, - normalizedDisplayName: uniqueName.normalizedDisplayName, - updatedAt: new Date().toISOString(), - }) - .where(eq(schema.workspaces.id, workspace.id)); - - const updated = await getOwnedWorkspace(db, workspace.id, userId); - return c.json(toWorkspaceResponse(updated, c.env.BASE_DOMAIN)); -}); + const workspace = await getOwnedWorkspace(db, workspaceId, userId); + const nodeScopeId = workspace.nodeId ?? workspace.id; + const uniqueName = await resolveUniqueWorkspaceDisplayName( + db, + nodeScopeId, + body.displayName, + workspace.id + ); -crudRoutes.post('/', requireAuth(), requireApproved(), jsonValidator(CreateWorkspaceSchema), async (c) => { - const auth = getAuth(c); - const userId = auth.user.id; - const db = drizzle(c.env.DATABASE, { schema }); - const body = c.req.valid('json'); - const now = new Date().toISOString(); - const limits = getRuntimeLimits(c.env); - const projectId = body.projectId?.trim(); - const workspaceName = body.name?.trim(); - - if (!workspaceName) { - throw errors.badRequest('name is required'); - } + await db + .update(schema.workspaces) + .set({ + nodeId: nodeScopeId, + displayName: uniqueName.displayName, + normalizedDisplayName: uniqueName.normalizedDisplayName, + updatedAt: new Date().toISOString(), + }) + .where(eq(schema.workspaces.id, workspace.id)); - if (!projectId) { - throw errors.badRequest('projectId is required'); + const updated = await getOwnedWorkspace(db, workspace.id, userId); + return c.json(toWorkspaceResponse(updated, c.env.BASE_DOMAIN)); } +); + +crudRoutes.post( + '/', + requireAuth(), + requireApproved(), + jsonValidator(CreateWorkspaceSchema), + async (c) => { + const auth = getAuth(c); + const userId = auth.user.id; + const db = drizzle(c.env.DATABASE, { schema }); + const body = c.req.valid('json'); + const now = new Date().toISOString(); + const limits = getRuntimeLimits(c.env); + const projectId = body.projectId?.trim(); + const workspaceName = body.name?.trim(); + + if (!workspaceName) { + throw errors.badRequest('name is required'); + } - const linkedProject = await requireProjectCapability(db, projectId, userId, 'workspace:write'); - const resolvedInstallationId = linkedProject.installationId; - const resolvedRepository = linkedProject.repository; - const resolvedBranch = body.branch?.trim() || linkedProject.defaultBranch; + if (!projectId) { + throw errors.badRequest('projectId is required'); + } - if (!resolvedRepository || !resolvedInstallationId) { - throw errors.badRequest('repository and installationId are required'); - } - const normalizedRepository = resolvedRepository.toLowerCase(); - - // Fail-fast user∩app GitHub repo-access gate. Re-verify the user still has - // access to the bound repository through the app installation BEFORE - // provisioning any node or creating any workspace. Throws 403 if access was - // revoked or the repository id drifted. Also covers installation ownership - // (via requireOwnedInstallation), so no separate ownership query is needed. - await requireRepositoryUserAccess(c, db, linkedProject, userId); - - const vmSize = body.vmSize ?? DEFAULT_VM_SIZE; - const vmLocation = body.vmLocation ?? DEFAULT_VM_LOCATION; - - // Validate branch name — reject shell metacharacters to prevent command injection. - // Git branch names allow: alphanumeric, hyphens, underscores, slashes, dots. - // See INJ-VULN-02 in Shannon security assessment. - const SAFE_BRANCH_PATTERN = /^[a-zA-Z0-9._\-/]+$/; - if (!SAFE_BRANCH_PATTERN.test(resolvedBranch)) { - throw errors.badRequest( - 'branch contains invalid characters. Only alphanumeric, hyphens, underscores, slashes, and dots are allowed.' - ); - } - const branch = resolvedBranch; - - let nodeId = body.nodeId; - let mustProvisionNode = false; - let credentialAttributionSource: import('@simple-agent-manager/shared').CredentialSource = 'user'; - // Use COUNT instead of fetching all node IDs (P1 fix). - // Exclude deleted/stopped nodes — only active ones count toward the limit. - const [userNodeCount] = await db - .select({ count: count() }) - .from(schema.nodes) - .where(and( - eq(schema.nodes.userId, userId), - inArray(schema.nodes.status, ['running', 'creating', 'recovery']), - eq(schema.nodes.nodeRole, 'workspace') - )); - const userNodeCountVal = userNodeCount?.count ?? 0; + const linkedProject = await requireProjectCapability(db, projectId, userId, 'workspace:write'); + const resolvedInstallationId = linkedProject.installationId; + const resolvedRepository = linkedProject.repository; + const resolvedBranch = body.branch?.trim() || linkedProject.defaultBranch; - if (nodeId) { - const node = await getOwnedNode(db, nodeId, userId); - if (node.status === 'stopped' || node.healthStatus === 'unhealthy') { - throw errors.badRequest('Selected node is not ready for workspace creation'); - } - if (!isNodeAgentVersionCompatible(node.agentVersion, c.env.VM_AGENT_REQUIRED_VERSION)) { - throw errors.badRequest('Selected node is running an incompatible VM agent build'); - } - } else { - if (userNodeCountVal >= limits.maxNodesPerUser) { - throw errors.badRequest(`Maximum ${limits.maxNodesPerUser} nodes allowed`); + if (!resolvedRepository || !resolvedInstallationId) { + throw errors.badRequest('repository and installationId are required'); } - - const { resolveCredentialSource } = await import('../../services/provider-credentials'); - const credResult = await resolveCredentialSource(db, userId, body.provider, linkedProject.id); - if (!credResult) { - throw errors.forbidden('Cloud provider credentials required. Connect your account in Settings.'); + const normalizedRepository = resolvedRepository.toLowerCase(); + + // Fail-fast user∩app GitHub repo-access gate. Re-verify the user still has + // access to the bound repository through the app installation BEFORE + // provisioning any node or creating any workspace. Throws 403 if access was + // revoked or the repository id drifted. Also covers installation ownership + // (via requireOwnedInstallation), so no separate ownership query is needed. + await requireRepositoryUserAccess(c, db, linkedProject, userId); + + const vmSize = body.vmSize ?? DEFAULT_VM_SIZE; + const vmLocation = body.vmLocation ?? DEFAULT_VM_LOCATION; + + // Validate branch name — reject shell metacharacters to prevent command injection. + // Git branch names allow: alphanumeric, hyphens, underscores, slashes, dots. + // See INJ-VULN-02 in Shannon security assessment. + const SAFE_BRANCH_PATTERN = /^[a-zA-Z0-9._\-/]+$/; + if (!SAFE_BRANCH_PATTERN.test(resolvedBranch)) { + throw errors.badRequest( + 'branch contains invalid characters. Only alphanumeric, hyphens, underscores, slashes, and dots are allowed.' + ); } - credentialAttributionSource = credResult.credentialSource; - const effectiveProvider = body.provider ?? credResult.providerName; + const branch = resolvedBranch; - const createdNode = await createNodeRecord(c.env, { + // Use COUNT instead of fetching all node IDs (P1 fix). + // Exclude deleted/stopped nodes — only active ones count toward the limit. + const [userNodeCount] = await db + .select({ count: count() }) + .from(schema.nodes) + .where( + and( + eq(schema.nodes.userId, userId), + inArray(schema.nodes.status, ['running', 'creating', 'recovery']), + eq(schema.nodes.nodeRole, 'workspace') + ) + ); + const userNodeCountVal = userNodeCount?.count ?? 0; + const placement = await resolveManualWorkspacePlacement({ + db, + env: c.env, userId, - credentialAttributionUserId: userId, - credentialAttributionProjectId: credentialAttributionSource === 'project' ? linkedProject.id : null, - credentialAttributionSource, - name: `${workspaceName} Node`, + projectId: linkedProject.id, + workspaceName, vmSize, vmLocation, - cloudProvider: effectiveProvider, + preferredNodeId: body.nodeId, + provider: body.provider, + activeNodeCount: userNodeCountVal, + maxNodesPerUser: limits.maxNodesPerUser, heartbeatStaleAfterSeconds: limits.nodeHeartbeatStaleSeconds, + now, }); + const targetNodeId = placement.nodeId; + const mustProvisionNode = placement.mustProvisionNode; + const credentialAttributionSource = placement.credentialAttributionSource; + const placementExplanation = placement.explanation; + + // Count active workspaces on this node (for telemetry — no hard count limit, + // resource thresholds handle capacity in the task runner path). + const [nodeWorkspaceCount] = await db + .select({ count: count() }) + .from(schema.workspaces) + .where( + and( + eq(schema.workspaces.userId, userId), + eq(schema.workspaces.nodeId, targetNodeId), + inArray(schema.workspaces.status, ['running', 'creating', 'recovery']) + ) + ); + const nodeWorkspaceCountVal = nodeWorkspaceCount?.count ?? 0; - nodeId = createdNode.id; - mustProvisionNode = true; - } - const targetNodeId = nodeId; - if (!targetNodeId) { - throw errors.internal('Failed to determine target node'); - } + const uniqueName = await resolveUniqueWorkspaceDisplayName(db, targetNodeId, workspaceName); - // Count active workspaces on this node (for telemetry — no hard count limit, - // resource thresholds handle capacity in the task runner path). - const [nodeWorkspaceCount] = await db - .select({ count: count() }) - .from(schema.workspaces) - .where(and( - eq(schema.workspaces.userId, userId), - eq(schema.workspaces.nodeId, targetNodeId), - inArray(schema.workspaces.status, ['running', 'creating', 'recovery']) - )); - const nodeWorkspaceCountVal = nodeWorkspaceCount?.count ?? 0; + const workspaceId = ulid(); - const uniqueName = await resolveUniqueWorkspaceDisplayName(db, targetNodeId, workspaceName); + await db.insert(schema.workspaces).values({ + id: workspaceId, + nodeId: targetNodeId, + projectId: linkedProject.id, + userId, + installationId: resolvedInstallationId, + name: workspaceName, + displayName: uniqueName.displayName, + normalizedDisplayName: uniqueName.normalizedDisplayName, + repository: resolvedRepository, + branch, + status: 'creating', + vmSize, + vmLocation, + placementExplanationJson: JSON.stringify(placementExplanation), + createdAt: now, + updatedAt: now, + }); - const workspaceId = ulid(); + const chatTaskId = ulid(); + await db.insert(schema.tasks).values({ + id: chatTaskId, + projectId: linkedProject.id, + userId, + workspaceId, + title: workspaceName, + status: 'queued', + executionStep: 'workspace_creation', + taskMode: 'conversation', + triggeredBy: 'user', + credentialAttributionUserId: userId, + credentialAttributionSource, + placementExplanationJson: JSON.stringify(placementExplanation), + createdBy: userId, + createdAt: now, + updatedAt: now, + }); - await db.insert(schema.workspaces).values({ - id: workspaceId, - nodeId: targetNodeId, - projectId: linkedProject.id, - userId, - installationId: resolvedInstallationId, - name: workspaceName, - displayName: uniqueName.displayName, - normalizedDisplayName: uniqueName.normalizedDisplayName, - repository: resolvedRepository, - branch, - status: 'creating', - vmSize, - vmLocation, - createdAt: now, - updatedAt: now, - }); + // Create chat session in ProjectData DO (workspace always linked to project) + try { + const chatSessionId = await projectDataService.createSession( + c.env, + linkedProject.id, + workspaceId, + workspaceName, + chatTaskId, + userId + ); + await db + .update(schema.workspaces) + .set({ chatSessionId, updatedAt: now }) + .where(eq(schema.workspaces.id, workspaceId)); + await db + .update(schema.tasks) + .set({ + chatSessionId, + status: 'in_progress', + executionStep: 'workspace_ready', + updatedAt: now, + }) + .where(eq(schema.tasks.id, chatTaskId)); + } catch (err) { + // Best-effort: session creation failure should not block workspace creation + log.error('workspace.chat_session_create_failed', { + workspaceId, + error: err instanceof Error ? err.message : String(err), + }); + } - const chatTaskId = ulid(); - await db.insert(schema.tasks).values({ - id: chatTaskId, projectId: linkedProject.id, userId, workspaceId, - title: workspaceName, status: 'queued', executionStep: 'workspace_creation', - taskMode: 'conversation', triggeredBy: 'user', - credentialAttributionUserId: userId, credentialAttributionSource, - createdBy: userId, createdAt: now, updatedAt: now, - }); + const nodeCountForUser = userNodeCountVal + (mustProvisionNode ? 1 : 0); + const reusedExistingNode = !mustProvisionNode; + const workspaceCountOnNodeBefore = nodeWorkspaceCountVal; - // Create chat session in ProjectData DO (workspace always linked to project) - try { - const chatSessionId = await projectDataService.createSession( - c.env, - linkedProject.id, - workspaceId, - workspaceName, - chatTaskId, - userId + recordNodeRoutingMetric( + { + metric: 'sc_002_workspace_creation_flow', + nodeId: targetNodeId, + workspaceId, + userId, + repository: normalizedRepository, + reusedExistingNode, + workspaceCountOnNodeBefore, + nodeCountForUser, + }, + c.env ); - await db - .update(schema.workspaces) - .set({ chatSessionId, updatedAt: now }) - .where(eq(schema.workspaces.id, workspaceId)); - await db.update(schema.tasks).set({ - chatSessionId, status: "in_progress", executionStep: "workspace_ready", updatedAt: now, - }).where(eq(schema.tasks.id, chatTaskId)); - } catch (err) { - // Best-effort: session creation failure should not block workspace creation - log.error('workspace.chat_session_create_failed', { workspaceId, error: err instanceof Error ? err.message : String(err) }); - } - - const nodeCountForUser = userNodeCountVal + (mustProvisionNode ? 1 : 0); - const reusedExistingNode = !mustProvisionNode; - const workspaceCountOnNodeBefore = nodeWorkspaceCountVal; - recordNodeRoutingMetric( - { - metric: 'sc_002_workspace_creation_flow', - nodeId: targetNodeId, - workspaceId, - userId, - repository: normalizedRepository, - reusedExistingNode, - workspaceCountOnNodeBefore, - nodeCountForUser, - }, - c.env - ); - - recordNodeRoutingMetric( - { - metric: 'sc_006_node_efficiency', - nodeId: targetNodeId, + log.info('workspace.manual_placement_decided', { workspaceId, - userId, - repository: normalizedRepository, - reusedExistingNode, - nodeCountForUser, - }, - c.env - ); - - // Start compute usage metering (best-effort — failure should not block workspace creation) - try { - const [nodeRow] = await db - .select({ - cloudProvider: schema.nodes.cloudProvider, - credentialSource: schema.nodes.credentialSource, - }) - .from(schema.nodes) - .where(eq(schema.nodes.id, targetNodeId)) - .limit(1); - await startComputeTracking(db, { - userId, - workspaceId, - nodeId: targetNodeId, - vmSize, - cloudProvider: nodeRow?.cloudProvider, - credentialSource: (nodeRow?.credentialSource as import('@simple-agent-manager/shared').CredentialSource) ?? 'user', + taskId: chatTaskId, + placement: placementExplanation, }); - } catch (err) { - log.error('workspace.compute_tracking_start_failed', { - workspaceId, - error: err instanceof Error ? err.message : String(err), - }); - } - c.executionCtx.waitUntil( - (async () => { - const innerDb = drizzle(c.env.DATABASE, { schema }); - if (mustProvisionNode) { - await provisionNode(targetNodeId, c.env); - - const nodeRows = await innerDb - .select({ - status: schema.nodes.status, - errorMessage: schema.nodes.errorMessage, - }) - .from(schema.nodes) - .where(eq(schema.nodes.id, targetNodeId)) - .limit(1); - - const provisionedNode = nodeRows[0]; - if (!provisionedNode || provisionedNode.status !== 'running') { - await innerDb - .update(schema.workspaces) - .set({ - status: 'error', - errorMessage: provisionedNode?.errorMessage || 'Node provisioning failed', - updatedAt: new Date().toISOString(), - }) - .where(eq(schema.workspaces.id, workspaceId)); - return; - } - - try { - await waitForNodeAgentReady(targetNodeId, c.env); - } catch (err) { - await innerDb - .update(schema.workspaces) - .set({ - status: 'error', - errorMessage: - err instanceof Error ? err.message : 'Node agent not reachable after provisioning', - updatedAt: new Date().toISOString(), - }) - .where(eq(schema.workspaces.id, workspaceId)); - return; - } - } - - await scheduleWorkspaceCreateOnNode( - c.env, + recordNodeRoutingMetric( + { + metric: 'sc_006_node_efficiency', + nodeId: targetNodeId, workspaceId, - targetNodeId, userId, - resolvedRepository, - branch, - linkedProject, - auth.user.name, - auth.user.email - ); - })() - ); + repository: normalizedRepository, + reusedExistingNode, + nodeCountForUser, + }, + c.env + ); - const created = await getOwnedWorkspace(db, workspaceId, userId); + // Start compute usage metering (best-effort — failure should not block workspace creation) + try { + const [nodeRow] = await db + .select({ + cloudProvider: schema.nodes.cloudProvider, + credentialSource: schema.nodes.credentialSource, + }) + .from(schema.nodes) + .where(eq(schema.nodes.id, targetNodeId)) + .limit(1); + await startComputeTracking(db, { + userId, + workspaceId, + nodeId: targetNodeId, + vmSize, + cloudProvider: nodeRow?.cloudProvider, + credentialSource: + (nodeRow?.credentialSource as import('@simple-agent-manager/shared').CredentialSource) ?? + 'user', + }); + } catch (err) { + log.error('workspace.compute_tracking_start_failed', { + workspaceId, + error: err instanceof Error ? err.message : String(err), + }); + } - // Record activity event for workspace creation - c.executionCtx.waitUntil( - projectDataService.recordActivityEvent( - c.env, linkedProject.id, 'workspace.created', 'user', userId, - workspaceId, null, null, { name: created.name, repository: resolvedRepository } - ).catch((e) => { log.warn('workspace.activity_created_failed', { workspaceId, error: String(e) }); }) - ); + c.executionCtx.waitUntil( + completeManualWorkspacePlacement({ + env: c.env, + explanation: placementExplanation, + mustProvisionNode, + nodeId: targetNodeId, + workspaceId, + taskId: chatTaskId, + userId, + repository: resolvedRepository, + branch, + project: linkedProject, + vmSize, + vmLocation, + gitUserName: auth.user.name, + gitUserEmail: auth.user.email, + }) + ); - return c.json(toWorkspaceResponse(created, c.env.BASE_DOMAIN), 201); -}); + const created = await getOwnedWorkspace(db, workspaceId, userId); + + c.executionCtx.waitUntil( + projectDataService + .recordActivityEvent( + c.env, + linkedProject.id, + 'workspace.created', + 'user', + userId, + workspaceId, + null, + null, + { name: created.name, repository: resolvedRepository } + ) + .catch((e) => { + log.warn('workspace.activity_created_failed', { workspaceId, error: String(e) }); + }) + ); + + return c.json(toWorkspaceResponse(created, c.env.BASE_DOMAIN), 201); + } +); crudRoutes.delete('/:id', requireAuth(), requireApproved(), async (c) => { const userId = getUserId(c); diff --git a/apps/api/src/routes/workspaces/manual-placement.ts b/apps/api/src/routes/workspaces/manual-placement.ts new file mode 100644 index 0000000000..ebb00b5fb1 --- /dev/null +++ b/apps/api/src/routes/workspaces/manual-placement.ts @@ -0,0 +1,293 @@ +import type { + CredentialProvider, + CredentialSource, + PlacementExplanation, + VMSize, +} from '@simple-agent-manager/shared'; +import { eq } from 'drizzle-orm'; +import { type drizzle } from 'drizzle-orm/d1'; + +import * as schema from '../../db/schema'; +import type { Env } from '../../env'; +import { log } from '../../lib/logger'; +import { ulid } from '../../lib/ulid'; +import { errors } from '../../middleware/error'; +import { waitForNodeAgentReady } from '../../services/node-agent'; +import { selectNodeWithExplanation } from '../../services/node-selector'; +import { createNodeRecord, provisionNode } from '../../services/nodes'; +import { + appendProvisioningAttempt, + createPlacementExplanation, + failPlacement, + requireProvisioning, + resolvePlacementRequest, +} from '../../services/placement-explanation'; +import { resolveCredentialSource } from '../../services/provider-credentials'; +import type { WorkspaceGitSourceProject } from '../../services/workspace-git-source'; +import { scheduleWorkspaceCreateOnNode } from './_helpers'; + +type WorkspaceDb = ReturnType>; + +export interface ManualPlacementResult { + nodeId: string; + mustProvisionNode: boolean; + credentialAttributionSource: CredentialSource; + explanation: PlacementExplanation; +} + +async function persistRejectedManualPlacement( + db: WorkspaceDb, + input: Pick< + Parameters[0], + 'projectId' | 'userId' | 'workspaceName' | 'now' + >, + explanation: PlacementExplanation, + errorMessage: string +): Promise { + const taskId = ulid(); + await db.insert(schema.tasks).values({ + id: taskId, + projectId: input.projectId, + userId: input.userId, + title: input.workspaceName, + status: 'failed', + taskMode: 'conversation', + triggeredBy: 'user', + credentialAttributionUserId: input.userId, + credentialAttributionSource: 'user', + placementExplanationJson: JSON.stringify(explanation), + errorMessage, + createdBy: input.userId, + createdAt: input.now, + updatedAt: input.now, + }); + log.info('workspace.manual_placement_rejected', { taskId, placement: explanation }); +} + +export async function resolveManualWorkspacePlacement(input: { + db: WorkspaceDb; + env: Env; + userId: string; + projectId: string; + workspaceName: string; + vmSize: VMSize; + vmLocation: string; + preferredNodeId?: string; + provider?: CredentialProvider; + activeNodeCount: number; + maxNodesPerUser: number; + heartbeatStaleAfterSeconds: number; + now: string; +}): Promise { + if (input.preferredNodeId) { + const placement = await selectNodeWithExplanation(input.db, input.userId, input.env, { + vmSize: input.vmSize, + vmLocation: input.vmLocation, + preferredNodeId: input.preferredNodeId, + preferredOnly: true, + selectionPath: 'manual', + }); + if (placement.node) { + return { + nodeId: placement.node.id, + mustProvisionNode: false, + credentialAttributionSource: 'user', + explanation: placement.explanation, + }; + } + + await persistRejectedManualPlacement( + input.db, + input, + placement.explanation, + 'Selected node is not eligible for workspace placement' + ); + throw errors.badRequest('Selected node is not eligible for workspace placement'); + } + + const request = resolvePlacementRequest(input.env, input.vmSize, input.vmLocation); + const provisioning = requireProvisioning(createPlacementExplanation(request, 'manual')); + if (input.activeNodeCount >= input.maxNodesPerUser) { + await persistRejectedManualPlacement( + input.db, + input, + failPlacement(provisioning, 'node-limit'), + `Maximum ${input.maxNodesPerUser} nodes allowed` + ); + throw errors.badRequest(`Maximum ${input.maxNodesPerUser} nodes allowed`); + } + const credential = await resolveCredentialSource( + input.db, + input.userId, + input.provider, + input.projectId + ); + if (!credential) { + await persistRejectedManualPlacement( + input.db, + input, + failPlacement(provisioning, 'credentials-unavailable'), + 'Cloud provider credentials required' + ); + throw errors.forbidden( + 'Cloud provider credentials required. Connect your account in Settings.' + ); + } + let createdNode: Awaited>; + try { + createdNode = await createNodeRecord(input.env, { + userId: input.userId, + credentialAttributionUserId: input.userId, + credentialAttributionProjectId: + credential.credentialSource === 'project' ? input.projectId : null, + credentialAttributionSource: credential.credentialSource, + name: `${input.workspaceName} Node`, + vmSize: input.vmSize, + vmLocation: input.vmLocation, + cloudProvider: input.provider ?? credential.providerName, + heartbeatStaleAfterSeconds: input.heartbeatStaleAfterSeconds, + }); + } catch (error) { + await persistRejectedManualPlacement( + input.db, + input, + failPlacement(provisioning, 'provider-failed'), + 'Node provisioning could not be started' + ); + throw error; + } + return { + nodeId: createdNode.id, + mustProvisionNode: true, + credentialAttributionSource: credential.credentialSource, + explanation: appendProvisioningAttempt( + provisioning, + { vmSize: input.vmSize, vmLocation: input.vmLocation, outcome: 'started' }, + createdNode.id + ), + }; +} + +export async function completeManualWorkspacePlacement(input: { + env: Env; + explanation: PlacementExplanation; + mustProvisionNode: boolean; + nodeId: string; + workspaceId: string; + taskId: string; + userId: string; + repository: string; + branch: string; + project: WorkspaceGitSourceProject; + vmSize: VMSize; + vmLocation: string; + gitUserName?: string | null; + gitUserEmail?: string | null; +}): Promise { + const { drizzle: createDrizzle } = await import('drizzle-orm/d1'); + const db = createDrizzle(input.env.DATABASE, { schema }); + let explanation = input.explanation; + const persist = async (): Promise => { + const updatedAt = new Date().toISOString(); + const json = JSON.stringify(explanation); + await db + .update(schema.workspaces) + .set({ placementExplanationJson: json, updatedAt }) + .where(eq(schema.workspaces.id, input.workspaceId)); + await db + .update(schema.tasks) + .set({ placementExplanationJson: json, updatedAt }) + .where(eq(schema.tasks.id, input.taskId)); + }; + const persistFailure = async (errorMessage: string): Promise => { + await persist(); + const updatedAt = new Date().toISOString(); + await db + .update(schema.workspaces) + .set({ status: 'error', errorMessage, updatedAt }) + .where(eq(schema.workspaces.id, input.workspaceId)); + await db + .update(schema.tasks) + .set({ status: 'failed', errorMessage, updatedAt }) + .where(eq(schema.tasks.id, input.taskId)); + }; + + if (input.mustProvisionNode) { + try { + await provisionNode(input.nodeId, input.env); + } catch { + explanation = appendProvisioningAttempt( + explanation, + { + vmSize: input.vmSize, + vmLocation: input.vmLocation, + outcome: 'failed', + failureReason: 'provider-failed', + }, + input.nodeId + ); + await persistFailure('Node provisioning failed'); + return; + } + + const [node] = await db + .select({ status: schema.nodes.status, errorMessage: schema.nodes.errorMessage }) + .from(schema.nodes) + .where(eq(schema.nodes.id, input.nodeId)) + .limit(1); + if (!node || node.status !== 'running') { + explanation = appendProvisioningAttempt( + explanation, + { + vmSize: input.vmSize, + vmLocation: input.vmLocation, + outcome: 'failed', + failureReason: 'node-unavailable', + }, + input.nodeId + ); + await persistFailure(node?.errorMessage || 'Node provisioning failed'); + return; + } + + explanation = appendProvisioningAttempt( + explanation, + { vmSize: input.vmSize, vmLocation: input.vmLocation, outcome: 'succeeded' }, + input.nodeId + ); + await persist(); + try { + await waitForNodeAgentReady(input.nodeId, input.env); + } catch { + explanation = appendProvisioningAttempt( + explanation, + { + vmSize: input.vmSize, + vmLocation: input.vmLocation, + outcome: 'failed', + failureReason: 'readiness-timeout', + }, + input.nodeId + ); + await persistFailure('Node agent not reachable after provisioning'); + return; + } + } else { + // The create route writes the initial decision to both records. Repeat that + // write at the async boundary so every successful reusable path has the + // same finalized persistence guarantee as provisioning paths. + await persist(); + } + + await scheduleWorkspaceCreateOnNode( + input.env, + input.workspaceId, + input.nodeId, + input.userId, + input.repository, + input.branch, + input.project, + input.gitUserName, + input.gitUserEmail + ); +} diff --git a/apps/api/src/schemas/placement.ts b/apps/api/src/schemas/placement.ts new file mode 100644 index 0000000000..f7c5ee4eda --- /dev/null +++ b/apps/api/src/schemas/placement.ts @@ -0,0 +1,23 @@ +import { + SAM_NODE_ID_LENGTH, + SAM_NODE_ID_REGEX, + VM_LOCATION_ID_MAX_LENGTH, + VM_LOCATION_ID_REGEX, +} from '@simple-agent-manager/shared'; +import * as v from 'valibot'; + +export const PlacementNodeIdSchema = v.pipe( + v.string(), + v.length(SAM_NODE_ID_LENGTH, 'nodeId must be a valid SAM node ID'), + v.regex(SAM_NODE_ID_REGEX, 'nodeId must be a valid SAM node ID') +); + +export const VMLocationSchema = v.pipe( + v.string(), + v.minLength(1, 'vmLocation is required'), + v.maxLength( + VM_LOCATION_ID_MAX_LENGTH, + `vmLocation must be at most ${VM_LOCATION_ID_MAX_LENGTH} characters` + ), + v.regex(VM_LOCATION_ID_REGEX, 'vmLocation must be a valid provider location identifier') +); diff --git a/apps/api/src/schemas/projects.ts b/apps/api/src/schemas/projects.ts index 858cab2ea9..add53739a0 100644 --- a/apps/api/src/schemas/projects.ts +++ b/apps/api/src/schemas/projects.ts @@ -12,6 +12,8 @@ const CredentialProviderSchema = v.picklist([ ]); const VMSizeSchema = v.picklist(['small', 'medium', 'large']); const WorkspaceProfileSchema = v.picklist(['full', 'lightweight']); +const PositiveIntegerSchema = v.pipe(v.number(), v.integer(), v.minValue(1)); +const PercentageSchema = v.pipe(v.number(), v.integer(), v.minValue(0), v.maxValue(100)); // Per-agent-type override (model + permission mode). Both fields are optional and nullable. // Null = clear the override for that field; missing = leave unchanged. @@ -54,9 +56,9 @@ export const UpdateProjectSchema = v.object({ maxDispatchDepth: v.optional(v.nullable(v.number())), maxSubTasksPerTask: v.optional(v.nullable(v.number())), warmNodeTimeoutMs: v.optional(v.nullable(v.number())), - maxWorkspacesPerNode: v.optional(v.nullable(v.number())), - nodeCpuThresholdPercent: v.optional(v.nullable(v.number())), - nodeMemoryThresholdPercent: v.optional(v.nullable(v.number())), + maxWorkspacesPerNode: v.optional(v.nullable(PositiveIntegerSchema)), + nodeCpuThresholdPercent: v.optional(v.nullable(PercentageSchema)), + nodeMemoryThresholdPercent: v.optional(v.nullable(PercentageSchema)), }); export const UpsertProjectRuntimeEnvVarSchema = v.object({ diff --git a/apps/api/src/schemas/tasks.ts b/apps/api/src/schemas/tasks.ts index c2a7f302e4..e4930f982b 100644 --- a/apps/api/src/schemas/tasks.ts +++ b/apps/api/src/schemas/tasks.ts @@ -4,8 +4,9 @@ import { } from '@simple-agent-manager/shared'; import * as v from 'valibot'; +import { PlacementNodeIdSchema, VMLocationSchema } from './placement'; + const VMSizeSchema = v.picklist(['small', 'medium', 'large']); -const VMLocationSchema = v.string(); const WorkspaceProfileSchema = v.picklist(['full', 'lightweight']); const CredentialProviderSchema = v.picklist([ 'hetzner', @@ -83,7 +84,7 @@ export const SubmitTaskSchema = v.object({ message: v.string(), vmSize: v.optional(VMSizeSchema), vmLocation: v.optional(VMLocationSchema), - nodeId: v.optional(v.string()), + nodeId: v.optional(PlacementNodeIdSchema), agentType: v.optional(v.string()), workspaceProfile: v.optional(WorkspaceProfileSchema), devcontainerConfigName: v.optional(v.nullable(DevcontainerConfigNameSchema)), @@ -136,7 +137,7 @@ export const RunTaskSchema = v.object({ vmLocation: v.optional(VMLocationSchema), workspaceProfile: v.optional(WorkspaceProfileSchema), devcontainerConfigName: v.optional(v.nullable(DevcontainerConfigNameSchema)), - nodeId: v.optional(v.string()), + nodeId: v.optional(PlacementNodeIdSchema), branch: v.optional(v.string()), }); diff --git a/apps/api/src/schemas/workspaces.ts b/apps/api/src/schemas/workspaces.ts index 2f93dcd4fd..846fa5f83a 100644 --- a/apps/api/src/schemas/workspaces.ts +++ b/apps/api/src/schemas/workspaces.ts @@ -1,5 +1,7 @@ import * as v from 'valibot'; +import { PlacementNodeIdSchema, VMLocationSchema } from './placement'; + const CredentialProviderSchema = v.picklist([ 'hetzner', 'scaleway', @@ -15,11 +17,11 @@ const CredentialKindSchema = v.picklist(['api-key', 'oauth-token']); export const CreateWorkspaceSchema = v.object({ name: v.string(), projectId: v.string(), - nodeId: v.optional(v.string()), + nodeId: v.optional(PlacementNodeIdSchema), repository: v.optional(v.string()), branch: v.optional(v.string()), vmSize: v.optional(VMSizeSchema), - vmLocation: v.optional(v.string()), + vmLocation: v.optional(VMLocationSchema), installationId: v.optional(v.string()), provider: v.optional(CredentialProviderSchema), }); diff --git a/apps/api/src/services/node-selector.ts b/apps/api/src/services/node-selector.ts index 4d92fb48e0..1e017911ee 100644 --- a/apps/api/src/services/node-selector.ts +++ b/apps/api/src/services/node-selector.ts @@ -1,29 +1,28 @@ -import type { NodeMetrics } from '@simple-agent-manager/shared'; -import { - canSatisfyVmSize, - DEFAULT_MAX_WORKSPACES_PER_NODE, - DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, +import type { + NodeMetrics, + PlacementExplanation, + PlacementNodeEvaluation, + PlacementSelectionPath, + VMSize, } from '@simple-agent-manager/shared'; -import { and, count, eq, inArray, isNotNull } from 'drizzle-orm'; +import { DEFAULT_VM_LOCATION, DEFAULT_VM_SIZE } from '@simple-agent-manager/shared'; +import { and, eq, inArray } from 'drizzle-orm'; import { type drizzle } from 'drizzle-orm/d1'; -import * as v from 'valibot'; import * as schema from '../db/schema'; -import { isNodeAgentVersionCompatible } from './node-agent-compatibility'; import * as nodeLifecycle from './node-lifecycle'; - -// Mirrors NodeMetrics (packages/shared/src/types/workspace.ts) exactly: all -// three fields are optional heartbeat samples. A present field with the wrong -// type (e.g. a stringified number) previously slipped through the old -// `typeof parsed.x === 'number'`-on-at-least-one-field check and got blindly -// cast, so a single mistyped field poisoned scoreNodeLoad()/nodeHasCapacity() -// with NaN instead of being treated as absent metrics. -const nodeMetricsSchema = v.object({ - cpuLoadAvg1: v.optional(v.number()), - memoryPercent: v.optional(v.number()), - diskPercent: v.optional(v.number()), -}); +import { + createPlacementExplanation, + evaluatePlacementNode, + type PlacementLimitsEnv, + type PlacementLimitsOverride, + placementLoadScore, + type PlacementNodeInput, + requireProvisioning, + resolvePlacementRequest, + sanitizePlacementExplanation, + selectPlacementNode, +} from './placement-explanation'; export interface NodeCandidate { id: string; @@ -35,318 +34,328 @@ export interface NodeCandidate { activeWorkspaceCount: number; } -export interface NodeSelectionResult { - nodeId: string; - autoProvisioned: boolean; -} - -export interface NodeSelectorEnv { - TASK_RUN_NODE_CPU_THRESHOLD_PERCENT?: string; - TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT?: string; - MAX_WORKSPACES_PER_NODE?: string; +export interface NodeSelectorEnv extends PlacementLimitsEnv { NODE_LIFECYCLE?: DurableObjectNamespace; - VM_AGENT_REQUIRED_VERSION?: string; } -function parseThreshold(value: string | undefined, fallback: number): number { - if (!value) return fallback; - const parsed = Number.parseInt(value, 10); - if (!Number.isFinite(parsed) || parsed < 0 || parsed > 100) return fallback; - return parsed; +export interface NodePlacementOptions { + vmSize: VMSize; + vmLocation: string; + taskId?: string; + preferredNodeId?: string; + preferredOnly?: boolean; + selectionPath?: 'trial' | 'manual'; + limits?: PlacementLimitsOverride; + nowMs?: number; } -function parsePositiveInt(value: string | undefined, fallback: number): number { - if (!value) return fallback; - const parsed = Number.parseInt(value, 10); - if (!Number.isFinite(parsed) || parsed < 1) return fallback; - return parsed; +export interface NodePlacementResult { + node: NodeCandidate | null; + explanation: PlacementExplanation; } +type PlacementDb = ReturnType>; + function parseMetrics(raw: string | null): NodeMetrics | null { if (!raw) return null; - let parsed: unknown; try { - parsed = JSON.parse(raw); + const value = JSON.parse(raw) as unknown; + if (typeof value !== 'object' || value === null || Array.isArray(value)) return null; + const candidate = value as Record; + if ( + ['cpuLoadAvg1', 'memoryPercent', 'diskPercent'].some( + (field) => + field in candidate && + (typeof candidate[field] !== 'number' || !Number.isFinite(candidate[field])) + ) + ) { + return null; + } + const metrics: NodeMetrics = {}; + if (typeof candidate.cpuLoadAvg1 === 'number' && Number.isFinite(candidate.cpuLoadAvg1)) { + metrics.cpuLoadAvg1 = candidate.cpuLoadAvg1; + } + if (typeof candidate.memoryPercent === 'number' && Number.isFinite(candidate.memoryPercent)) { + metrics.memoryPercent = candidate.memoryPercent; + } + if (typeof candidate.diskPercent === 'number' && Number.isFinite(candidate.diskPercent)) { + metrics.diskPercent = candidate.diskPercent; + } + return Object.keys(metrics).length > 0 ? metrics : null; } catch { return null; } - const result = v.safeParse(nodeMetricsSchema, parsed); - if (!result.success) return null; - const metrics = result.output; - // Preserve the existing "at least one recognized field present" gate — a - // technically-valid-but-empty metrics object is still treated as absent. - if ( - metrics.cpuLoadAvg1 === undefined && - metrics.memoryPercent === undefined && - metrics.diskPercent === undefined - ) { - return null; - } - return metrics; } -/** - * Score a node by its resource usage. Lower score = more available capacity. - * Returns a value between 0 and 100, where 0 is fully idle and 100 is fully loaded. - * Returns null if metrics are unavailable (node can still be used but ranked lower). - */ export function scoreNodeLoad(metrics: NodeMetrics | null): number | null { if (!metrics) return null; - - const cpu = metrics.cpuLoadAvg1 ?? 0; - const memory = metrics.memoryPercent ?? 0; - - // Weighted average: 40% CPU, 60% memory (memory is more constraining for agent workloads) - return cpu * 0.4 + memory * 0.6; + return Math.max(metrics.cpuLoadAvg1 ?? 0, metrics.memoryPercent ?? 0); } -/** - * Determine if a node has capacity for another workspace based on resource thresholds. - * This checks CPU/memory metrics only. The hard workspace count limit is enforced - * separately in selectNodeForTaskRun() after computing activeCount. - */ export function nodeHasCapacity( metrics: NodeMetrics | null, cpuThreshold: number, memoryThreshold: number ): boolean { - if (!metrics) { - // If no metrics available, allow it (node may still be starting up) - return true; + if (!metrics) return true; + return ( + (metrics.cpuLoadAvg1 ?? 0) < cpuThreshold && (metrics.memoryPercent ?? 0) < memoryThreshold + ); +} + +async function loadPlacementNodes(db: PlacementDb, userId: string): Promise { + const rows = await db + .select({ + id: schema.nodes.id, + status: schema.nodes.status, + runtime: schema.nodes.runtime, + vmSize: schema.nodes.vmSize, + vmLocation: schema.nodes.vmLocation, + healthStatus: schema.nodes.healthStatus, + lastHeartbeatAt: schema.nodes.lastHeartbeatAt, + agentReadyAt: schema.nodes.agentReadyAt, + agentVersion: schema.nodes.agentVersion, + lastMetrics: schema.nodes.lastMetrics, + warmSince: schema.nodes.warmSince, + }) + .from(schema.nodes) + .where(and(eq(schema.nodes.userId, userId), eq(schema.nodes.nodeRole, 'workspace'))); + + if (rows.length === 0) return []; + const counts = await db + .select({ nodeId: schema.workspaces.nodeId, id: schema.workspaces.id }) + .from(schema.workspaces) + .where( + and( + eq(schema.workspaces.userId, userId), + inArray(schema.workspaces.status, ['running', 'creating', 'recovery']) + ) + ); + const countByNode = new Map(); + for (const row of counts) { + if (row.nodeId) countByNode.set(row.nodeId, (countByNode.get(row.nodeId) ?? 0) + 1); } + return rows.map((row) => ({ + ...row, + activeWorkspaceCount: countByNode.get(row.id) ?? 0, + })); +} - const cpu = metrics.cpuLoadAvg1 ?? 0; - const memory = metrics.memoryPercent ?? 0; +function toCandidate(node: PlacementNodeInput): NodeCandidate { + return { + id: node.id, + status: node.status, + healthStatus: node.healthStatus ?? 'unknown', + vmSize: node.vmSize, + vmLocation: node.vmLocation, + lastMetrics: parseMetrics(node.lastMetrics), + activeWorkspaceCount: node.activeWorkspaceCount, + }; +} - return cpu < cpuThreshold && memory < memoryThreshold; +function sortEvaluations( + evaluations: PlacementNodeEvaluation[], + vmSize: string, + vmLocation: string +): PlacementNodeEvaluation[] { + return evaluations.sort((a, b) => { + const aLocation = a.snapshot.vmLocation === vmLocation ? 1 : 0; + const bLocation = b.snapshot.vmLocation === vmLocation ? 1 : 0; + if (aLocation !== bLocation) return bLocation - aLocation; + const aSize = a.snapshot.vmSize === vmSize ? 1 : 0; + const bSize = b.snapshot.vmSize === vmSize ? 1 : 0; + if (aSize !== bSize) return bSize - aSize; + const aScore = placementLoadScore(a); + const bScore = placementLoadScore(b); + if (aScore === null && bScore === null) return a.nodeId.localeCompare(b.nodeId); + if (aScore === null) return 1; + if (bScore === null) return -1; + return aScore - bScore || a.nodeId.localeCompare(b.nodeId); + }); } -/** - * Select the best available node for a task run, or indicate that a new node is needed. - * - * Selection algorithm: - * 0. Try to claim a warm node first (fast startup for sequential tasks) - * 1. Get all running, healthy (or stale) nodes for the user - * 2. For each node, check resource metrics (CPU/memory thresholds) - * 3. Filter to nodes with capacity - * 4. If a specific vmLocation is requested, prefer nodes in that location - * 5. Sort by load score (lowest first) — prefer the least loaded node - * 6. Return the best node, or null if no node has capacity - */ -export async function selectNodeForTaskRun( - db: ReturnType>, +function missingNodeEvaluation( + nodeId: string, + path: PlacementNodeEvaluation['path'] +): PlacementNodeEvaluation { + return { + nodeId, + path, + accepted: false, + rejectionReasons: ['node-not-found'], + snapshot: { + runtime: 'vm', + vmSize: 'unknown', + vmLocation: 'unknown', + healthStatus: 'unknown', + agentVersionCompatible: false, + heartbeatAgeSeconds: null, + activeWorkspaceCount: 0, + cpuLoadAvg1: null, + memoryPercent: null, + }, + }; +} + +function placementResult( + node: NodeCandidate | null, + explanation: PlacementExplanation +): NodePlacementResult { + return { node, explanation: sanitizePlacementExplanation(explanation) }; +} + +export async function selectNodeWithExplanation( + db: PlacementDb, userId: string, env: NodeSelectorEnv, - preferredLocation?: string, - preferredSize?: string, - taskId?: string -): Promise { - const cpuThreshold = parseThreshold( - env.TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT - ); - const memoryThreshold = parseThreshold( - env.TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, - DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT - ); - const maxWorkspacesPerNode = parsePositiveInt( - env.MAX_WORKSPACES_PER_NODE, - DEFAULT_MAX_WORKSPACES_PER_NODE - ); + options: NodePlacementOptions +): Promise { + const nowMs = options.nowMs ?? Date.now(); + const now = new Date(nowMs).toISOString(); + const request = resolvePlacementRequest(env, options.vmSize, options.vmLocation, options.limits); + const initialPath: PlacementSelectionPath = options.preferredNodeId + ? options.selectionPath === 'manual' + ? 'manual' + : 'preferred' + : (options.selectionPath ?? 'capacity'); + let explanation = createPlacementExplanation(request, initialPath, now); + const nodes = await loadPlacementNodes(db, userId); + + if (options.preferredNodeId) { + const node = nodes.find((candidate) => candidate.id === options.preferredNodeId); + const path = initialPath as 'preferred' | 'manual'; + const evaluation = node + ? evaluatePlacementNode(node, request, path, env.VM_AGENT_REQUIRED_VERSION, nowMs) + : missingNodeEvaluation(options.preferredNodeId, path); + explanation.evaluatedNodes.push(evaluation); + if (node && evaluation.accepted) { + return placementResult(toCandidate(node), selectPlacementNode(explanation, evaluation, now)); + } + explanation.summary = 'The preferred node was rejected.'; + explanation.outcome = 'failed'; + return placementResult(null, explanation); + } - // Step 0: Try to claim a warm node (fast startup for sequential tasks) - if (taskId && env.NODE_LIFECYCLE) { - const warmNodes = await db - .select({ - id: schema.nodes.id, - status: schema.nodes.status, - healthStatus: schema.nodes.healthStatus, - vmSize: schema.nodes.vmSize, - vmLocation: schema.nodes.vmLocation, - lastMetrics: schema.nodes.lastMetrics, - warmSince: schema.nodes.warmSince, - agentVersion: schema.nodes.agentVersion, - }) - .from(schema.nodes) - .where( - and( - eq(schema.nodes.userId, userId), - eq(schema.nodes.status, 'running'), - eq(schema.nodes.nodeRole, 'workspace'), - isNotNull(schema.nodes.warmSince) - ) + if (options.taskId && env.NODE_LIFECYCLE && !options.selectionPath) { + const warmExclusions = new Map(); + const warmEvaluations = sortEvaluations( + nodes.map((node) => + evaluatePlacementNode(node, request, 'warm', env.VM_AGENT_REQUIRED_VERSION, nowMs) + ), + options.vmSize, + options.vmLocation + ); + explanation.evaluatedNodes.push(...warmEvaluations); + for (const evaluation of warmEvaluations.filter((item) => item.accepted)) { + // Refresh every eligibility input before taking the atomic lifecycle + // claim. If the node changed since the candidate read, exclude it without + // consuming its warm state or cancelling its teardown alarm. + const freshNode = (await loadPlacementNodes(db, userId)).find( + (candidate) => candidate.id === evaluation.nodeId ); - - // Try each warm node that can satisfy the requested size, preferring exact size/location. - const sortedWarm = warmNodes - .filter((node) => - isNodeAgentVersionCompatible(node.agentVersion, env.VM_AGENT_REQUIRED_VERSION) - ) - .filter((node) => canSatisfyVmSize(node.vmSize, preferredSize)) - .sort((a, b) => { - const aSizeMatch = preferredSize && a.vmSize === preferredSize ? 1 : 0; - const bSizeMatch = preferredSize && b.vmSize === preferredSize ? 1 : 0; - if (aSizeMatch !== bSizeMatch) return bSizeMatch - aSizeMatch; - const aLocMatch = preferredLocation && a.vmLocation === preferredLocation ? 1 : 0; - const bLocMatch = preferredLocation && b.vmLocation === preferredLocation ? 1 : 0; - return bLocMatch - aLocMatch; - }); - - for (const warmNode of sortedWarm) { + const freshEvaluation = freshNode + ? evaluatePlacementNode(freshNode, request, 'warm', env.VM_AGENT_REQUIRED_VERSION, nowMs) + : missingNodeEvaluation(evaluation.nodeId, 'warm'); + evaluation.snapshot = freshEvaluation.snapshot; + if (!freshNode || !freshEvaluation.accepted) { + evaluation.accepted = false; + evaluation.rejectionReasons = freshEvaluation.rejectionReasons; + warmExclusions.set(evaluation.nodeId, evaluation.rejectionReasons); + continue; + } try { - // Defense-in-depth: re-check D1 status before DO call to avoid - // unnecessary DO round-trips for nodes that changed between query and claim. - const [freshNode] = await db - .select({ - status: schema.nodes.status, - warmSince: schema.nodes.warmSince, - agentVersion: schema.nodes.agentVersion, - }) - .from(schema.nodes) - .where(eq(schema.nodes.id, warmNode.id)) - .limit(1); - - if ( - !freshNode || - freshNode.status !== 'running' || - !freshNode.warmSince || - !isNodeAgentVersionCompatible(freshNode.agentVersion, env.VM_AGENT_REQUIRED_VERSION) - ) { - continue; // Node state changed since initial query - } - - const result = await nodeLifecycle.tryClaim( + const claimed = await nodeLifecycle.tryClaim( env as unknown as import('../env').Env, - warmNode.id, - taskId + evaluation.nodeId, + options.taskId ); - if (result.claimed) { - // Defense-in-depth: verify workspace count even for warm nodes - const [wsCountRow] = await db - .select({ count: count() }) - .from(schema.workspaces) - .where( - and( - eq(schema.workspaces.nodeId, warmNode.id), - eq(schema.workspaces.userId, userId), - inArray(schema.workspaces.status, ['running', 'creating', 'recovery']) - ) - ); - const warmActiveCount = wsCountRow?.count ?? 0; - if (warmActiveCount >= maxWorkspacesPerNode) { - continue; // At capacity despite being warm — skip - } - return { - id: warmNode.id, - status: warmNode.status, - healthStatus: warmNode.healthStatus, - vmSize: warmNode.vmSize, - vmLocation: warmNode.vmLocation, - lastMetrics: parseMetrics(warmNode.lastMetrics), - activeWorkspaceCount: warmActiveCount, - }; + if (claimed.claimed) { + return placementResult( + toCandidate(freshNode), + selectPlacementNode(explanation, evaluation, now) + ); } } catch { - // tryClaim failed (e.g. concurrent claim) — try next + // The typed reason below intentionally replaces raw DO/provider errors. } + evaluation.accepted = false; + evaluation.rejectionReasons.push('warm-claim-lost'); + warmExclusions.set(evaluation.nodeId, evaluation.rejectionReasons); } - } - - // Get all running nodes for this user (deployment-role nodes excluded — not eligible for task placement) - const nodes = await db - .select() - .from(schema.nodes) - .where( - and( - eq(schema.nodes.userId, userId), - eq(schema.nodes.status, 'running'), - eq(schema.nodes.nodeRole, 'workspace') - ) + const evaluations = sortEvaluations( + nodes.map((node) => { + const evaluation = evaluatePlacementNode( + node, + request, + 'capacity', + env.VM_AGENT_REQUIRED_VERSION, + nowMs + ); + const exclusionReasons = warmExclusions.get(evaluation.nodeId); + if (exclusionReasons) { + evaluation.accepted = false; + evaluation.rejectionReasons = [ + ...new Set([...evaluation.rejectionReasons, ...exclusionReasons]), + ]; + } + return evaluation; + }), + options.vmSize, + options.vmLocation ); - - if (nodes.length === 0) { - return null; - } - - // Filter by resource metrics (CPU/memory thresholds) - const candidates: NodeCandidate[] = []; - for (const node of nodes) { - // Skip unhealthy nodes - if (node.healthStatus === 'unhealthy') { - continue; - } - if (!isNodeAgentVersionCompatible(node.agentVersion, env.VM_AGENT_REQUIRED_VERSION)) { - continue; - } - if (!canSatisfyVmSize(node.vmSize, preferredSize)) { - continue; - } - - const [wsCountRow] = await db - .select({ count: count() }) - .from(schema.workspaces) - .where( - and( - eq(schema.workspaces.nodeId, node.id), - eq(schema.workspaces.userId, userId), - inArray(schema.workspaces.status, ['running', 'creating', 'recovery']) - ) + explanation.evaluatedNodes.push(...evaluations); + const selected = evaluations.find((evaluation) => evaluation.accepted); + const selectedNode = selected + ? (nodes.find((candidate) => candidate.id === selected.nodeId) ?? null) + : null; + if (selected && selectedNode) { + return placementResult( + toCandidate(selectedNode), + selectPlacementNode(explanation, selected, now) ); - - const activeCount = wsCountRow?.count ?? 0; - - // Hard workspace count limit — reject node regardless of CPU/memory metrics - if (activeCount >= maxWorkspacesPerNode) { - continue; } - - const metrics = parseMetrics(node.lastMetrics); - - const candidate: NodeCandidate = { - id: node.id, - status: node.status, - healthStatus: node.healthStatus, - vmSize: node.vmSize, - vmLocation: node.vmLocation, - lastMetrics: metrics, - activeWorkspaceCount: activeCount, - }; - - if (nodeHasCapacity(metrics, cpuThreshold, memoryThreshold)) { - candidates.push(candidate); + } else { + const capacityPath = options.selectionPath ?? 'capacity'; + const evaluations = sortEvaluations( + nodes.map((node) => + evaluatePlacementNode(node, request, capacityPath, env.VM_AGENT_REQUIRED_VERSION, nowMs) + ), + options.vmSize, + options.vmLocation + ); + explanation.evaluatedNodes.push(...evaluations); + const selected = evaluations.find((evaluation) => evaluation.accepted); + const selectedNode = selected + ? (nodes.find((candidate) => candidate.id === selected.nodeId) ?? null) + : null; + if (selected && selectedNode) { + return placementResult( + toCandidate(selectedNode), + selectPlacementNode(explanation, selected, now) + ); } } + explanation = requireProvisioning(explanation, now); + return placementResult(null, explanation); +} - if (candidates.length === 0) { - return null; - } - - // Sort candidates: prefer matching location/size, then lowest load - candidates.sort((a, b) => { - // Prefer matching location - const aLocationMatch = preferredLocation && a.vmLocation === preferredLocation ? 1 : 0; - const bLocationMatch = preferredLocation && b.vmLocation === preferredLocation ? 1 : 0; - if (aLocationMatch !== bLocationMatch) return bLocationMatch - aLocationMatch; - - // Prefer matching size - const aSizeMatch = preferredSize && a.vmSize === preferredSize ? 1 : 0; - const bSizeMatch = preferredSize && b.vmSize === preferredSize ? 1 : 0; - if (aSizeMatch !== bSizeMatch) return bSizeMatch - aSizeMatch; - - // Prefer lowest load score - const aScore = scoreNodeLoad(a.lastMetrics); - const bScore = scoreNodeLoad(b.lastMetrics); - if (aScore === null && bScore === null) return 0; - if (aScore === null) return 1; - if (bScore === null) return -1; - return aScore - bScore; - }); - - const best = candidates[0]; - if (!best) { - // candidates.length === 0 was already handled above, and sort() does not - // change the array length — this should never happen. - return null; - } - return best; +/** Backward-compatible selector wrapper for callers that only need the selected node. */ +export async function selectNodeForTaskRun( + db: PlacementDb, + userId: string, + env: NodeSelectorEnv, + preferredLocation = DEFAULT_VM_LOCATION, + preferredSize: string = DEFAULT_VM_SIZE, + taskId?: string +): Promise { + const vmSize = ['small', 'medium', 'large'].includes(preferredSize) + ? (preferredSize as VMSize) + : DEFAULT_VM_SIZE; + return ( + await selectNodeWithExplanation(db, userId, env, { + vmSize, + vmLocation: preferredLocation, + taskId, + }) + ).node; } diff --git a/apps/api/src/services/placement-explanation.ts b/apps/api/src/services/placement-explanation.ts new file mode 100644 index 0000000000..ccdd06dc9c --- /dev/null +++ b/apps/api/src/services/placement-explanation.ts @@ -0,0 +1,343 @@ +import type { + PlacementExplanation, + PlacementNodeEvaluation, + PlacementProvisioningAttempt, + PlacementProvisioningFailureReason, + PlacementRequestSnapshot, + PlacementSelectionPath, + VMSize, +} from '@simple-agent-manager/shared'; +import { + canSatisfyVmSize, + DEFAULT_MAX_WORKSPACES_PER_NODE, + DEFAULT_NODE_HEARTBEAT_STALE_SECONDS, + DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, + DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, + isSafeVmLocationId, +} from '@simple-agent-manager/shared'; + +import { isNodeAgentVersionCompatible } from './node-agent-compatibility'; + +export interface PlacementLimitsEnv { + TASK_RUN_NODE_CPU_THRESHOLD_PERCENT?: string; + TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT?: string; + MAX_WORKSPACES_PER_NODE?: string; + NODE_HEARTBEAT_STALE_SECONDS?: string; + VM_AGENT_REQUIRED_VERSION?: string; +} + +export interface PlacementLimitsOverride { + cpuThresholdPercent?: number; + memoryThresholdPercent?: number; + maxWorkspacesPerNode?: number; + heartbeatStaleSeconds?: number; +} + +export interface PlacementNodeInput { + id: string; + status: string; + runtime: string | null; + vmSize: string; + vmLocation: string; + healthStatus: string | null; + lastHeartbeatAt: string | null; + agentReadyAt: string | null; + agentVersion: string | null; + lastMetrics: string | null; + activeWorkspaceCount: number; + warmSince: string | null; +} + +function envInt( + value: string | undefined, + fallback: number, + min: number, + max = Number.MAX_SAFE_INTEGER +): number { + const parsed = Number.parseInt(value ?? '', 10); + return Number.isSafeInteger(parsed) && parsed >= min && parsed <= max ? parsed : fallback; +} + +function overrideInt( + value: number | undefined, + fallback: number, + min: number, + max?: number +): number { + return value === undefined || + !Number.isSafeInteger(value) || + value < min || + (max !== undefined && value > max) + ? fallback + : value; +} + +function bounded(value: unknown, min: number, max: number): number | null { + return typeof value === 'number' && Number.isFinite(value) && value >= min + ? Math.min(value, max) + : null; +} + +function safeLocation(value: string): string { + const normalized = value.trim(); + return isSafeVmLocationId(normalized) ? normalized : 'unknown'; +} + +function parseMetricSnapshot(raw: string | null): { + cpuLoadAvg1: number | null; + memoryPercent: number | null; +} { + if (!raw) return { cpuLoadAvg1: null, memoryPercent: null }; + try { + const value = JSON.parse(raw) as unknown; + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + return { cpuLoadAvg1: null, memoryPercent: null }; + } + const metrics = value as Record; + const invalidRecognizedField = ['cpuLoadAvg1', 'memoryPercent'].some( + (field) => + field in metrics && (typeof metrics[field] !== 'number' || !Number.isFinite(metrics[field])) + ); + if (invalidRecognizedField) return { cpuLoadAvg1: null, memoryPercent: null }; + return { + cpuLoadAvg1: bounded(metrics.cpuLoadAvg1, 0, 100), + memoryPercent: bounded(metrics.memoryPercent, 0, 100), + }; + } catch { + return { cpuLoadAvg1: null, memoryPercent: null }; + } +} + +export function resolvePlacementRequest( + env: PlacementLimitsEnv, + vmSize: VMSize, + vmLocation: string, + override: PlacementLimitsOverride = {} +): PlacementRequestSnapshot { + return { + runtime: 'vm', + vmSize, + vmLocation: safeLocation(vmLocation), + maxWorkspacesPerNode: overrideInt( + override.maxWorkspacesPerNode, + envInt(env.MAX_WORKSPACES_PER_NODE, DEFAULT_MAX_WORKSPACES_PER_NODE, 1), + 1 + ), + cpuThresholdPercent: overrideInt( + override.cpuThresholdPercent, + envInt( + env.TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, + DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, + 0, + 100 + ), + 0, + 100 + ), + memoryThresholdPercent: overrideInt( + override.memoryThresholdPercent, + envInt( + env.TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, + DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT, + 0, + 100 + ), + 0, + 100 + ), + heartbeatStaleSeconds: overrideInt( + override.heartbeatStaleSeconds, + envInt(env.NODE_HEARTBEAT_STALE_SECONDS, DEFAULT_NODE_HEARTBEAT_STALE_SECONDS, 1), + 1 + ), + }; +} + +export function createPlacementExplanation( + request: PlacementRequestSnapshot, + selectionPath: PlacementSelectionPath, + now = new Date().toISOString() +): PlacementExplanation { + return { + schemaVersion: 2, + outcome: selectionPath === 'preferred' || selectionPath === 'manual' ? 'failed' : 'provisioned', + selectionPath, + selectedNodeId: null, + summary: 'No reusable node qualified; provisioning is required.', + request, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: now, + updatedAt: now, + }; +} + +export function evaluatePlacementNode( + node: PlacementNodeInput, + request: PlacementRequestSnapshot, + path: Exclude, + requiredAgentVersion: string | undefined, + nowMs = Date.now() +): PlacementNodeEvaluation { + const reasons: PlacementNodeEvaluation['rejectionReasons'] = []; + const metrics = parseMetricSnapshot(node.lastMetrics); + const heartbeatMs = node.lastHeartbeatAt ? Date.parse(node.lastHeartbeatAt) : Number.NaN; + const heartbeatAgeSeconds = Number.isFinite(heartbeatMs) + ? bounded(Math.max(0, (nowMs - heartbeatMs) / 1000), 0, Number.MAX_SAFE_INTEGER) + : null; + const agentVersionCompatible = isNodeAgentVersionCompatible( + node.agentVersion, + requiredAgentVersion + ); + + if (node.status !== 'running') reasons.push('not-running'); + if (node.runtime === 'cf-container') reasons.push('wrong-runtime'); + if (node.healthStatus !== 'healthy') reasons.push('unhealthy'); + if (!node.lastHeartbeatAt || heartbeatAgeSeconds === null) reasons.push('heartbeat-missing'); + else if (heartbeatAgeSeconds >= request.heartbeatStaleSeconds) reasons.push('heartbeat-stale'); + if (!node.agentReadyAt) reasons.push('agent-not-ready'); + if (!agentVersionCompatible) reasons.push('agent-version-mismatch'); + if (!canSatisfyVmSize(node.vmSize, request.vmSize)) reasons.push('undersized'); + if (node.activeWorkspaceCount >= request.maxWorkspacesPerNode) { + reasons.push('workspace-limit'); + } + if (metrics.cpuLoadAvg1 !== null && metrics.cpuLoadAvg1 >= request.cpuThresholdPercent) { + reasons.push('cpu-threshold'); + } + if (metrics.memoryPercent !== null && metrics.memoryPercent >= request.memoryThresholdPercent) { + reasons.push('memory-threshold'); + } + if (path === 'warm' && !node.warmSince) reasons.push('not-warm'); + + const healthStatus = ['healthy', 'stale', 'unhealthy'].includes(node.healthStatus ?? '') + ? (node.healthStatus as 'healthy' | 'stale' | 'unhealthy') + : 'unknown'; + return { + nodeId: node.id, + path, + accepted: reasons.length === 0, + rejectionReasons: reasons, + snapshot: { + runtime: node.runtime === 'cf-container' ? 'other' : 'vm', + vmSize: node.vmSize, + vmLocation: safeLocation(node.vmLocation), + healthStatus, + agentVersionCompatible, + heartbeatAgeSeconds, + activeWorkspaceCount: Math.max( + 0, + Math.min(node.activeWorkspaceCount, Number.MAX_SAFE_INTEGER) + ), + cpuLoadAvg1: metrics.cpuLoadAvg1, + memoryPercent: metrics.memoryPercent, + }, + }; +} + +export function selectPlacementNode( + explanation: PlacementExplanation, + evaluation: PlacementNodeEvaluation, + now = new Date().toISOString() +): PlacementExplanation { + return { + ...explanation, + outcome: 'reused', + selectionPath: evaluation.path, + selectedNodeId: evaluation.nodeId, + summary: `Reused node ${evaluation.nodeId} through the ${evaluation.path} path.`, + updatedAt: now, + }; +} + +/** + * Remove cross-project/trial host identifiers from unselected candidates before + * any placement explanation is persisted, logged, or exposed. The selected + * node remains visible because it is already part of the workspace contract. + */ +export function sanitizePlacementExplanation( + explanation: PlacementExplanation +): PlacementExplanation { + const aliases = new Map(); + const evaluatedNodes = explanation.evaluatedNodes.map((evaluation) => { + if (evaluation.nodeId === explanation.selectedNodeId) return evaluation; + let alias = aliases.get(evaluation.nodeId); + if (!alias) { + alias = `candidate-${aliases.size + 1}`; + aliases.set(evaluation.nodeId, alias); + } + return { ...evaluation, nodeId: alias }; + }); + return { + ...explanation, + request: { + ...explanation.request, + vmLocation: safeLocation(explanation.request.vmLocation), + }, + evaluatedNodes, + provisioningAttempts: explanation.provisioningAttempts.map((attempt) => ({ + ...attempt, + vmLocation: safeLocation(attempt.vmLocation), + })), + }; +} + +export function requireProvisioning( + explanation: PlacementExplanation, + now = new Date().toISOString() +): PlacementExplanation { + return { + ...explanation, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: null, + summary: 'No reusable node qualified; provisioning a new node.', + updatedAt: now, + }; +} + +export function appendProvisioningAttempt( + explanation: PlacementExplanation, + attempt: PlacementProvisioningAttempt, + selectedNodeId?: string | null, + now = new Date().toISOString() +): PlacementExplanation { + const resolvedNodeId = selectedNodeId === undefined ? explanation.selectedNodeId : selectedNodeId; + return { + ...explanation, + outcome: attempt.outcome === 'failed' ? 'failed' : 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: resolvedNodeId, + summary: + attempt.outcome === 'succeeded' + ? `Provisioned node ${resolvedNodeId ?? 'successfully'}.` + : attempt.outcome === 'failed' + ? 'Node provisioning failed.' + : 'Provisioning a new node.', + provisioningAttempts: [ + ...explanation.provisioningAttempts, + { ...attempt, vmLocation: safeLocation(attempt.vmLocation) }, + ], + updatedAt: now, + }; +} + +export function failPlacement( + explanation: PlacementExplanation, + failureReason: PlacementProvisioningFailureReason, + vmSize = explanation.request.vmSize, + vmLocation = explanation.request.vmLocation, + now = new Date().toISOString() +): PlacementExplanation { + return appendProvisioningAttempt( + explanation, + { vmSize, vmLocation, outcome: 'failed', failureReason }, + explanation.selectedNodeId, + now + ); +} + +export function placementLoadScore(evaluation: PlacementNodeEvaluation): number | null { + const { cpuLoadAvg1, memoryPercent } = evaluation.snapshot; + if (cpuLoadAvg1 === null && memoryPercent === null) return null; + return Math.max(cpuLoadAvg1 ?? 0, memoryPercent ?? 0); +} diff --git a/apps/api/tests/integration/compute-quotas.test.ts b/apps/api/tests/integration/compute-quotas.test.ts index 5bda8fec8e..6afe62b4cd 100644 --- a/apps/api/tests/integration/compute-quotas.test.ts +++ b/apps/api/tests/integration/compute-quotas.test.ts @@ -18,17 +18,37 @@ import { describe, expect, it } from 'vitest'; describe('compute quota pipeline', () => { const schemaFile = readFileSync(resolve(process.cwd(), 'src/db/schema.ts'), 'utf8'); - const serviceFile = readFileSync(resolve(process.cwd(), 'src/services/compute-quotas.ts'), 'utf8'); - const providerCredsFile = readFileSync(resolve(process.cwd(), 'src/services/provider-credentials.ts'), 'utf8'); + const serviceFile = readFileSync( + resolve(process.cwd(), 'src/services/compute-quotas.ts'), + 'utf8' + ); + const providerCredsFile = readFileSync( + resolve(process.cwd(), 'src/services/provider-credentials.ts'), + 'utf8' + ); const indexFile = readFileSync(resolve(process.cwd(), 'src/index.ts'), 'utf8'); const envFile = readFileSync(resolve(process.cwd(), 'src/env.ts'), 'utf8'); - const adminQuotaRoute = readFileSync(resolve(process.cwd(), 'src/routes/admin-quotas.ts'), 'utf8'); + const adminQuotaRoute = readFileSync( + resolve(process.cwd(), 'src/routes/admin-quotas.ts'), + 'utf8' + ); const usageRoute = readFileSync(resolve(process.cwd(), 'src/routes/usage.ts'), 'utf8'); const submitRoute = readFileSync(resolve(process.cwd(), 'src/routes/tasks/submit.ts'), 'utf8'); - const nodeStepsFile = readFileSync(resolve(process.cwd(), 'src/durable-objects/task-runner/node-steps.ts'), 'utf8'); + const nodeStepsFile = [ + 'src/durable-objects/task-runner/node-steps.ts', + 'src/durable-objects/task-runner/provisioning-guards.ts', + ] + .map((file) => readFileSync(resolve(process.cwd(), file), 'utf8')) + .join('\n'); const nodesRoute = readFileSync(resolve(process.cwd(), 'src/routes/nodes.ts'), 'utf8'); - const migrationFile = readFileSync(resolve(process.cwd(), 'src/db/migrations/0039_compute_quotas.sql'), 'utf8'); - const dispatchToolFile = readFileSync(resolve(process.cwd(), 'src/routes/mcp/dispatch-tool.ts'), 'utf8'); + const migrationFile = readFileSync( + resolve(process.cwd(), 'src/db/migrations/0039_compute_quotas.sql'), + 'utf8' + ); + const dispatchToolFile = readFileSync( + resolve(process.cwd(), 'src/routes/mcp/dispatch-tool.ts'), + 'utf8' + ); // =========================================================================== // Migration @@ -47,7 +67,9 @@ describe('compute quota pipeline', () => { }); it('creates index on user_quotas', () => { - expect(migrationFile).toContain('CREATE INDEX idx_user_quotas_user_id ON user_quotas(user_id)'); + expect(migrationFile).toContain( + 'CREATE INDEX idx_user_quotas_user_id ON user_quotas(user_id)' + ); }); }); @@ -139,13 +161,17 @@ describe('compute quota pipeline', () => { }); it('checks user credentials for the target provider first', () => { - expect(providerCredsFile).toContain("eq(schema.credentials.credentialType, 'cloud-provider')"); + expect(providerCredsFile).toContain( + "eq(schema.credentials.credentialType, 'cloud-provider')" + ); // When targetProvider is passed, it filters by provider expect(providerCredsFile).toContain('eq(schema.credentials.provider, targetProvider)'); }); it('falls back to platform credentials', () => { - expect(providerCredsFile).toContain("eq(schema.platformCredentials.credentialType, 'cloud-provider')"); + expect(providerCredsFile).toContain( + "eq(schema.platformCredentials.credentialType, 'cloud-provider')" + ); expect(providerCredsFile).toContain('eq(schema.platformCredentials.isEnabled, true)'); }); @@ -168,7 +194,7 @@ describe('compute quota pipeline', () => { // =========================================================================== describe('admin quota routes', () => { it('routes are mounted at /api/admin/quotas', () => { - expect(indexFile).toContain("adminQuotaRoutes"); + expect(indexFile).toContain('adminQuotaRoutes'); expect(indexFile).toContain("'/api/admin/quotas'"); }); @@ -398,7 +424,9 @@ describe('compute quota pipeline', () => { }); it('MCP dispatch does NOT use raw credential existence check in Promise.all', () => { - expect(dispatchToolFile).not.toContain("eq(schema.credentials.credentialType, 'cloud-provider')"); + expect(dispatchToolFile).not.toContain( + "eq(schema.credentials.credentialType, 'cloud-provider')" + ); }); it('all four enforcement points use resolveCredentialSource', () => { diff --git a/apps/api/tests/integration/node-selection.test.ts b/apps/api/tests/integration/node-selection.test.ts index c85ec4b178..aa33a4aa5f 100644 --- a/apps/api/tests/integration/node-selection.test.ts +++ b/apps/api/tests/integration/node-selection.test.ts @@ -1,311 +1,71 @@ -/** - * Integration tests for node selection subsystem (TDF-3). - * - * Source contract tests verifying cross-module wiring: - * 1. Concurrent warm pool claiming: two tasks try to claim same node - * 2. D1 state changes between query and claim - * 3. Node selector -> NodeLifecycle DO -> D1 state coordination - * - * These tests validate the wiring between modules, not the individual - * function behavior (which is covered by unit tests). - */ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { describe, expect, it } from 'vitest'; -/** - * Extract a section of source code between two marker strings. - * Throws if either marker is not found or if end precedes start. - */ -function extractSection(source: string, startMarker: string, endMarker: string): string { - const start = source.indexOf(startMarker); - const end = source.indexOf(endMarker, start + 1); - if (start === -1) throw new Error(`Start marker not found: ${startMarker}`); - if (end === -1) throw new Error(`End marker not found: ${endMarker}`); - if (end <= start) - throw new Error(`End marker "${endMarker}" precedes start marker "${startMarker}"`); - return source.slice(start, end); +function read(path: string): string { + return readFileSync(resolve(process.cwd(), path), 'utf8'); } -const selectorSource = readFileSync( - resolve(process.cwd(), 'src/services/node-selector.ts'), - 'utf8' -); -const doSource = readFileSync( - resolve(process.cwd(), 'src/durable-objects/node-lifecycle.ts'), - 'utf8' -); -const serviceSource = readFileSync( - resolve(process.cwd(), 'src/services/node-lifecycle.ts'), - 'utf8' -); -const taskRunnerSource = [ - 'index.ts', - 'types.ts', +const selector = read('src/services/node-selector.ts'); +const evaluator = read('src/services/placement-explanation.ts'); +const taskRunner = [ 'node-steps.ts', - 'node-selection.ts', 'workspace-steps.ts', - 'agent-session-step.ts', + 'placement.ts', + 'provisioning-guards.ts', 'state-machine.ts', - 'helpers.ts', ] - .map((f) => readFileSync(resolve(process.cwd(), 'src/durable-objects/task-runner', f), 'utf8')) + .map((file) => read(`src/durable-objects/task-runner/${file}`)) .join('\n'); +const trial = ['steps.ts', 'placement.ts', 'index.ts'] + .map((file) => read(`src/durable-objects/trial-orchestrator/${file}`)) + .join('\n'); +const manual = ['crud.ts', 'manual-placement.ts'] + .map((file) => read(`src/routes/workspaces/${file}`)) + .join('\n'); +const lifecycle = read('src/durable-objects/node-lifecycle.ts'); -// ============================================================================= -// Concurrent warm pool claiming — safety mechanisms -// ============================================================================= - -describe('concurrent warm pool claiming safety', () => { - describe('NodeLifecycle DO tryClaim is the single point of truth', () => { - it('tryClaim only succeeds when status is warm', () => { - const tryClaimSection = extractSection(doSource, 'async tryClaim(', 'async getStatus('); - expect(tryClaimSection).toContain("state.status !== 'warm'"); - expect(tryClaimSection).toContain('claimed: false'); - }); - - it('tryClaim transitions warm -> active atomically via DO storage', () => { - const tryClaimSection = extractSection(doSource, 'async tryClaim(', 'async getStatus('); - // State update via storage.put is atomic within a DO - expect(tryClaimSection).toContain("state.status = 'active'"); - expect(tryClaimSection).toContain("this.ctx.storage.put('state', state)"); - }); - - it('tryClaim sets claimedByTask for traceability', () => { - const tryClaimSection = extractSection(doSource, 'async tryClaim(', 'async getStatus('); - expect(tryClaimSection).toContain('state.claimedByTask = taskId'); - }); - - it('second claim on same node returns claimed: false (already active)', () => { - // Once tryClaim succeeds, status is 'active'. Next tryClaim sees - // status !== 'warm' and returns false. - const tryClaimSection = extractSection(doSource, 'async tryClaim(', 'async getStatus('); - expect(tryClaimSection).toContain("state.status !== 'warm'"); - expect(tryClaimSection).toContain('{ claimed: false, state: this.toPublicState(state) }'); - }); - - it('tryClaim returns false for null state (uninitialized DO)', () => { - const tryClaimSection = doSource.slice( - doSource.indexOf('async tryClaim('), - doSource.indexOf('async getStatus(') - ); - expect(tryClaimSection).toContain('if (!state)'); - expect(tryClaimSection).toContain('claimed: false'); - }); - }); - - describe('defense-in-depth: D1 re-check before DO call', () => { - it('selectNodeForTaskRun re-queries D1 before each tryClaim', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - // The defense-in-depth check re-queries D1 - expect(warmSection).toContain('freshNode'); - expect(warmSection).toContain('eq(schema.nodes.id, warmNode.id)'); - }); - - it('skips node if D1 shows status changed to non-running', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain("freshNode.status !== 'running'"); - expect(warmSection).toContain('continue'); - }); - - it('skips node if D1 shows warmSince cleared', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain('!freshNode.warmSince'); - expect(warmSection).toContain('continue'); - }); - - it('skips node if D1 query returns no results', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain('!freshNode'); - expect(warmSection).toContain('continue'); - }); - }); - - describe('TaskRunner DO warm claiming uses same pattern', () => { - it('TaskRunner tryClaimWarmNode re-checks D1 freshness', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function tryClaimWarmNode('), - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(section).toContain("status = 'running' AND warm_since IS NOT NULL"); - // Fresh check query - expect(section).toContain("WHERE id = ? AND status = 'running' AND warm_since IS NOT NULL"); - }); - - it('TaskRunner tryClaimWarmNode claims via NodeLifecycle DO stub', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function tryClaimWarmNode('), - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(section).toContain('NODE_LIFECYCLE.idFromName(warmNode.id)'); - expect(section).toContain('stub.tryClaim(state.taskId)'); - }); - - it('TaskRunner tryClaimWarmNode catches claim failures and tries next', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function tryClaimWarmNode('), - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(section).toContain('} catch {'); - }); - - it('TaskRunner tryClaimWarmNode returns null if no warm node claimed', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function tryClaimWarmNode('), - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(section).toContain('return null'); - }); - }); - - describe('NodeLifecycle service wrapper', () => { - it('service.tryClaim uses idFromName for deterministic DO mapping', () => { - expect(serviceSource).toContain('env.NODE_LIFECYCLE.idFromName(nodeId)'); - }); - - it('service.tryClaim forwards taskId to DO stub', () => { - expect(serviceSource).toContain('stub.tryClaim(taskId)'); - }); - - it('selectNodeForTaskRun uses service.tryClaim (not direct DO access)', () => { - expect(selectorSource).toContain('nodeLifecycle.tryClaim'); - expect(selectorSource).toContain("import * as nodeLifecycle from './node-lifecycle'"); - }); - }); -}); - -// ============================================================================= -// End-to-end node selection flow: selection -> provisioning wiring -// ============================================================================= - -describe('node selection to provisioning flow wiring', () => { - it('selectNodeForTaskRun returns null when no node available (triggers provisioning)', () => { - // selectNodeForTaskRun returns null in two places - const nullReturns = selectorSource.match(/return null/g); - expect(nullReturns).not.toBeNull(); - expect(nullReturns!.length).toBeGreaterThanOrEqual(2); // zero nodes, no capacity - }); - - it('TaskRunner handleNodeSelection falls through to provisioning on null', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('export async function handleNodeSelection('), - taskRunnerSource.indexOf('export async function handleNodeProvisioning(') - ); - // When no node found, advance to provisioning - expect(section).toContain("advanceToStep(state, 'node_provisioning')"); - }); - - it('TaskRunner handleNodeSelection tries warm pool before capacity', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('export async function handleNodeSelection('), - taskRunnerSource.indexOf('export async function handleNodeProvisioning(') - ); - const warmIdx = section.indexOf('tryClaimWarmNode'); - const capacityIdx = section.indexOf('findNodeWithCapacity'); - expect(warmIdx).toBeGreaterThan(-1); - expect(capacityIdx).toBeGreaterThan(warmIdx); - }); - - it('TaskRunner handleNodeSelection checks preferred node before warm pool', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('export async function handleNodeSelection('), - taskRunnerSource.indexOf('export async function handleNodeProvisioning(') - ); - const preferredIdx = section.indexOf('preferredNodeId'); - const warmIdx = section.indexOf('tryClaimWarmNode'); - expect(preferredIdx).toBeGreaterThan(-1); - expect(warmIdx).toBeGreaterThan(preferredIdx); - }); - - it('preferred node check validates status is running', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('export async function handleNodeSelection('), - taskRunnerSource.indexOf('// Try warm pool first') - ); - expect(section).toContain("node.status !== 'running'"); - expect(section).toContain('permanent: true'); - }); - - it('preferred node check validates ownership (user_id match)', () => { - const section = taskRunnerSource.slice( - taskRunnerSource.indexOf('export async function handleNodeSelection('), - taskRunnerSource.indexOf('// Try warm pool first') - ); - expect(section).toContain('user_id = ?'); - expect(section).toContain('state.userId'); +describe('node placement vertical wiring', () => { + it('retains atomic warm claims in NodeLifecycle', () => { + expect(lifecycle).toContain("state.status !== 'warm'"); + expect(lifecycle).toContain("state.status = 'active'"); + expect(lifecycle).toContain('state.claimedByTask = taskId'); }); -}); - -// ============================================================================= -// Capacity scoring consistency between selector and TaskRunner -// ============================================================================= - -describe('capacity scoring consistency', () => { - it('both selector and TaskRunner use same 0.4/0.6 weighting', () => { - // node-selector.ts - expect(selectorSource).toContain('cpu * 0.4 + memory * 0.6'); - // task-runner.ts findNodeWithCapacity - const trSection = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(trSection).toContain('cpu * 0.4 + mem * 0.6'); + it('has one reusable selector for TaskRunner, trials, and manual placement', () => { + expect(taskRunner).toContain('selectNodeWithExplanation('); + expect(trial).toContain('selectNodeWithExplanation('); + expect(manual).toContain('selectNodeWithExplanation('); }); - it('both use same location-first then size-then-load sorting order', () => { - // node-selector.ts - const selectorSort = selectorSource.slice( - selectorSource.indexOf('Sort candidates'), - selectorSource.indexOf('const best = candidates[0]') - ); - expect(selectorSort).toContain('aLocationMatch'); - - // task-runner.ts - const trSort = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function findNodeWithCapacity('), - taskRunnerSource.indexOf( - '// ====', - taskRunnerSource.indexOf('async function findNodeWithCapacity(') + 100 - ) - ); - expect(trSort).toContain('aLoc'); - expect(trSort).toContain('aSize'); + it('retains the exact compatibility predicate in the canonical evaluator', () => { + expect(evaluator).toContain('isNodeAgentVersionCompatible('); + expect(selector).not.toContain('agentVersion ==='); }); - it('both skip unhealthy nodes', () => { - expect(selectorSource).toContain("node.healthStatus === 'unhealthy'"); - const trSection = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function findNodeWithCapacity(') + it('persists TaskRunner decisions before provisioning and copies them to workspaces', () => { + const selectIndex = taskRunner.indexOf( + 'persistTaskPlacement(state, rc, placement.explanation)' ); - expect(trSection).toContain("health_status != 'unhealthy'"); + const provisionIndex = taskRunner.indexOf("advanceToStep(state, 'node_provisioning')"); + expect(selectIndex).toBeGreaterThan(-1); + expect(provisionIndex).toBeGreaterThan(selectIndex); + expect(taskRunner).toContain('placementExplanationJson: state.placementExplanation'); }); - it('both enforce hard workspace count limit (MAX_WORKSPACES_PER_NODE)', () => { - // node-selector.ts - expect(selectorSource).toContain('activeCount >= maxWorkspacesPerNode'); - - // task-runner.ts - const trSection = taskRunnerSource.slice( - taskRunnerSource.indexOf('async function findNodeWithCapacity(') - ); - expect(trSection).toContain('>= maxWorkspaces'); + it('persists trial and manual decisions, including failure updates', () => { + expect(trial).toContain('UPDATE trials SET placement_explanation_json = ?'); + expect(trial).toContain('recordTrialPlacementFailure'); + expect(trial).toContain('placementExplanationJson: state.placementExplanation'); + expect(manual).toContain('placementExplanationJson: JSON.stringify(placementExplanation)'); + expect(manual).toContain("failureReason: 'readiness-timeout'"); }); - it('both use DEFAULT_MAX_WORKSPACES_PER_NODE as fallback', () => { - expect(selectorSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); - expect(taskRunnerSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); + it('logs only the allowlisted placement object at decision boundaries', () => { + expect(taskRunner).toContain("log.info('task_runner_do.placement_decided'"); + expect(trial).toContain("log.info('trial_orchestrator_do.placement_decided'"); + expect(evaluator).not.toContain('providerError'); + expect(evaluator).not.toContain('repository'); }); }); diff --git a/apps/api/tests/integration/task-runner-placement-vertical-slice.test.ts b/apps/api/tests/integration/task-runner-placement-vertical-slice.test.ts new file mode 100644 index 0000000000..a8825bfe5d --- /dev/null +++ b/apps/api/tests/integration/task-runner-placement-vertical-slice.test.ts @@ -0,0 +1,310 @@ +/** + * Behavioral placement persistence coverage for the TaskRunner production path. + * + * The D1 boundary is backed by real in-memory SQLite. Node selection, provisioning + * state transitions, and workspace creation all run through their production + * handlers; only external Durable Object scheduling/storage is substituted. + */ +import Database from 'better-sqlite3'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import * as schema from '../../src/db/schema'; +import { + handleNodeAgentReady, + handleNodeProvisioning, + handleNodeSelection, +} from '../../src/durable-objects/task-runner/node-steps'; +import { recordProvisioningAttempt } from '../../src/durable-objects/task-runner/placement'; +import type { + TaskRunnerContext, + TaskRunnerState, +} from '../../src/durable-objects/task-runner/types'; +import { handleWorkspaceCreation } from '../../src/durable-objects/task-runner/workspace-steps'; +import type { Env } from '../../src/env'; +import { createAllSchemaTables, createSqliteD1 } from '../helpers/sqlite-d1'; + +const REQUIRED_AGENT_VERSION = 'a'.repeat(40); +const REUSED_NODE_ID = '01KZR2JAP92AK3SKW951E4H21M'; +const PROVISIONED_NODE_ID = '01KZRHQ4PVD1V55YP18H3BWKBF'; + +type PlacementRow = { + placement_explanation_json: string | null; +}; + +let sqlite: Database.Database | null = null; + +function createDatabase(): Env { + sqlite = new Database(':memory:'); + createAllSchemaTables(sqlite, schema); + return { + DATABASE: createSqliteD1(sqlite), + MAX_WORKSPACES_PER_NODE: '5', + NODE_HEARTBEAT_STALE_SECONDS: '180', + VM_AGENT_REQUIRED_VERSION: REQUIRED_AGENT_VERSION, + } as Env; +} + +function seedTask(): void { + const now = new Date().toISOString(); + sqlite + ?.prepare( + `INSERT INTO tasks + (id, project_id, user_id, title, status, task_mode, created_by, created_at, updated_at) + VALUES ('task-1', 'project-1', 'user-1', 'Placement task', 'queued', 'task', + 'user-1', ?, ?)` + ) + .run(now, now); +} + +function seedNode(id: string, status: 'creating' | 'running'): void { + const now = new Date().toISOString(); + sqlite + ?.prepare( + `INSERT INTO nodes + (id, user_id, name, status, vm_size, vm_location, runtime, node_role, + node_mode, health_status, heartbeat_stale_after_seconds, + last_heartbeat_at, agent_ready_at, agent_version, last_metrics, + credential_source, created_at, updated_at) + VALUES (?, 'user-1', 'Placement node', ?, 'medium', 'hel1', 'vm', 'workspace', + 'shared', 'healthy', 180, ?, ?, ?, ?, 'user', ?, ?)` + ) + .run( + id, + status, + now, + now, + REQUIRED_AGENT_VERSION, + JSON.stringify({ cpuLoadAvg1: 2, memoryPercent: 10 }), + now, + now + ); +} + +function createState(): TaskRunnerState { + return { + version: 1, + taskId: 'task-1', + projectId: 'project-1', + userId: 'user-1', + currentStep: 'node_selection', + stepResults: { + nodeId: null, + autoProvisioned: false, + workspaceId: null, + chatSessionId: null, + agentSessionId: null, + agentStarted: false, + mcpToken: null, + provisionedVmSize: null, + }, + config: { + vmSize: 'medium', + vmLocation: 'hel1', + branch: 'main', + preferredNodeId: null, + userName: 'Test User', + userEmail: 'user-1@example.test', + githubId: null, + taskTitle: 'Placement task', + taskDescription: null, + repository: 'owner/repository', + installationId: 'installation-1', + outputBranch: null, + defaultBranch: 'main', + projectDefaultVmSize: null, + chatSessionId: null, + agentType: null, + workspaceProfile: null, + devcontainerConfigName: null, + cloudProvider: null, + credentialAttributionUserId: 'user-1', + credentialAttributionProjectId: null, + credentialAttributionSource: 'user', + taskMode: 'task', + model: null, + effort: null, + permissionMode: null, + opencodeProvider: null, + opencodeBaseUrl: null, + systemPromptAppend: null, + agentProfileHint: null, + attachments: null, + projectScaling: null, + }, + retryCount: 0, + workspaceReadyReceived: false, + workspaceReadyStatus: null, + workspaceErrorMessage: null, + createdAt: Date.now(), + lastStepAt: Date.now(), + provisioningStartedAt: null, + agentReadyStartedAt: null, + workspaceReadyStartedAt: null, + workspaceDispatchStartedAt: null, + workspaceDispatchAttempts: 0, + workspaceDispatchLastAttemptAt: null, + workspaceDispatchLastError: null, + workspaceDispatchAckedAt: null, + lastD1Step: null, + completed: false, + }; +} + +function parsePlacement(table: 'tasks' | 'workspaces'): Record { + const row = sqlite?.prepare(`SELECT placement_explanation_json FROM ${table} LIMIT 1`).get() as + | PlacementRow + | undefined; + expect(row?.placement_explanation_json).toBeTypeOf('string'); + return JSON.parse(row?.placement_explanation_json ?? '{}') as Record; +} + +function createContext( + env: Env, + advances: string[], + provisioningTimeoutMs = 15 * 60 * 1000 +): TaskRunnerContext { + return { + env, + ctx: { + storage: { + get: vi.fn().mockResolvedValue(null), + put: vi.fn().mockResolvedValue(undefined), + setAlarm: vi.fn().mockResolvedValue(undefined), + }, + } as unknown as DurableObjectState, + advanceToStep: vi.fn(async (_state, nextStep) => { + if (nextStep === 'workspace_creation') { + // Selection or provisioning evidence must be durable before the runner + // can proceed to workspace creation. + expect(parsePlacement('tasks')).toMatchObject({ schemaVersion: 2 }); + } + if (nextStep === 'workspace_dispatch') { + // Workspace evidence must exist before dispatch can be scheduled. + expect(parsePlacement('workspaces')).toEqual(parsePlacement('tasks')); + } + advances.push(nextStep); + }), + getAgentPollIntervalMs: () => 1_000, + getAgentReadyTimeoutMs: () => 15 * 60 * 1000, + getWorkspaceDispatchTimeoutMs: () => 15 * 60 * 1000, + getWorkspaceDispatchBaseDelayMs: () => 1_000, + getWorkspaceDispatchMaxDelayMs: () => 30_000, + getWorkspaceReadyTimeoutMs: () => 30 * 60 * 1000, + getWorkspaceReadyPollIntervalMs: () => 1_000, + getProvisionPollIntervalMs: () => 1_000, + getProvisionTimeoutMs: () => provisioningTimeoutMs, + updateD1ExecutionStep: vi.fn().mockResolvedValue(undefined), + }; +} + +afterEach(() => { + sqlite?.close(); + sqlite = null; +}); + +describe('TaskRunner placement persistence vertical slice', () => { + it('persists a reused-node decision to the task before workspace creation and dispatch', async () => { + const env = createDatabase(); + seedTask(); + seedNode(REUSED_NODE_ID, 'running'); + const state = createState(); + const advances: string[] = []; + const rc = createContext(env, advances); + + await handleNodeSelection(state, rc); + + expect(state.stepResults.nodeId).toBe(REUSED_NODE_ID); + expect(parsePlacement('tasks')).toMatchObject({ + schemaVersion: 2, + outcome: 'reused', + selectedNodeId: REUSED_NODE_ID, + evaluatedNodes: [{ nodeId: REUSED_NODE_ID, accepted: true, rejectionReasons: [] }], + }); + + await handleWorkspaceCreation(state, rc); + + expect(parsePlacement('workspaces')).toEqual(parsePlacement('tasks')); + expect(sqlite?.prepare(`SELECT status FROM tasks WHERE id = 'task-1'`).get()).toEqual({ + status: 'delegated', + }); + expect(advances).toEqual(['workspace_creation', 'workspace_dispatch']); + }); + + it('carries finalized provision-new evidence from selection through workspace dispatch', async () => { + const env = createDatabase(); + seedTask(); + const state = createState(); + const advances: string[] = []; + const rc = createContext(env, advances); + + await handleNodeSelection(state, rc); + expect(parsePlacement('tasks')).toMatchObject({ + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: null, + }); + + seedNode(PROVISIONED_NODE_ID, 'running'); + state.stepResults.nodeId = PROVISIONED_NODE_ID; + state.stepResults.autoProvisioned = true; + state.provisioningStartedAt = Date.now(); + await recordProvisioningAttempt( + state, + rc, + { vmSize: 'medium', vmLocation: 'hel1', outcome: 'started' }, + PROVISIONED_NODE_ID + ); + await handleNodeProvisioning(state, rc); + await handleNodeAgentReady(state, rc); + await handleWorkspaceCreation(state, rc); + + expect(parsePlacement('tasks')).toMatchObject({ + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: PROVISIONED_NODE_ID, + provisioningAttempts: [{ outcome: 'started' }, { outcome: 'succeeded' }], + }); + expect(parsePlacement('workspaces')).toEqual(parsePlacement('tasks')); + expect(advances).toEqual([ + 'node_provisioning', + 'node_agent_ready', + 'workspace_creation', + 'workspace_dispatch', + ]); + }); + + it('persists a typed terminal provisioning failure without creating a workspace', async () => { + const env = createDatabase(); + seedTask(); + const state = createState(); + const advances: string[] = []; + const rc = createContext(env, advances, 1_000); + + await handleNodeSelection(state, rc); + seedNode(PROVISIONED_NODE_ID, 'creating'); + state.stepResults.nodeId = PROVISIONED_NODE_ID; + state.stepResults.autoProvisioned = true; + state.provisioningStartedAt = Date.now() - 2_000; + await recordProvisioningAttempt( + state, + rc, + { vmSize: 'medium', vmLocation: 'hel1', outcome: 'started' }, + PROVISIONED_NODE_ID + ); + + await expect(handleNodeProvisioning(state, rc)).rejects.toMatchObject({ permanent: true }); + + expect(parsePlacement('tasks')).toMatchObject({ + schemaVersion: 2, + outcome: 'failed', + selectedNodeId: PROVISIONED_NODE_ID, + provisioningAttempts: [ + { outcome: 'started' }, + { outcome: 'failed', failureReason: 'provisioning-timeout' }, + ], + }); + expect(sqlite?.prepare(`SELECT COUNT(*) AS count FROM workspaces`).get()).toEqual({ count: 0 }); + expect(advances).toEqual(['node_provisioning']); + }); +}); diff --git a/apps/api/tests/integration/warm-node-pooling.test.ts b/apps/api/tests/integration/warm-node-pooling.test.ts index e4af9059ae..a405a2cf43 100644 --- a/apps/api/tests/integration/warm-node-pooling.test.ts +++ b/apps/api/tests/integration/warm-node-pooling.test.ts @@ -14,14 +14,29 @@ import { resolve } from 'node:path'; import { describe, expect, it } from 'vitest'; describe('warm node pooling lifecycle integration', () => { - const taskRunnerFile = readFileSync(resolve(process.cwd(), 'src/services/task-runner.ts'), 'utf8'); - const selectorFile = readFileSync(resolve(process.cwd(), 'src/services/node-selector.ts'), 'utf8'); - const doFile = readFileSync(resolve(process.cwd(), 'src/durable-objects/node-lifecycle.ts'), 'utf8'); + const taskRunnerFile = readFileSync( + resolve(process.cwd(), 'src/services/task-runner.ts'), + 'utf8' + ); + const selectorFile = readFileSync( + resolve(process.cwd(), 'src/services/node-selector.ts'), + 'utf8' + ); + const doFile = readFileSync( + resolve(process.cwd(), 'src/durable-objects/node-lifecycle.ts'), + 'utf8' + ); const cleanupFile = ['index.ts', 'shared.ts', 'node-phases.ts', 'workspace-phases.ts'] .map((f) => readFileSync(resolve(process.cwd(), `src/scheduled/node-cleanup/${f}`), 'utf8')) .join('\n'); - const serviceFile = readFileSync(resolve(process.cwd(), 'src/services/node-lifecycle.ts'), 'utf8'); - const constantsFile = readFileSync(resolve(process.cwd(), '../../packages/shared/src/constants/node-pooling.ts'), 'utf8'); + const serviceFile = readFileSync( + resolve(process.cwd(), 'src/services/node-lifecycle.ts'), + 'utf8' + ); + const constantsFile = readFileSync( + resolve(process.cwd(), '../../packages/shared/src/constants/node-pooling.ts'), + 'utf8' + ); describe('flow: task complete → workspace destroyed → node warm', () => { it('cleanupTaskRun calls cleanupAutoProvisionedNode', () => { @@ -51,7 +66,8 @@ describe('warm node pooling lifecycle integration', () => { describe('flow: new task → claim warm node → fast startup', () => { it('selectNodeForTaskRun queries warm nodes in D1', () => { - expect(selectorFile).toContain('isNotNull(schema.nodes.warmSince)'); + expect(selectorFile).toContain('warmSince: schema.nodes.warmSince'); + expect(selectorFile).toContain("evaluatePlacementNode(node, request, 'warm'"); }); it('selectNodeForTaskRun calls nodeLifecycle.tryClaim', () => { diff --git a/apps/api/tests/unit/db/trial-placement-migration.test.ts b/apps/api/tests/unit/db/trial-placement-migration.test.ts new file mode 100644 index 0000000000..490a0755eb --- /dev/null +++ b/apps/api/tests/unit/db/trial-placement-migration.test.ts @@ -0,0 +1,18 @@ +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import * as schema from '../../../src/db/schema'; + +describe('trial placement migration', () => { + it('adds one nullable JSON audit column without rewriting trial rows', () => { + const sql = readFileSync( + resolve(process.cwd(), 'src/db/migrations/0110_trial_placement_explanation.sql'), + 'utf8' + ); + expect(sql).toContain('ALTER TABLE trials ADD COLUMN placement_explanation_json TEXT;'); + expect(sql).not.toMatch(/DROP|DELETE|UPDATE/i); + expect(schema.trials.placementExplanationJson).toBeDefined(); + }); +}); diff --git a/apps/api/tests/unit/durable-objects/placement-persistence.test.ts b/apps/api/tests/unit/durable-objects/placement-persistence.test.ts new file mode 100644 index 0000000000..6359e4e7e6 --- /dev/null +++ b/apps/api/tests/unit/durable-objects/placement-persistence.test.ts @@ -0,0 +1,126 @@ +import type { PlacementExplanation } from '@simple-agent-manager/shared'; +import { describe, expect, it, vi } from 'vitest'; + +import { + persistTaskPlacement, + recordPlacementFailure, + recordProvisioningAttempt, +} from '../../../src/durable-objects/task-runner/placement'; +import { + persistTrialPlacement, + recordTrialPlacementFailure, + recordTrialProvisioningAttempt, +} from '../../../src/durable-objects/trial-orchestrator/placement'; + +function explanation( + outcome: PlacementExplanation['outcome'] = 'provisioned' +): PlacementExplanation { + return { + schemaVersion: 2, + outcome, + selectionPath: outcome === 'reused' ? 'capacity' : 'provisioning', + selectedNodeId: outcome === 'reused' ? 'node-reused' : null, + summary: outcome === 'reused' ? 'Reused node-reused.' : 'Provisioning a new node.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + }; +} + +function persistenceContext() { + const writes: Array<{ sql: string; values: unknown[] }> = []; + const storagePut = vi.fn(); + const database = { + prepare(sql: string) { + return { + bind(...values: unknown[]) { + return { + async run() { + writes.push({ sql, values }); + return { meta: { changes: 1 } }; + }, + }; + }, + }; + }, + }; + return { + writes, + storagePut, + context: { env: { DATABASE: database }, ctx: { storage: { put: storagePut } } }, + }; +} + +describe('placement persistence vertical slices', () => { + it('persists a reused TaskRunner decision to D1 and durable state', async () => { + const harness = persistenceContext(); + const state = { taskId: 'task-1' }; + const reused = explanation('reused'); + + await persistTaskPlacement(state as never, harness.context as never, reused); + + expect(state).toMatchObject({ placementExplanation: reused }); + expect(harness.writes).toHaveLength(1); + expect(harness.writes[0]?.sql).toContain('UPDATE tasks SET placement_explanation_json'); + expect(JSON.parse(harness.writes[0]?.values[0] as string)).toEqual(reused); + expect(harness.storagePut).toHaveBeenCalledWith('state', state); + }); + + it('appends provisioning success and terminal failure to TaskRunner evidence', async () => { + const harness = persistenceContext(); + const state = { taskId: 'task-2', placementExplanation: explanation() }; + + await recordProvisioningAttempt( + state as never, + harness.context as never, + { vmSize: 'medium', vmLocation: 'hel1', outcome: 'succeeded' }, + 'node-new' + ); + await recordPlacementFailure(state as never, harness.context as never, 'readiness-timeout'); + + expect(state.placementExplanation).toMatchObject({ + outcome: 'failed', + selectedNodeId: 'node-new', + provisioningAttempts: [ + { outcome: 'succeeded' }, + { outcome: 'failed', failureReason: 'readiness-timeout' }, + ], + }); + expect(harness.writes).toHaveLength(2); + }); + + it('persists trial placement before a workspace exists and retains failures', async () => { + const harness = persistenceContext(); + const state = { trialId: 'trial-1' }; + + await persistTrialPlacement(state as never, harness.context as never, explanation()); + await recordTrialProvisioningAttempt(state as never, harness.context as never, { + vmSize: 'large', + vmLocation: 'hel1', + outcome: 'capacity-rejected', + }); + await recordTrialPlacementFailure(state as never, harness.context as never, 'provider-failed'); + + expect(harness.writes[0]?.sql).toContain('UPDATE trials SET placement_explanation_json'); + expect(state).toMatchObject({ + placementExplanation: { + outcome: 'failed', + provisioningAttempts: [ + { outcome: 'capacity-rejected', vmSize: 'large' }, + { outcome: 'failed', failureReason: 'provider-failed' }, + ], + }, + }); + expect(harness.writes).toHaveLength(3); + }); +}); diff --git a/apps/api/tests/unit/durable-objects/task-runner-node-selection.test.ts b/apps/api/tests/unit/durable-objects/task-runner-node-selection.test.ts index 3289ed9e96..bdb8b5e9cf 100644 --- a/apps/api/tests/unit/durable-objects/task-runner-node-selection.test.ts +++ b/apps/api/tests/unit/durable-objects/task-runner-node-selection.test.ts @@ -1,11 +1,16 @@ import { describe, expect, it, vi } from 'vitest'; +import * as schema from '../../../src/db/schema'; import { handleNodeSelection } from '../../../src/durable-objects/task-runner/node-steps'; import type { TaskRunnerContext, TaskRunnerState, } from '../../../src/durable-objects/task-runner/types'; +vi.mock('drizzle-orm/d1', () => ({ + drizzle: (database: unknown) => database, +})); + type D1ResultMap = { preferredNode?: { id: string; @@ -44,36 +49,55 @@ type D1ResultMap = { >; }; -function createStatement(sql: string, results: D1ResultMap) { - let bound: unknown[] = []; +function createDatabase(results: D1ResultMap) { + const now = new Date().toISOString(); + const placementNodes = [ + ...(results.preferredNode ? [results.preferredNode] : []), + ...(results.warmNodes ?? []), + ...(results.existingNodes ?? []), + ].map((candidate) => { + const fresh = results.healthByNode?.[candidate.id]; + return { + id: candidate.id, + status: 'status' in candidate ? candidate.status : 'running', + runtime: 'vm', + vmSize: candidate.vm_size, + vmLocation: 'vm_location' in candidate ? candidate.vm_location : 'fsn1', + healthStatus: + fresh?.health_status ?? + ('health_status' in candidate ? candidate.health_status : 'healthy'), + lastHeartbeatAt: fresh?.last_heartbeat_at ?? now, + agentReadyAt: fresh?.agent_ready_at ?? now, + agentVersion: fresh?.agent_version ?? candidate.agent_version ?? null, + lastMetrics: 'last_metrics' in candidate ? candidate.last_metrics : null, + warmSince: (results.warmNodes ?? []).some((node) => node.id === candidate.id) ? now : null, + }; + }); + const workspaceRows = (results.workspaceCounts ?? []).flatMap((count) => + Array.from({ length: count.c }, (_, index) => ({ + id: `workspace-${count.node_id}-${index}`, + nodeId: count.node_id, + })) + ); + return { - bind(...args: unknown[]) { - bound = args; - return this; - }, - first() { - if (sql.includes('SELECT id, status, vm_size')) { - return Promise.resolve(results.preferredNode ?? null); - } - if (sql.includes('SELECT status, warm_since')) { - return Promise.resolve(results.freshWarmNode ?? null); - } - if (sql.includes('SELECT health_status, last_heartbeat_at, agent_ready_at')) { - return Promise.resolve(results.healthByNode?.[String(bound[0])] ?? null); - } - return Promise.resolve(null); + select() { + return { + from(table: unknown) { + return { + where() { + return Promise.resolve(table === schema.nodes ? placementNodes : workspaceRows); + }, + }; + }, + }; }, - all() { - if (sql.includes('warm_since IS NOT NULL')) { - return Promise.resolve({ results: results.warmNodes ?? [] }); - } - if (sql.includes('SELECT id, vm_size, vm_location, health_status, last_metrics')) { - return Promise.resolve({ results: results.existingNodes ?? [] }); - } - if (sql.includes('SELECT node_id, COUNT(*) as c FROM workspaces')) { - return Promise.resolve({ results: results.workspaceCounts ?? [] }); - } - return Promise.resolve({ results: [] }); + prepare() { + return { + bind() { + return { run: async () => ({ meta: { changes: 1 } }) }; + }, + }; }, }; } @@ -82,9 +106,7 @@ function createContext(results: D1ResultMap): TaskRunnerContext { return { env: { DATABASE: { - prepare(sql: string) { - return createStatement(sql, results); - }, + ...createDatabase(results), }, NODE_HEARTBEAT_STALE_SECONDS: '180', MAX_WORKSPACES_PER_NODE: '5', @@ -93,6 +115,7 @@ function createContext(results: D1ResultMap): TaskRunnerContext { ctx: { storage: { setAlarm: vi.fn(), + put: vi.fn(), }, }, advanceToStep: vi.fn().mockResolvedValue(undefined), diff --git a/apps/api/tests/unit/durable-objects/trial-orchestrator-steps.test.ts b/apps/api/tests/unit/durable-objects/trial-orchestrator-steps.test.ts index 930624259e..55ce89e52f 100644 --- a/apps/api/tests/unit/durable-objects/trial-orchestrator-steps.test.ts +++ b/apps/api/tests/unit/durable-objects/trial-orchestrator-steps.test.ts @@ -27,6 +27,14 @@ const { startDiscoveryAgentMock, emitTrialEventMock } = vi.hoisted(() => ({ startDiscoveryAgentMock: vi.fn(), emitTrialEventMock: vi.fn(async () => {}), })); +const { drizzleMock, signCallbackTokenMock } = vi.hoisted(() => ({ + drizzleMock: vi.fn(), + signCallbackTokenMock: vi.fn(async () => 'callback-token'), +})); +vi.mock('drizzle-orm/d1', () => ({ drizzle: drizzleMock })); +vi.mock('../../../src/services/jwt', () => ({ + signCallbackToken: signCallbackTokenMock, +})); vi.mock('../../../src/services/trial/trial-runner', () => ({ emitTrialEvent: emitTrialEventMock, emitTrialEventForProject: vi.fn(async () => {}), @@ -77,9 +85,14 @@ vi.mock('../../../src/services/limits', () => ({ getRuntimeLimits: vi.fn(() => ({ nodeHeartbeatStaleSeconds: 120 })), })); -const { handleRunning, handleDiscoveryAgentStart, handleNodeProvisioning, handleNodeAgentReady } = await import( - '../../../src/durable-objects/trial-orchestrator/steps' -); +const { + handleRunning, + handleDiscoveryAgentStart, + handleNodeSelection, + handleNodeProvisioning, + handleNodeAgentReady, + handleWorkspaceCreation, +} = await import('../../../src/durable-objects/trial-orchestrator/steps'); type Storage = Map; @@ -149,12 +162,15 @@ function makeRc(ctx: ReturnType, advanced: string[]) { getNodeReadyTimeoutMs: () => 180_000, getHeartbeatSkewMs: () => 30_000, _dbFirst: firstMock, + _dbBind: bindMock, } as unknown as Parameters[1]; } describe('handleRunning', () => { beforeEach(() => { vi.clearAllMocks(); + createNodeRecordMock.mockResolvedValue({ id: 'node_new_123' }); + provisionNodeMock.mockResolvedValue(undefined); }); it('marks state.completed = true and persists', async () => { @@ -199,6 +215,100 @@ describe('handleNodeProvisioning', () => { expect(state.autoProvisionedNode).toBe(true); expect(advanced).toEqual(['node_agent_ready']); }); + + it('persists a typed terminal provider failure from the handler boundary', async () => { + const ctx = makeCtx(); + const rc = makeRc(ctx, []) as Parameters[1] & { + _dbFirst: ReturnType; + }; + const decidedAt = new Date().toISOString(); + const state = makeState({ + currentStep: 'node_provisioning', + nodeId: 'node_failed', + placementExplanation: { + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: 'node_failed', + summary: 'Provisioning a new node.', + request: { + runtime: 'vm', + vmSize: 'small', + vmLocation: 'fsn1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 80, + memoryThresholdPercent: 85, + heartbeatStaleSeconds: 120, + }, + evaluatedNodes: [], + provisioningAttempts: [{ vmSize: 'small', vmLocation: 'fsn1', outcome: 'started' }], + decidedAt, + updatedAt: decidedAt, + }, + }); + rc._dbFirst.mockResolvedValue({ status: 'error', error_message: 'provider detail' }); + + await expect(handleNodeProvisioning(state, rc)).rejects.toThrow('provider detail'); + expect(state.placementExplanation.provisioningAttempts.at(-1)).toMatchObject({ + outcome: 'failed', + failureReason: 'provider-failed', + }); + }); +}); + +describe('handleNodeSelection', () => { + it('reuses a compatible D1 candidate and persists the selected explanation', async () => { + const requiredVersion = 'a'.repeat(40); + const queryResults: unknown[][] = [ + [ + { + id: '01KZR2JAP92AK3SKW951E4H21M', + status: 'running', + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + lastHeartbeatAt: new Date().toISOString(), + agentReadyAt: new Date().toISOString(), + agentVersion: requiredVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 2, memoryPercent: 10 }), + warmSince: null, + }, + ], + [], + ]; + drizzleMock.mockReturnValue({ + select: vi.fn(() => { + const rows = queryResults.shift() ?? []; + return { from: vi.fn(() => ({ where: vi.fn(async () => rows) })) }; + }), + }); + const ctx = makeCtx(); + const advanced: string[] = []; + const rc = makeRc(ctx, advanced) as Parameters[1] & { + _dbBind: ReturnType; + }; + Object.assign(rc.env, { + TRIAL_VM_SIZE: 'medium', + TRIAL_VM_LOCATION: 'hel1', + VM_AGENT_REQUIRED_VERSION: requiredVersion, + }); + const state = makeState({ currentStep: 'node_selection', projectId: 'project-1' }); + + await handleNodeSelection(state, rc); + + expect(state.nodeId).toBe('01KZR2JAP92AK3SKW951E4H21M'); + expect(state.placementExplanation).toMatchObject({ + outcome: 'reused', + selectionPath: 'trial', + selectedNodeId: '01KZR2JAP92AK3SKW951E4H21M', + }); + expect(rc._dbBind).toHaveBeenCalledWith( + expect.stringContaining('"schemaVersion":2'), + state.trialId + ); + expect(advanced).toEqual(['workspace_creation']); + }); }); describe('handleNodeAgentReady', () => { @@ -248,11 +358,105 @@ describe('handleNodeAgentReady', () => { last_heartbeat_at: new Date(waitStartedAt + 2_000).toISOString(), agent_ready_at: new Date(waitStartedAt + 1_000).toISOString(), }); + const decidedAt = new Date(waitStartedAt - 1_000).toISOString(); + state.placementExplanation = { + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: null, + summary: 'No reusable node was eligible; provisioning is required.', + request: { + runtime: 'vm', + vmSize: 'small', + vmLocation: 'fsn1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 80, + memoryThresholdPercent: 85, + heartbeatStaleSeconds: 120, + }, + evaluatedNodes: [], + provisioningAttempts: [{ vmSize: 'small', vmLocation: 'fsn1', outcome: 'started' }], + decidedAt, + updatedAt: decidedAt, + }; await handleNodeAgentReady(state, rc); expect(advanced).toEqual(['workspace_creation']); expect(ctx.storage.setAlarm).not.toHaveBeenCalled(); + expect(state.placementExplanation.provisioningAttempts).toEqual([ + { vmSize: 'small', vmLocation: 'fsn1', outcome: 'started' }, + { vmSize: 'small', vmLocation: 'fsn1', outcome: 'succeeded' }, + ]); + }); + + it('carries a successful provision explanation through the workspace D1 insert', async () => { + const ctx = makeCtx(); + const advanced: string[] = []; + const rc = makeRc(ctx, advanced) as Parameters[1] & { + _dbFirst: ReturnType; + }; + const decidedAt = new Date(Date.now() - 10_000).toISOString(); + const state = makeState({ + currentStep: 'node_provisioning', + projectId: 'project-1', + defaultBranch: 'main', + placementExplanation: { + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: null, + summary: 'No reusable node qualified; provisioning a new node.', + request: { + runtime: 'vm', + vmSize: 'small', + vmLocation: 'fsn1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 80, + memoryThresholdPercent: 85, + heartbeatStaleSeconds: 120, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt, + updatedAt: decidedAt, + }, + }); + + await handleNodeProvisioning(state, rc); + expect(state.placementExplanation.provisioningAttempts.at(-1)?.outcome).toBe('started'); + + const waitStartedAt = Date.now() - 5_000; + state.nodeAgentReadyStartedAt = waitStartedAt; + rc._dbFirst.mockResolvedValue({ + status: 'running', + health_status: 'healthy', + last_heartbeat_at: new Date(waitStartedAt + 2_000).toISOString(), + agent_ready_at: new Date(waitStartedAt + 1_000).toISOString(), + agent_version: null, + }); + await handleNodeAgentReady(state, rc); + expect(state.placementExplanation.provisioningAttempts.at(-1)?.outcome).toBe('succeeded'); + + const inserted: Array> = []; + drizzleMock.mockReturnValue({ + select: vi.fn(() => ({ + from: vi.fn(() => ({ where: vi.fn(async () => []) })), + })), + insert: vi.fn(() => ({ + values: vi.fn(async (value: Record) => { + inserted.push(value); + }), + })), + }); + await handleWorkspaceCreation(state, rc as never); + + expect(JSON.parse(inserted[0]?.placementExplanationJson as string)).toMatchObject({ + outcome: 'provisioned', + selectedNodeId: 'node_new_123', + provisioningAttempts: [{ outcome: 'started' }, { outcome: 'succeeded' }], + }); + expect(advanced).toEqual(['node_agent_ready', 'workspace_creation', 'workspace_ready']); }); }); diff --git a/apps/api/tests/unit/lib/placement-mappers.test.ts b/apps/api/tests/unit/lib/placement-mappers.test.ts new file mode 100644 index 0000000000..e32bb70937 --- /dev/null +++ b/apps/api/tests/unit/lib/placement-mappers.test.ts @@ -0,0 +1,108 @@ +import type { PlacementExplanation } from '@simple-agent-manager/shared'; +import { describe, expect, it } from 'vitest'; + +import type * as schema from '../../../src/db/schema'; +import { toTaskResponse, toWorkspaceResponse } from '../../../src/lib/mappers'; + +const placement: PlacementExplanation = { + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: 'node-1', + summary: 'Reused node node-1 through the capacity path.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', +}; + +describe('placement response mappers', () => { + it('returns parsed workspace placement and rejects malformed JSON', () => { + const base = { + id: 'workspace-1', + nodeId: 'node-1', + projectId: 'project-1', + userId: 'user-1', + installationId: 'installation-1', + displayName: 'Workspace', + name: 'Workspace', + repository: 'owner/repo', + branch: 'main', + status: 'running', + vmSize: 'medium', + vmLocation: 'hel1', + workspaceProfile: 'full', + devcontainerConfigName: null, + vmIp: null, + lastActivityAt: null, + portsPublicEnabled: false, + errorMessage: null, + createdAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + chatSessionId: null, + placementExplanationJson: JSON.stringify(placement), + } as unknown as schema.Workspace; + + expect(toWorkspaceResponse(base, 'example.test').placementExplanation).toEqual(placement); + expect( + toWorkspaceResponse( + { ...base, placementExplanationJson: '{bad json' } as schema.Workspace, + 'example.test' + ).placementExplanation + ).toBeNull(); + }); + + it('retains raw task JSON while adding a safely parsed field', () => { + const raw = JSON.stringify(placement); + const task = { + id: 'task-1', + projectId: 'project-1', + userId: 'user-1', + parentTaskId: null, + workspaceId: null, + title: 'Task', + description: null, + status: 'queued', + executionStep: 'node_selection', + priority: 0, + taskMode: 'task', + dispatchDepth: 0, + agentProfileHint: null, + skillId: null, + skillHint: null, + triggeredBy: 'user', + triggerId: null, + triggerExecutionId: null, + requestedVmSize: null, + requestedVmSizeSource: null, + provisionedVmSize: null, + resourceRequirementsJson: null, + resourceRequirementsSource: null, + resolvedReservationJson: null, + placementExplanationJson: raw, + startedAt: null, + completedAt: null, + errorMessage: null, + outputSummary: null, + outputBranch: null, + outputPrUrl: null, + completionEvidence: null, + finalizedAt: null, + createdAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + } as unknown as schema.Task; + + const response = toTaskResponse(task); + expect(response.placementExplanationJson).toBe(raw); + expect(response.placementExplanation).toEqual(placement); + }); +}); diff --git a/apps/api/tests/unit/node-provisioning.test.ts b/apps/api/tests/unit/node-provisioning.test.ts index b418184e0e..965effbc98 100644 --- a/apps/api/tests/unit/node-provisioning.test.ts +++ b/apps/api/tests/unit/node-provisioning.test.ts @@ -14,7 +14,7 @@ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; -import { describe, expect,it } from 'vitest'; +import { describe, expect, it } from 'vitest'; import { parseEnvInt } from '../../src/durable-objects/task-runner/helpers'; @@ -34,6 +34,10 @@ const indexSource = readFileSync( resolve(process.cwd(), 'src/env.ts'), 'utf8' ); +const provisioningGuardsSource = readFileSync( + resolve(doDir, 'provisioning-guards.ts'), + 'utf8' +); // ============================================================================= // Node Limit Enforcement @@ -45,21 +49,15 @@ describe('node limit enforcement', () => { expect(indexSource).toContain("MAX_NODES_PER_USER?: string"); }); - it('handleNodeProvisioning reads MAX_NODES_PER_USER via parseEnvInt', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') - ); - expect(section).toContain('MAX_NODES_PER_USER'); - expect(section).toContain('parseEnvInt'); + it('the provisioning guard reads MAX_NODES_PER_USER via parseEnvInt', () => { + expect(provisioningGuardsSource).toContain('MAX_NODES_PER_USER'); + expect(provisioningGuardsSource).toContain('parseEnvInt'); }); - it('defaults to 10 when env var is not set', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') + it('uses the shared default when the env var is not set', () => { + expect(provisioningGuardsSource).toContain( + 'parseEnvInt(rc.env.MAX_NODES_PER_USER, DEFAULT_MAX_NODES_PER_USER)' ); - expect(section).toContain('parseEnvInt(rc.env.MAX_NODES_PER_USER, 10)'); }); }); @@ -93,51 +91,33 @@ describe('node limit enforcement', () => { }); }); - describe('limit check in handleNodeProvisioning', () => { + describe('limit check in the provisioning guard', () => { it('queries node count for the user from D1', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') + expect(provisioningGuardsSource).toContain( + "SELECT COUNT(*) as c FROM nodes WHERE user_id = ? AND status IN ('running', 'creating', 'recovery')" ); - expect(section).toContain("SELECT COUNT(*) as c FROM nodes WHERE user_id = ? AND status IN ('running', 'creating', 'recovery')"); }); it('only counts active nodes (excludes deleted/stopped) in limit check', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') - ); // Must filter by active statuses to avoid false limit hits from deleted/stopped nodes. // See: 2026-03-09-fix-node-workspace-limit-count-filters - expect(section).toContain("status IN ('running', 'creating', 'recovery')"); + expect(provisioningGuardsSource).toContain("status IN ('running', 'creating', 'recovery')"); }); it('throws permanent error when at or over limit', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') - ); - expect(section).toContain('>= maxNodes'); - expect(section).toContain('Cannot auto-provision'); - expect(section).toContain('permanent: true'); + expect(provisioningGuardsSource).toContain('>= maxNodes'); + expect(provisioningGuardsSource).toContain('Cannot auto-provision'); + expect(provisioningGuardsSource).toContain('permanent: true'); }); it('error message includes the actual limit value', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') - ); - expect(section).toContain('`Maximum ${maxNodes} nodes allowed'); + expect(provisioningGuardsSource).toContain('`Maximum ${maxNodes} nodes allowed'); }); it('uses >= comparison (at limit = rejected)', () => { - const section = doSource.slice( - doSource.indexOf('export async function handleNodeProvisioning('), - doSource.indexOf('export async function handleNodeAgentReady(') - ); // Verify it uses >= not > - expect(section).toContain('>= maxNodes'); - expect(section).not.toContain('> maxNodes'); + expect(provisioningGuardsSource).toContain('>= maxNodes'); + expect(provisioningGuardsSource).not.toContain('> maxNodes'); }); }); }); diff --git a/apps/api/tests/unit/node-role-exemption.test.ts b/apps/api/tests/unit/node-role-exemption.test.ts index 2a22ad8767..dbcf444394 100644 --- a/apps/api/tests/unit/node-role-exemption.test.ts +++ b/apps/api/tests/unit/node-role-exemption.test.ts @@ -18,9 +18,7 @@ import { describe, expect, it, vi } from 'vitest'; const SRC_DIR = path.resolve(__dirname, '../../src'); import * as schema from '../../src/db/schema'; -import { - selectNodeForTaskRun, -} from '../../src/services/node-selector'; +import { selectNodeForTaskRun } from '../../src/services/node-selector'; vi.mock('../../src/services/node-lifecycle', () => ({ tryClaim: vi.fn(), @@ -38,6 +36,10 @@ type MockNode = { vmSize: string; vmLocation: string; nodeRole: string; + runtime: string; + lastHeartbeatAt: string; + agentReadyAt: string; + agentVersion: string | null; lastMetrics: string | null; warmSince?: string | null; }; @@ -51,6 +53,10 @@ function makeNode(overrides: Partial = {}): MockNode { vmSize: 'medium', vmLocation: 'fsn1', nodeRole: 'workspace', + runtime: 'vm', + lastHeartbeatAt: new Date().toISOString(), + agentReadyAt: new Date().toISOString(), + agentVersion: null, lastMetrics: JSON.stringify({ cpuLoadAvg1: 5, memoryPercent: 10 }), warmSince: null, ...overrides, @@ -69,36 +75,25 @@ function createMockDb({ workspaceCount?: number; }) { return { - select(selection?: Record) { + select(_selection?: Record) { return { from(table: unknown) { return { where(..._args: unknown[]) { if (table === schema.workspaces) { - return Promise.resolve([{ count: workspaceCount }]); + return Promise.resolve( + Array.from({ length: workspaceCount }, (_, index) => ({ + id: `workspace-${index}`, + nodeId: allNodes.find((node) => node.nodeRole === 'workspace')?.id ?? null, + })) + ); } if (table === schema.nodes) { - // For warm-node freshness re-checks (select by ID with limit) - if (selection && 'warmSince' in selection && 'status' in selection) { - return { - limit() { - return Promise.resolve([{ status: 'running', warmSince: new Date().toISOString() }]); - }, - }; - } - // The real Drizzle queries include eq(schema.nodes.nodeRole, 'workspace'). // Filter the mock data the same way the DB would. const filtered = allNodes.filter((n) => n.nodeRole === 'workspace'); - - // For warm nodes query (has warmSince in selection) - if (selection && 'warmSince' in selection) { - return Promise.resolve(filtered.filter((n) => n.warmSince)); - } - - // For main node query - return Promise.resolve(filtered.filter((n) => n.status === 'running')); + return Promise.resolve(filtered); } return Promise.resolve([]); @@ -132,11 +127,7 @@ describe('selectNodeForTaskRun — node_role filtering', () => { // Only a deployment node is available const db = createMockDb({ allNodes: [deploymentNode] }); - const result = await selectNodeForTaskRun( - db as any, - 'user-1', - env - ); + const result = await selectNodeForTaskRun(db as any, 'user-1', env); // Should return null — deployment node is not eligible expect(result).toBeNull(); @@ -155,11 +146,7 @@ describe('selectNodeForTaskRun — node_role filtering', () => { const db = createMockDb({ allNodes: [deploymentNode, workspaceNode] }); - const result = await selectNodeForTaskRun( - db as any, - 'user-1', - env - ); + const result = await selectNodeForTaskRun(db as any, 'user-1', env); expect(result).not.toBeNull(); expect(result!.id).toBe('node-ws-1'); @@ -173,11 +160,7 @@ describe('selectNodeForTaskRun — node_role filtering', () => { const db = createMockDb({ allNodes: nodes }); - const result = await selectNodeForTaskRun( - db as any, - 'user-1', - env - ); + const result = await selectNodeForTaskRun(db as any, 'user-1', env); expect(result).toBeNull(); }); @@ -235,13 +218,13 @@ describe('task-runner node-steps — node_role filtering', () => { it('node quota count query excludes deployment nodes', async () => { const fs = await import('fs'); const source = fs.readFileSync( - path.join(SRC_DIR, 'durable-objects/task-runner/node-steps.ts'), + path.join(SRC_DIR, 'durable-objects/task-runner/provisioning-guards.ts'), 'utf-8' ); // The COUNT query for user node limit must include node_role filter const quotaSection = source.slice( - source.indexOf('Check user node limit'), + source.indexOf('const maxNodes'), source.indexOf('.bind(state.userId)') ); expect(quotaSection).toContain("node_role = 'workspace'"); @@ -249,34 +232,18 @@ describe('task-runner node-steps — node_role filtering', () => { it('warm node query excludes deployment nodes', async () => { const fs = await import('fs'); - const source = fs.readFileSync( - path.join(SRC_DIR, 'durable-objects/task-runner/node-selection.ts'), - 'utf-8' - ); + const source = fs.readFileSync(path.join(SRC_DIR, 'services/node-selector.ts'), 'utf-8'); - // The warm node search query must include node_role filter - const warmQueryStart = source.indexOf( - 'SELECT id, vm_size, vm_location, agent_version FROM nodes' - ); - const warmQueryEnd = source.indexOf('.bind(state.userId)', warmQueryStart); - const warmSection = source.slice(warmQueryStart, warmQueryEnd); - expect(warmSection).toContain("node_role = 'workspace'"); + expect(source).toContain("eq(schema.nodes.nodeRole, 'workspace')"); + expect(source).toContain("evaluatePlacementNode(node, request, 'warm'"); }); it('fallback node selection query excludes deployment nodes', async () => { const fs = await import('fs'); - const source = fs.readFileSync( - path.join(SRC_DIR, 'durable-objects/task-runner/node-selection.ts'), - 'utf-8' - ); + const source = fs.readFileSync(path.join(SRC_DIR, 'services/node-selector.ts'), 'utf-8'); - // The fallback "find existing running node" query must include node_role filter - const fallbackQueryStart = source.indexOf( - 'SELECT id, vm_size, vm_location, health_status, last_metrics, agent_version FROM nodes' - ); - const fallbackQueryEnd = source.indexOf('.bind(state.userId)', fallbackQueryStart); - const fallbackSection = source.slice(fallbackQueryStart, fallbackQueryEnd); - expect(fallbackSection).toContain("node_role = 'workspace'"); + expect(source).toContain("eq(schema.nodes.nodeRole, 'workspace')"); + expect(source).toContain("'capacity'"); }); }); @@ -287,10 +254,7 @@ describe('task-runner node-steps — node_role filtering', () => { describe('workspace creation node quota — node_role filtering', () => { it('workspace CRUD node count excludes deployment nodes', async () => { const fs = await import('fs'); - const source = fs.readFileSync( - path.join(SRC_DIR, 'routes/workspaces/crud.ts'), - 'utf-8' - ); + const source = fs.readFileSync(path.join(SRC_DIR, 'routes/workspaces/crud.ts'), 'utf-8'); // The node count for workspace creation quota must filter by nodeRole const countSection = source.slice( diff --git a/apps/api/tests/unit/node-selector-flow.test.ts b/apps/api/tests/unit/node-selector-flow.test.ts index 1a8b021952..0b75d2d4b3 100644 --- a/apps/api/tests/unit/node-selector-flow.test.ts +++ b/apps/api/tests/unit/node-selector-flow.test.ts @@ -1,14 +1,6 @@ -/** - * Tests for node selection logic (TDF-3). - * - * Includes: - * - Behavioral tests for nodeHasCapacity() and scoreNodeLoad() with actual function calls - * - Source contract tests for selectNodeForTaskRun() algorithm structure - */ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; -import type { NodeMetrics } from '@simple-agent-manager/shared'; import { DEFAULT_MAX_WORKSPACES_PER_NODE, DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT, @@ -18,423 +10,48 @@ import { describe, expect, it } from 'vitest'; import { nodeHasCapacity, scoreNodeLoad } from '../../src/services/node-selector'; -// ============================================================================= -// Behavioral tests — nodeHasCapacity() -// ============================================================================= - -describe('nodeHasCapacity', () => { - const cpuThreshold = DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT; // 50 - const memThreshold = DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT; // 50 - - it('returns true when all metrics are below thresholds', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 30, memoryPercent: 30, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(true); - }); - - it('returns false when CPU exceeds threshold', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 51, memoryPercent: 30, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(false); - }); - - it('returns false when memory exceeds threshold', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 30, memoryPercent: 51, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(false); - }); - - it('returns false when CPU equals threshold exactly', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 50, memoryPercent: 30, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(false); - }); - - it('returns false when memory equals threshold exactly', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 30, memoryPercent: 50, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(false); - }); - - it('returns true just below threshold', () => { - const metrics: NodeMetrics = { cpuLoadAvg1: 49, memoryPercent: 49, diskPercent: 20 }; - expect(nodeHasCapacity(metrics, cpuThreshold, memThreshold)).toBe(true); - }); - - it('returns true with null metrics (node may still be starting up)', () => { - expect(nodeHasCapacity(null, cpuThreshold, memThreshold)).toBe(true); - }); - - it('uses the correct default thresholds (50% CPU, 50% memory)', () => { - expect(DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT).toBe(50); - expect(DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT).toBe(50); - }); -}); - -// ============================================================================= -// Behavioral tests — scoreNodeLoad() -// ============================================================================= - -describe('scoreNodeLoad', () => { - it('returns null for null metrics', () => { - expect(scoreNodeLoad(null)).toBeNull(); - }); - - it('returns 0 for idle node', () => { - expect(scoreNodeLoad({ cpuLoadAvg1: 0, memoryPercent: 0, diskPercent: 0 })).toBe(0); - }); - - it('weights memory 60% and CPU 40%', () => { - const score = scoreNodeLoad({ cpuLoadAvg1: 100, memoryPercent: 0, diskPercent: 0 }); - expect(score).toBe(40); // 100 * 0.4 + 0 * 0.6 - - const score2 = scoreNodeLoad({ cpuLoadAvg1: 0, memoryPercent: 100, diskPercent: 0 }); - expect(score2).toBe(60); // 0 * 0.4 + 100 * 0.6 - }); - - it('computes weighted average correctly', () => { - const score = scoreNodeLoad({ cpuLoadAvg1: 50, memoryPercent: 50, diskPercent: 50 }); - expect(score).toBe(50); // 50 * 0.4 + 50 * 0.6 - }); -}); - -// ============================================================================= -// Source contract tests -// ============================================================================= - const selectorSource = readFileSync( resolve(process.cwd(), 'src/services/node-selector.ts'), 'utf8' ); +const evaluatorSource = readFileSync( + resolve(process.cwd(), 'src/services/placement-explanation.ts'), + 'utf8' +); +const taskRunnerSource = readFileSync( + resolve(process.cwd(), 'src/durable-objects/task-runner/node-steps.ts'), + 'utf8' +); -// ============================================================================= -// Algorithm structure — warm pool path -// ============================================================================= - -describe('selectNodeForTaskRun warm pool path', () => { - it('warm pool check runs before capacity check (Step 0 before regular query)', () => { - const step0Idx = selectorSource.indexOf('Step 0'); - const regularQueryIdx = selectorSource.indexOf('Get all running nodes'); - expect(step0Idx).toBeGreaterThan(-1); - expect(regularQueryIdx).toBeGreaterThan(step0Idx); - }); - - it('warm pool path is conditional on taskId AND NODE_LIFECYCLE binding', () => { - expect(selectorSource).toContain('if (taskId && env.NODE_LIFECYCLE)'); - }); - - it('queries only running nodes with non-null warmSince for the user', () => { - const warmQuery = selectorSource.slice( - selectorSource.indexOf('Step 0'), - selectorSource.indexOf('sortedWarm') - ); - expect(warmQuery).toContain('eq(schema.nodes.userId, userId)'); - expect(warmQuery).toContain("eq(schema.nodes.status, 'running')"); - expect(warmQuery).toContain('isNotNull(schema.nodes.warmSince)'); - }); - - it('sorts warm nodes by size match first, then location match', () => { - const sortSection = selectorSource.slice( - selectorSource.indexOf('const sortedWarm = warmNodes'), - selectorSource.indexOf('for (const warmNode') - ); - expect(sortSection).toContain('canSatisfyVmSize(node.vmSize, preferredSize)'); - // Size match is compared first - expect(sortSection).toContain('aSizeMatch'); - expect(sortSection).toContain('bSizeMatch'); - // Then location match - expect(sortSection).toContain('aLocMatch'); - expect(sortSection).toContain('bLocMatch'); - // Size uses preferredSize - expect(sortSection).toContain('preferredSize'); - // Location uses preferredLocation - expect(sortSection).toContain('preferredLocation'); - }); - - it('iterates warm nodes and tries to claim each one', () => { - expect(selectorSource).toContain('for (const warmNode of sortedWarm)'); - expect(selectorSource).toContain('nodeLifecycle.tryClaim'); - }); - - it('defense-in-depth: re-checks D1 status before DO claim', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - // Re-queries D1 to verify node is still running and warm - expect(warmSection).toContain('freshNode'); - expect(warmSection).toContain("freshNode.status !== 'running'"); - expect(warmSection).toContain('!freshNode.warmSince'); - expect(warmSection).toContain('continue'); - }); - - it('returns the first successfully claimed warm node', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain('result.claimed'); - expect(warmSection).toContain('return {'); - expect(warmSection).toContain('warmNode.id'); - }); - - it('checks workspace count before returning claimed warm nodes', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain('warmActiveCount >= maxWorkspacesPerNode'); - expect(warmSection).toContain('activeWorkspaceCount: warmActiveCount'); - }); - - it('catches claim failures and tries the next warm node', () => { - const warmSection = selectorSource.slice( - selectorSource.indexOf('for (const warmNode'), - selectorSource.indexOf('Get all running nodes') - ); - expect(warmSection).toContain('} catch {'); - }); - - it('falls through to capacity-based selection after all warm claims fail', () => { - // After the warm node loop, the regular node query runs - const afterWarmLoop = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(afterWarmLoop).toContain('.select()'); - }); -}); - -// ============================================================================= -// Algorithm structure — capacity-based path -// ============================================================================= - -describe('selectNodeForTaskRun capacity path', () => { - it('queries all running nodes for the user', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain('eq(schema.nodes.userId, userId)'); - expect(capacitySection).toContain("eq(schema.nodes.status, 'running')"); - }); - - it('returns null when no running nodes exist', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain('nodes.length === 0'); - expect(capacitySection).toContain('return null'); - }); - - it('skips unhealthy nodes', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain("node.healthStatus === 'unhealthy'"); - expect(capacitySection).toContain('continue'); - }); - - it('counts active workspaces per node (running, creating, recovery)', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain('count()'); - expect(capacitySection).toContain("'running', 'creating', 'recovery'"); - }); - - it('filters candidates by nodeHasCapacity', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain('canSatisfyVmSize(node.vmSize, preferredSize)'); - expect(capacitySection).toContain('nodeHasCapacity('); - expect(capacitySection).toContain('candidates.push(candidate)'); - }); - - it('returns null when no candidates have capacity', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - expect(capacitySection).toContain('candidates.length === 0'); - expect(capacitySection).toContain('return null'); - }); - - it('sorts candidates by location match, then size match, then load score', () => { - const sortSection = selectorSource.slice( - selectorSource.indexOf('Sort candidates'), - selectorSource.indexOf('const best = candidates[0]') - ); - // Location first - expect(sortSection).toContain('aLocationMatch'); - expect(sortSection).toContain('bLocationMatch'); - // Size second - expect(sortSection).toContain('aSizeMatch'); - expect(sortSection).toContain('bSizeMatch'); - // Load score last - expect(sortSection).toContain('scoreNodeLoad'); - expect(sortSection).toContain('aScore'); - expect(sortSection).toContain('bScore'); - }); - - it('returns the first candidate (lowest load, best match)', () => { - expect(selectorSource).toContain('const best = candidates[0];'); - expect(selectorSource).toContain('return best;'); - }); - - it('nodes with null metrics are ranked lower than nodes with scores', () => { - const sortSection = selectorSource.slice( - selectorSource.indexOf('Sort candidates'), - selectorSource.indexOf('const best = candidates[0]') - ); - // null scores go to end - expect(sortSection).toContain('aScore === null'); - expect(sortSection).toContain('return 1'); // null goes after - expect(sortSection).toContain('return -1'); // non-null goes before - }); -}); - -// ============================================================================= -// Threshold configuration -// ============================================================================= - -describe('selectNodeForTaskRun threshold configuration', () => { - it('reads CPU threshold from TASK_RUN_NODE_CPU_THRESHOLD_PERCENT env var', () => { - expect(selectorSource).toContain('env.TASK_RUN_NODE_CPU_THRESHOLD_PERCENT'); - }); - - it('reads memory threshold from TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT env var', () => { - expect(selectorSource).toContain('env.TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT'); - }); - - it('defaults CPU threshold to shared constant', () => { - expect(selectorSource).toContain('DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT'); - }); - - it('defaults memory threshold to shared constant', () => { - expect(selectorSource).toContain('DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT'); - }); - - it('parseThreshold rejects values < 0 or > 100', () => { - // Verify the parseThreshold function is present and validates range - expect(selectorSource).toContain('parsed < 0'); - expect(selectorSource).toContain('parsed > 100'); - }); -}); - -// ============================================================================= -// parseMetrics function -// ============================================================================= - -describe('parseMetrics (internal)', () => { - it('returns null for null input', () => { - expect(selectorSource).toContain('if (!raw) return null'); - }); - - it('returns null for invalid JSON', () => { - expect(selectorSource).toContain('} catch {'); - expect(selectorSource).toContain('return null'); - }); - - it('validates the parsed object against the NodeMetrics schema', () => { - // parseMetrics now schema-validates via valibot (nodeMetricsSchema) - // instead of a manual `typeof parsed.x === 'number'` OR-chain followed by - // a blind cast — a mistyped field used to slip through this check and - // poison scoreNodeLoad()/nodeHasCapacity() with NaN. Behavioral coverage - // for the schema-validated cases (mistyped field, no recognized fields, - // array, invalid JSON, well-formed) lives in - // tests/unit/services/node-selector.test.ts, exercised through - // selectNodeForTaskRun()'s returned lastMetrics — parseMetrics itself is - // not exported. - expect(selectorSource).toContain('const nodeMetricsSchema = v.object({'); - expect(selectorSource).toContain('v.safeParse(nodeMetricsSchema, parsed)'); - }); -}); - -// ============================================================================= -// Edge cases — structural validation -// ============================================================================= - -describe('selectNodeForTaskRun edge cases', () => { - it('skips warm pool when taskId is undefined or NODE_LIFECYCLE binding is missing', () => { - // Both conditions must be truthy for the warm pool path - expect(selectorSource).toContain('if (taskId && env.NODE_LIFECYCLE)'); - }); - - it('handles preferred size and location being undefined in warm sort', () => { - // The ternary checks if preferredSize is defined before comparing - const warmSort = selectorSource.slice( - selectorSource.indexOf('const sortedWarm = warmNodes'), - selectorSource.indexOf('for (const warmNode') - ); - expect(warmSort).toContain('preferredSize &&'); - expect(warmSort).toContain('preferredLocation &&'); - }); - - it('handles preferred location and size being undefined in capacity sort', () => { - const capacitySort = selectorSource.slice( - selectorSource.indexOf('Sort candidates'), - selectorSource.indexOf('const best = candidates[0]') - ); - expect(capacitySort).toContain('preferredLocation &&'); - expect(capacitySort).toContain('preferredSize &&'); - }); - - it('function signature accepts optional preferredLocation and preferredSize', () => { - expect(selectorSource).toContain('preferredLocation?: string'); - expect(selectorSource).toContain('preferredSize?: string'); - }); - - it('function signature accepts optional taskId', () => { - expect(selectorSource).toContain('taskId?: string'); +describe('canonical node selector flow', () => { + it('ranks by the most saturated resource without hardcoded weights', () => { + expect(scoreNodeLoad({ cpuLoadAvg1: 25, memoryPercent: 50 })).toBe(50); + expect(nodeHasCapacity({ cpuLoadAvg1: 49, memoryPercent: 49 }, 50, 50)).toBe(true); + expect(nodeHasCapacity({ cpuLoadAvg1: 50, memoryPercent: 10 }, 50, 50)).toBe(false); }); -}); - -// ============================================================================= -// Workspace count limit — behavioral + structural -// ============================================================================= -describe('workspace count limit (MAX_WORKSPACES_PER_NODE)', () => { - it('DEFAULT_MAX_WORKSPACES_PER_NODE is 3', () => { + it('uses shared scaling defaults', () => { expect(DEFAULT_MAX_WORKSPACES_PER_NODE).toBe(3); + expect(DEFAULT_TASK_RUN_NODE_CPU_THRESHOLD_PERCENT).toBe(50); + expect(DEFAULT_TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT).toBe(50); + expect(evaluatorSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); }); - it('NodeSelectorEnv includes MAX_WORKSPACES_PER_NODE', () => { - expect(selectorSource).toContain('MAX_WORKSPACES_PER_NODE?: string'); - }); - - it('reads MAX_WORKSPACES_PER_NODE from env with fallback to default', () => { - expect(selectorSource).toContain('env.MAX_WORKSPACES_PER_NODE'); - expect(selectorSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); - }); - - it('rejects nodes where activeCount >= maxWorkspacesPerNode before checking metrics', () => { - const capacitySection = selectorSource.slice(selectorSource.indexOf('Get all running nodes')); - // The workspace count check must appear before nodeHasCapacity - const wsCheckIdx = capacitySection.indexOf('activeCount >= maxWorkspacesPerNode'); - const metricsCheckIdx = capacitySection.indexOf('nodeHasCapacity('); - expect(wsCheckIdx).toBeGreaterThan(-1); - expect(metricsCheckIdx).toBeGreaterThan(wsCheckIdx); - }); - - it('continues to next node when workspace count limit is reached', () => { - // The workspace count check block includes a continue statement - const wsCheckStart = selectorSource.indexOf('activeCount >= maxWorkspacesPerNode'); - // Get the next ~100 chars after the check to capture the continue - const wsCheckBlock = selectorSource.slice(wsCheckStart, wsCheckStart + 100); - expect(wsCheckBlock).toContain('continue'); - }); -}); - -// ============================================================================= -// TaskRunner workspace count limit consistency -// ============================================================================= - -const taskRunnerSource = [ - 'index.ts', - 'types.ts', - 'node-steps.ts', - 'node-selection.ts', - 'workspace-steps.ts', - 'agent-session-step.ts', - 'state-machine.ts', - 'helpers.ts', -] - .map((f) => readFileSync(resolve(process.cwd(), 'src/durable-objects/task-runner', f), 'utf8')) - .join('\n'); - -describe('TaskRunner findNodeWithCapacity workspace count limit', () => { - it('imports DEFAULT_MAX_WORKSPACES_PER_NODE', () => { - expect(taskRunnerSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); + it('centralizes TaskRunner selection and persists the explanation immediately', () => { + expect(taskRunnerSource).toContain('selectNodeWithExplanation('); + expect(taskRunnerSource).toContain('persistTaskPlacement(state, rc, placement.explanation)'); + expect(taskRunnerSource).not.toContain('findNodeWithCapacity'); + expect(taskRunnerSource).not.toContain('tryClaimWarmNode'); }); - it('reads MAX_WORKSPACES_PER_NODE from env', () => { - const section = taskRunnerSource.slice(taskRunnerSource.indexOf('findNodeWithCapacity')); - expect(section).toContain('MAX_WORKSPACES_PER_NODE'); + it('evaluates preferred, warm, and capacity paths in the shared selector', () => { + expect(selectorSource).toContain("evaluatePlacementNode(node, request, 'warm'"); + expect(selectorSource).toContain("'capacity'"); + expect(selectorSource).toContain('options.preferredNodeId'); }); - it('queries workspace count per node and rejects at capacity', () => { - const section = taskRunnerSource.slice(taskRunnerSource.indexOf('findNodeWithCapacity')); - expect(section).toContain("status IN ('running', 'creating', 'recovery')"); - expect(section).toContain('>= maxWorkspaces'); + it('records warm claim loss and prevents immediate capacity reuse', () => { + expect(selectorSource).toContain("evaluation.rejectionReasons.push('warm-claim-lost')"); + expect(selectorSource).toContain('warmExclusions.get(evaluation.nodeId)'); }); }); diff --git a/apps/api/tests/unit/node-selector-warm.test.ts b/apps/api/tests/unit/node-selector-warm.test.ts index 2f214f3657..f2fa9ae752 100644 --- a/apps/api/tests/unit/node-selector-warm.test.ts +++ b/apps/api/tests/unit/node-selector-warm.test.ts @@ -1,72 +1,29 @@ -/** - * Source contract tests for warm node selection (T042). - * - * Verifies that selectNodeForTaskRun tries warm nodes first - * before falling through to capacity-based selection. - */ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { describe, expect, it } from 'vitest'; -describe('warm node selection source contract', () => { - const selectorFile = readFileSync(resolve(process.cwd(), 'src/services/node-selector.ts'), 'utf8'); - - describe('warm node query', () => { - it('queries D1 for nodes with non-null warm_since', () => { - expect(selectorFile).toContain('isNotNull(schema.nodes.warmSince)'); - }); - - it('only targets running nodes owned by user', () => { - expect(selectorFile).toContain("eq(schema.nodes.userId, userId)"); - expect(selectorFile).toContain("eq(schema.nodes.status, 'running')"); - }); - - it('sorts warm nodes by size/location preference', () => { - expect(selectorFile).toContain('sortedWarm'); - expect(selectorFile).toContain('preferredSize'); - expect(selectorFile).toContain('preferredLocation'); - }); - }); - - describe('tryClaim integration', () => { - it('calls nodeLifecycle.tryClaim for each warm node', () => { - expect(selectorFile).toContain('nodeLifecycle.tryClaim'); - }); - - it('returns claimed node on success', () => { - expect(selectorFile).toContain('result.claimed'); - }); - - it('continues to next warm node on claim failure', () => { - // Wrapped in try/catch to handle concurrent claims - const warmSection = selectorFile.slice(selectorFile.indexOf('Step 0')); - expect(warmSection).toContain('catch'); - }); - - it('falls through to capacity-based selection if no warm node claimed', () => { - // After the warm node loop, the regular selection logic runs - const warmSectionEnd = selectorFile.indexOf('Get all running nodes'); - const warmSectionStart = selectorFile.indexOf('Step 0'); - expect(warmSectionStart).toBeGreaterThan(-1); - expect(warmSectionEnd).toBeGreaterThan(warmSectionStart); - }); +const selectorSource = readFileSync( + resolve(process.cwd(), 'src/services/node-selector.ts'), + 'utf8' +); + +describe('warm node selection contract', () => { + it('uses the canonical evaluated-node list and NodeLifecycle claim', () => { + expect(selectorSource).toContain("evaluatePlacementNode(node, request, 'warm'"); + expect(selectorSource).toContain('nodeLifecycle.tryClaim'); + expect(selectorSource).toContain('options.taskId'); + expect(selectorSource).toContain('env.NODE_LIFECYCLE'); }); - describe('taskId parameter', () => { - it('selectNodeForTaskRun accepts optional taskId parameter', () => { - expect(selectorFile).toContain('taskId?: string'); - }); - - it('warm node selection only runs when taskId is provided', () => { - expect(selectorFile).toContain('if (taskId && env.NODE_LIFECYCLE)'); - }); - + it('tries another path after failed claims without silently accepting the lost node', () => { + expect(selectorSource).toContain('for (const evaluation of warmEvaluations'); + expect(selectorSource).toContain('catch {'); + expect(selectorSource).toContain("'warm-claim-lost'"); + expect(selectorSource).toContain('warmExclusions'); }); - describe('NodeSelectorEnv includes NODE_LIFECYCLE', () => { - it('has optional NODE_LIFECYCLE binding', () => { - expect(selectorFile).toContain('NODE_LIFECYCLE?: DurableObjectNamespace'); - }); + it('retains the optional lifecycle binding contract', () => { + expect(selectorSource).toContain('NODE_LIFECYCLE?: DurableObjectNamespace'); }); }); diff --git a/apps/api/tests/unit/openapi/sam-cli-openapi.test.ts b/apps/api/tests/unit/openapi/sam-cli-openapi.test.ts index ae9103ff2d..dec8fbebeb 100644 --- a/apps/api/tests/unit/openapi/sam-cli-openapi.test.ts +++ b/apps/api/tests/unit/openapi/sam-cli-openapi.test.ts @@ -10,6 +10,8 @@ type SchemaLike = { $ref?: string; type?: string | string[]; format?: string; + enum?: unknown[]; + nullable?: boolean; items?: SchemaLike; properties?: Record; }; @@ -115,6 +117,32 @@ describe('SAM CLI OpenAPI contract', () => { const sessionDetail = schema('SessionDetailResponse'); expect(refName(arrayItem(property(sessionDetail, 'messages')))).toBe('ChatMessage'); + + const provisioningAttempt = schema('PlacementProvisioningAttempt'); + const failureReason = property(provisioningAttempt, 'failureReason'); + expect(failureReason.nullable).toBeUndefined(); + expect(failureReason.enum).toEqual([ + 'capacity-unavailable', + 'node-limit', + 'quota-exceeded', + 'credentials-unavailable', + 'provider-failed', + 'provisioning-timeout', + 'readiness-timeout', + 'node-unavailable', + ]); + + const legacyExplanation = schema('LegacyPlacementExplanation'); + const reservation = property(legacyExplanation, 'reservation'); + expect(property(reservation, 'source').enum).toEqual([ + 'task', + 'trigger', + 'skill', + 'agent-profile', + 'project', + 'user', + 'platform', + ]); }); it('keeps the checked artifact in sync with the source document', async () => { diff --git a/apps/api/tests/unit/resolve-credential-source.test.ts b/apps/api/tests/unit/resolve-credential-source.test.ts index 81641ea3e6..73215a58ad 100644 --- a/apps/api/tests/unit/resolve-credential-source.test.ts +++ b/apps/api/tests/unit/resolve-credential-source.test.ts @@ -20,7 +20,7 @@ import { resolveCredentialSource } from '../../src/services/provider-credentials function makeCredentialSourceDbMock( projectRows: unknown[], userRows: unknown[], - platformRows: unknown[] = [], + platformRows: unknown[] = [] ) { const resultSets = [projectRows, userRows, platformRows]; let selectCount = 0; @@ -41,7 +41,7 @@ function makeCredentialSourceDbMock( describe('resolveCredentialSource', () => { const providerCredsSource = readFileSync( resolve(process.cwd(), 'src/services/provider-credentials.ts'), - 'utf8', + 'utf8' ); it('exports resolveCredentialSource function', async () => { @@ -65,34 +65,36 @@ describe('resolveCredentialSource', () => { }); it('filters platform credentials by targetProvider when specified', () => { - expect(providerCredsSource).toContain('eq(schema.platformCredentials.provider, targetProvider)'); + expect(providerCredsSource).toContain( + 'eq(schema.platformCredentials.provider, targetProvider)' + ); }); it('returns credentialSource user when user has matching credential', () => { // Verify the user credential path returns 'user' const resolveFunc = providerCredsSource.substring( - providerCredsSource.indexOf('export async function resolveCredentialSource'), + providerCredsSource.indexOf('export async function resolveCredentialSource') ); expect(resolveFunc).toContain("credentialSource: 'user'"); }); it('returns credentialSource platform when falling back to platform', () => { const resolveFunc = providerCredsSource.substring( - providerCredsSource.indexOf('export async function resolveCredentialSource'), + providerCredsSource.indexOf('export async function resolveCredentialSource') ); expect(resolveFunc).toContain("credentialSource: 'platform'"); }); it('returns null when no credentials exist', () => { const resolveFunc = providerCredsSource.substring( - providerCredsSource.indexOf('export async function resolveCredentialSource'), + providerCredsSource.indexOf('export async function resolveCredentialSource') ); expect(resolveFunc).toContain('return null;'); }); it('resolves project credentials BEFORE user and platform credentials', () => { const resolveFunc = providerCredsSource.substring( - providerCredsSource.indexOf('export async function resolveCredentialSource'), + providerCredsSource.indexOf('export async function resolveCredentialSource') ); const projectCheckIdx = resolveFunc.indexOf('resolveProjectComputeCredentialSource'); const userCheckIdx = resolveFunc.indexOf('schema.credentials.userId'); @@ -108,7 +110,7 @@ describe('resolveCredentialSource', () => { // the system will use whatever provider the user has). const createProviderFunc = providerCredsSource.substring( providerCredsSource.indexOf('export async function createProviderForUser'), - providerCredsSource.indexOf('export async function resolveCredentialSource'), + providerCredsSource.indexOf('export async function resolveCredentialSource') ); // createProviderForUser also checks user creds first, then platform expect(createProviderFunc).toContain("credentialSource: 'user'"); @@ -119,46 +121,47 @@ describe('resolveCredentialSource', () => { describe('resolveCredentialSource project compute precedence', () => { it('returns project source when an active project attachment exists', async () => { const db = makeCredentialSourceDbMock( - [{ - attachmentActive: true, - consumerTarget: 'hetzner', - configurationActive: true, - credentialId: 'cc-project-cred', - credentialActive: true, - }], - [{ id: 'personal-cred', provider: 'hetzner' }], + [ + { + attachmentActive: true, + consumerTarget: 'hetzner', + configurationActive: true, + credentialId: 'cc-project-cred', + credentialActive: true, + }, + ], + [{ id: 'personal-cred', provider: 'hetzner' }] ); await expect( - resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1') ).resolves.toEqual({ credentialSource: 'project', providerName: 'hetzner' }); }); it('halts on an inactive project attachment instead of falling through to personal credentials', async () => { const db = makeCredentialSourceDbMock( - [{ - attachmentActive: false, - consumerTarget: 'hetzner', - configurationActive: true, - credentialId: 'cc-project-cred', - credentialActive: true, - }], - [{ id: 'personal-cred', provider: 'hetzner' }], + [ + { + attachmentActive: false, + consumerTarget: 'hetzner', + configurationActive: true, + credentialId: 'cc-project-cred', + credentialActive: true, + }, + ], + [{ id: 'personal-cred', provider: 'hetzner' }] ); await expect( - resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1') ).resolves.toBeNull(); }); it('falls back to the pinned creator personal credential when no project attachment exists', async () => { - const db = makeCredentialSourceDbMock( - [], - [{ id: 'personal-cred', provider: 'hetzner' }], - ); + const db = makeCredentialSourceDbMock([], [{ id: 'personal-cred', provider: 'hetzner' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'hetzner', 'project-1') ).resolves.toEqual({ credentialSource: 'user', providerName: 'hetzner' }); }); @@ -166,7 +169,7 @@ describe('resolveCredentialSource project compute precedence', () => { const db = makeCredentialSourceDbMock([], [], []); await expect( - resolveCredentialSource(db as never, 'member-a', 'scaleway', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'scaleway', 'project-1') ).resolves.toBeNull(); }); }); @@ -186,38 +189,38 @@ describe('resolveCredentialSource vultr fallback matrix (rule 28)', () => { it('active project attachment → project', async () => { const db = makeCredentialSourceDbMock([activeProjectRow], [{ id: 'u', provider: 'vultr' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1') ).resolves.toEqual({ credentialSource: 'project', providerName: 'vultr' }); }); it('inactive project attachment halts — does NOT fall through to the user vultr credential', async () => { const db = makeCredentialSourceDbMock( [{ ...activeProjectRow, attachmentActive: false }], - [{ id: 'u', provider: 'vultr' }], + [{ id: 'u', provider: 'vultr' }] ); await expect( - resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1') ).resolves.toBeNull(); }); it('no project attachment → user vultr credential', async () => { const db = makeCredentialSourceDbMock([], [{ id: 'u', provider: 'vultr' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1') ).resolves.toEqual({ credentialSource: 'user', providerName: 'vultr' }); }); it('no project, no user → platform vultr credential', async () => { const db = makeCredentialSourceDbMock([], [], [{ id: 'p', provider: 'vultr' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1') ).resolves.toEqual({ credentialSource: 'platform', providerName: 'vultr' }); }); it('nothing at any tier → null', async () => { const db = makeCredentialSourceDbMock([], [], []); await expect( - resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'vultr', 'project-1') ).resolves.toBeNull(); }); }); @@ -225,7 +228,7 @@ describe('resolveCredentialSource vultr fallback matrix (rule 28)', () => { describe('userHasOwnCloudCredentials with targetProvider', () => { const serviceSource = readFileSync( resolve(process.cwd(), 'src/services/compute-quotas.ts'), - 'utf8', + 'utf8' ); it('exports userHasOwnCloudCredentials function', async () => { @@ -249,21 +252,17 @@ describe('userHasOwnCloudCredentials with targetProvider', () => { }); describe('quota enforcement pattern: credential source, not existence', () => { - const submitSource = readFileSync( - resolve(process.cwd(), 'src/routes/tasks/submit.ts'), - 'utf8', - ); - const nodeStepsSource = readFileSync( - resolve(process.cwd(), 'src/durable-objects/task-runner/node-steps.ts'), - 'utf8', - ); - const nodesSource = readFileSync( - resolve(process.cwd(), 'src/routes/nodes.ts'), - 'utf8', - ); + const submitSource = readFileSync(resolve(process.cwd(), 'src/routes/tasks/submit.ts'), 'utf8'); + const nodeStepsSource = [ + 'src/durable-objects/task-runner/node-steps.ts', + 'src/durable-objects/task-runner/provisioning-guards.ts', + ] + .map((file) => readFileSync(resolve(process.cwd(), file), 'utf8')) + .join('\n'); + const nodesSource = readFileSync(resolve(process.cwd(), 'src/routes/nodes.ts'), 'utf8'); const dispatchSource = readFileSync( resolve(process.cwd(), 'src/routes/mcp/dispatch-tool.ts'), - 'utf8', + 'utf8' ); // ========================================================================= @@ -287,7 +286,9 @@ describe('quota enforcement pattern: credential source, not existence', () => { it('dispatch-tool.ts does NOT have raw credential existence check in Promise.all', () => { // The old pattern: query credentials table in parallel and gate on !credential - expect(dispatchSource).not.toContain("eq(schema.credentials.credentialType, 'cloud-provider')"); + expect(dispatchSource).not.toContain( + "eq(schema.credentials.credentialType, 'cloud-provider')" + ); }); }); @@ -455,40 +456,43 @@ describe('resolveCredentialSource digitalocean fallback matrix (rule 28)', () => }; it('active project attachment → project', async () => { - const db = makeCredentialSourceDbMock([activeProjectRow], [{ id: 'u', provider: 'digitalocean' }]); + const db = makeCredentialSourceDbMock( + [activeProjectRow], + [{ id: 'u', provider: 'digitalocean' }] + ); await expect( - resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1') ).resolves.toEqual({ credentialSource: 'project', providerName: 'digitalocean' }); }); it('inactive project attachment halts — does NOT fall through to the user digitalocean credential', async () => { const db = makeCredentialSourceDbMock( [{ ...activeProjectRow, attachmentActive: false }], - [{ id: 'u', provider: 'digitalocean' }], + [{ id: 'u', provider: 'digitalocean' }] ); await expect( - resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1') ).resolves.toBeNull(); }); it('no project attachment → user digitalocean credential', async () => { const db = makeCredentialSourceDbMock([], [{ id: 'u', provider: 'digitalocean' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1') ).resolves.toEqual({ credentialSource: 'user', providerName: 'digitalocean' }); }); it('no project, no user → platform digitalocean credential', async () => { const db = makeCredentialSourceDbMock([], [], [{ id: 'p', provider: 'digitalocean' }]); await expect( - resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1') ).resolves.toEqual({ credentialSource: 'platform', providerName: 'digitalocean' }); }); it('nothing at any tier → null', async () => { const db = makeCredentialSourceDbMock([], [], []); await expect( - resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1'), + resolveCredentialSource(db as never, 'member-a', 'digitalocean', 'project-1') ).resolves.toBeNull(); }); }); diff --git a/apps/api/tests/unit/routes/manual-workspace-placement-boundary.test.ts b/apps/api/tests/unit/routes/manual-workspace-placement-boundary.test.ts new file mode 100644 index 0000000000..58a9c0e00c --- /dev/null +++ b/apps/api/tests/unit/routes/manual-workspace-placement-boundary.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from 'vitest'; + +import { resolveManualWorkspacePlacement } from '../../../src/routes/workspaces/manual-placement'; + +describe('manual workspace placement boundary', () => { + it('reuses an eligible real D1 candidate with a finalized typed decision', async () => { + const requiredVersion = 'a'.repeat(40); + const nodeId = '01KZR2JAP92AK3SKW951E4H21M'; + const queryResults: unknown[][] = [ + [ + { + id: nodeId, + status: 'running', + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + lastHeartbeatAt: new Date().toISOString(), + agentReadyAt: new Date().toISOString(), + agentVersion: requiredVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 2, memoryPercent: 10 }), + warmSince: null, + }, + ], + [], + ]; + const db = { + select() { + const rows = queryResults.shift() ?? []; + return { + from() { + return { where: async () => rows }; + }, + }; + }, + }; + + const result = await resolveManualWorkspacePlacement({ + db: db as never, + env: { VM_AGENT_REQUIRED_VERSION: requiredVersion } as never, + userId: 'user-1', + projectId: 'project-1', + workspaceName: 'Workspace', + vmSize: 'medium', + vmLocation: 'hel1', + preferredNodeId: nodeId, + activeNodeCount: 1, + maxNodesPerUser: 5, + heartbeatStaleAfterSeconds: 180, + now: '2026-08-11T00:00:00.000Z', + }); + + expect(result).toMatchObject({ + nodeId, + mustProvisionNode: false, + explanation: { + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'manual', + selectedNodeId: nodeId, + evaluatedNodes: [{ nodeId, path: 'manual', accepted: true, rejectionReasons: [] }], + }, + }); + }); + + it('evaluates a real D1 candidate and persists its tenant-safe typed rejection', async () => { + const requiredVersion = 'a'.repeat(40); + const rawAgentVersion = `wrong-CANARY_AGENT_VERSION-${'b'.repeat(40)}`; + const inserted: Array> = []; + const queryResults: unknown[][] = [ + [ + { + id: '01KZR2JAP92AK3SKW951E4H21M', + status: 'running', + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + lastHeartbeatAt: new Date().toISOString(), + agentReadyAt: new Date().toISOString(), + agentVersion: rawAgentVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 2, memoryPercent: 10 }), + warmSince: null, + }, + ], + [], + ]; + const db = { + select() { + const rows = queryResults.shift() ?? []; + return { + from() { + return { where: async () => rows }; + }, + }; + }, + insert() { + return { + async values(value: Record) { + inserted.push(value); + }, + }; + }, + }; + + await expect( + resolveManualWorkspacePlacement({ + db: db as never, + env: { VM_AGENT_REQUIRED_VERSION: requiredVersion } as never, + userId: 'user-1', + projectId: 'project-1', + workspaceName: 'Workspace', + vmSize: 'medium', + vmLocation: 'hel1', + preferredNodeId: '01KZR2JAP92AK3SKW951E4H21M', + activeNodeCount: 1, + maxNodesPerUser: 5, + heartbeatStaleAfterSeconds: 180, + now: '2026-08-11T00:00:00.000Z', + }) + ).rejects.toThrow('Selected node is not eligible'); + + const stored = JSON.parse(inserted[0]?.placementExplanationJson as string) as Record< + string, + unknown + >; + expect(stored).toMatchObject({ + schemaVersion: 2, + outcome: 'failed', + selectionPath: 'manual', + evaluatedNodes: [ + { + nodeId: 'candidate-1', + accepted: false, + rejectionReasons: ['agent-version-mismatch'], + }, + ], + }); + expect(JSON.stringify(stored)).not.toContain(rawAgentVersion); + expect(JSON.stringify(stored)).not.toContain('01KZR2JAP92AK3SKW951E4H21M'); + }); +}); diff --git a/apps/api/tests/unit/routes/manual-workspace-placement-route.test.ts b/apps/api/tests/unit/routes/manual-workspace-placement-route.test.ts new file mode 100644 index 0000000000..1f134b9d80 --- /dev/null +++ b/apps/api/tests/unit/routes/manual-workspace-placement-route.test.ts @@ -0,0 +1,273 @@ +import Database from 'better-sqlite3'; +import { Hono } from 'hono'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const mocks = vi.hoisted(() => ({ + createSession: vi.fn(), + createWorkspaceOnNode: vi.fn(), + provisionNode: vi.fn(), + recordActivityEvent: vi.fn(), + signCallbackToken: vi.fn(), + waitForNodeAgentReady: vi.fn(), +})); + +vi.mock('../../../src/middleware/auth', () => ({ + requireAuth: () => (_context: unknown, next: () => Promise) => next(), + requireApproved: () => (_context: unknown, next: () => Promise) => next(), + getAuth: () => ({ + user: { + id: 'user-1', + name: 'User One', + email: 'user-1@example.com', + status: 'active', + role: 'user', + }, + }), + getUserId: () => 'user-1', +})); +vi.mock('../../../src/services/node-agent', async (importOriginal) => ({ + ...(await importOriginal()), + createWorkspaceOnNode: mocks.createWorkspaceOnNode, + waitForNodeAgentReady: mocks.waitForNodeAgentReady, +})); +vi.mock('../../../src/services/nodes', async (importOriginal) => ({ + ...(await importOriginal()), + provisionNode: mocks.provisionNode, +})); +vi.mock('../../../src/services/jwt', async (importOriginal) => ({ + ...(await importOriginal()), + signCallbackToken: mocks.signCallbackToken, +})); +vi.mock('../../../src/services/project-data', async (importOriginal) => ({ + ...(await importOriginal()), + createSession: mocks.createSession, + recordActivityEvent: mocks.recordActivityEvent, +})); + +import * as schema from '../../../src/db/schema'; +import type { Env } from '../../../src/env'; +import { AppError } from '../../../src/middleware/error'; +import { crudRoutes } from '../../../src/routes/workspaces/crud'; +import { createAllSchemaTables, createSqliteD1 } from '../../helpers/sqlite-d1'; + +const NODE_ID = '01KZR2JAP92AK3SKW951E4H21M'; +const REQUIRED_VERSION = 'a'.repeat(40); + +describe('manual workspace placement route vertical slice', () => { + let sqlite: Database.Database; + let env: Env; + let app: Hono<{ Bindings: Env }>; + + beforeEach(() => { + vi.clearAllMocks(); + sqlite = new Database(':memory:'); + createAllSchemaTables(sqlite, schema); + env = { + DATABASE: createSqliteD1(sqlite), + BASE_DOMAIN: 'example.test', + VM_AGENT_REQUIRED_VERSION: REQUIRED_VERSION, + } as Env; + app = new Hono<{ Bindings: Env }>(); + app.onError((error, context) => + error instanceof AppError + ? context.json(error.toJSON(), error.statusCode as never) + : context.json({ error: 'INTERNAL_ERROR', message: error.message }, 500) + ); + app.route('/api/workspaces', crudRoutes); + + const now = '2026-08-11T00:00:00.000Z'; + sqlite + .prepare( + `INSERT INTO projects + (id, user_id, name, normalized_name, installation_id, repository, + default_branch, repo_provider, status, created_by, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, 'main', 'artifacts', 'active', ?, ?, ?)` + ) + .run( + 'project-1', + 'user-1', + 'Placement project', + 'placement project', + 'artifacts-installation', + 'artifacts/placement-project', + 'user-1', + now, + now + ); + sqlite + .prepare( + `INSERT INTO project_members + (project_id, user_id, role, status, created_at, updated_at) + VALUES (?, ?, 'owner', 'active', ?, ?)` + ) + .run('project-1', 'user-1', now, now); + sqlite + .prepare( + `INSERT INTO nodes + (id, user_id, name, status, vm_size, vm_location, runtime, node_role, + node_mode, health_status, heartbeat_stale_after_seconds, + last_heartbeat_at, agent_ready_at, agent_version, last_metrics, + credential_source, created_at, updated_at) + VALUES (?, ?, ?, 'running', 'medium', 'hel1', 'vm', 'workspace', + 'shared', 'healthy', 180, ?, ?, ?, ?, 'user', ?, ?)` + ) + .run( + NODE_ID, + 'user-1', + 'Existing node', + new Date().toISOString(), + new Date().toISOString(), + REQUIRED_VERSION, + JSON.stringify({ cpuLoadAvg1: 2, memoryPercent: 10 }), + now, + now + ); + + mocks.createSession.mockResolvedValue('session-1'); + mocks.recordActivityEvent.mockResolvedValue(undefined); + mocks.signCallbackToken.mockResolvedValue('callback-token'); + mocks.provisionNode.mockResolvedValue(undefined); + mocks.waitForNodeAgentReady.mockResolvedValue(undefined); + }); + + afterEach(() => sqlite.close()); + + it('persists selected-node reuse to its workspace and task before VM scheduling', async () => { + const scheduleObservations: Array<{ + workspace: Record; + task: Record; + }> = []; + mocks.createWorkspaceOnNode.mockImplementation(async () => { + const workspace = sqlite + .prepare( + `SELECT id, node_id, placement_explanation_json + FROM workspaces WHERE project_id = ?` + ) + .get('project-1') as Record; + const task = sqlite + .prepare( + `SELECT id, workspace_id, placement_explanation_json + FROM tasks WHERE project_id = ?` + ) + .get('project-1') as Record; + scheduleObservations.push({ workspace, task }); + return { workspaceId: workspace.id, status: 'creating' }; + }); + const waitUntilPromises: Promise[] = []; + const executionContext = { + waitUntil: vi.fn((promise: Promise) => waitUntilPromises.push(promise)), + passThroughOnException: vi.fn(), + } as unknown as ExecutionContext; + + const response = await app.request( + '/api/workspaces', + { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + name: 'Reused workspace', + projectId: 'project-1', + nodeId: NODE_ID, + vmSize: 'medium', + vmLocation: 'hel1', + }), + }, + env, + executionContext + ); + await Promise.all(waitUntilPromises); + + expect(response.status).toBe(201); + expect(mocks.createWorkspaceOnNode).toHaveBeenCalledOnce(); + expect(scheduleObservations).toHaveLength(1); + const [{ workspace, task }] = scheduleObservations; + expect(task.workspace_id).toBe(workspace.id); + for (const row of [workspace, task]) { + expect(JSON.parse(row.placement_explanation_json as string)).toMatchObject({ + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'manual', + selectedNodeId: NODE_ID, + evaluatedNodes: [{ nodeId: NODE_ID, path: 'manual', accepted: true, rejectionReasons: [] }], + }); + } + }); + + it('carries real credential resolution and create-node output into started then succeeded records', async () => { + const now = '2026-08-11T00:00:00.000Z'; + sqlite + .prepare( + `INSERT INTO credentials + (id, user_id, provider, credential_type, credential_kind, is_active, + encrypted_token, iv, created_at, updated_at) + VALUES (?, ?, 'hetzner', 'cloud-provider', 'api-key', 1, ?, ?, ?, ?)` + ) + .run('credential-1', 'user-1', 'encrypted-not-read-by-resolution', 'iv-not-read', now, now); + + let provisionedNodeId: string | null = null; + mocks.provisionNode.mockImplementation(async (nodeId: string) => { + provisionedNodeId = nodeId; + const workspace = sqlite + .prepare(`SELECT placement_explanation_json FROM workspaces WHERE project_id = ?`) + .get('project-1') as { placement_explanation_json: string }; + const task = sqlite + .prepare(`SELECT placement_explanation_json FROM tasks WHERE project_id = ?`) + .get('project-1') as { placement_explanation_json: string }; + for (const row of [workspace, task]) { + expect(JSON.parse(row.placement_explanation_json)).toMatchObject({ + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: nodeId, + provisioningAttempts: [{ vmSize: 'medium', vmLocation: 'hel1', outcome: 'started' }], + }); + } + sqlite.prepare(`UPDATE nodes SET status = 'running' WHERE id = ?`).run(nodeId); + }); + mocks.createWorkspaceOnNode.mockImplementation(async (nodeId: string) => { + const workspace = sqlite + .prepare(`SELECT placement_explanation_json FROM workspaces WHERE project_id = ?`) + .get('project-1') as { placement_explanation_json: string }; + const task = sqlite + .prepare(`SELECT placement_explanation_json FROM tasks WHERE project_id = ?`) + .get('project-1') as { placement_explanation_json: string }; + for (const row of [workspace, task]) { + expect(JSON.parse(row.placement_explanation_json)).toMatchObject({ + outcome: 'provisioned', + selectedNodeId: nodeId, + provisioningAttempts: [{ outcome: 'started' }, { outcome: 'succeeded' }], + }); + } + return { workspaceId: 'workspace-new', status: 'creating' }; + }); + const waitUntilPromises: Promise[] = []; + const executionContext = { + waitUntil: vi.fn((promise: Promise) => waitUntilPromises.push(promise)), + passThroughOnException: vi.fn(), + } as unknown as ExecutionContext; + + const response = await app.request( + '/api/workspaces', + { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + name: 'Provisioned workspace', + projectId: 'project-1', + provider: 'hetzner', + vmSize: 'medium', + vmLocation: 'hel1', + }), + }, + env, + executionContext + ); + await Promise.all(waitUntilPromises); + + expect(response.status).toBe(201); + expect(provisionedNodeId).toMatch(/^[0-9A-HJKMNP-TV-Z]{26}$/); + expect(mocks.provisionNode).toHaveBeenCalledOnce(); + expect(mocks.waitForNodeAgentReady).toHaveBeenCalledWith(provisionedNodeId, env); + expect(mocks.createWorkspaceOnNode).toHaveBeenCalledOnce(); + }); +}); diff --git a/apps/api/tests/unit/routes/manual-workspace-placement.test.ts b/apps/api/tests/unit/routes/manual-workspace-placement.test.ts new file mode 100644 index 0000000000..d3f4e34591 --- /dev/null +++ b/apps/api/tests/unit/routes/manual-workspace-placement.test.ts @@ -0,0 +1,359 @@ +import type { PlacementExplanation } from '@simple-agent-manager/shared'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import * as schema from '../../../src/db/schema'; + +const mocks = vi.hoisted(() => ({ + createNodeRecord: vi.fn(), + drizzle: vi.fn(), + provisionNode: vi.fn(), + resolveCredentialSource: vi.fn(), + scheduleWorkspaceCreateOnNode: vi.fn(), + selectNodeWithExplanation: vi.fn(), + waitForNodeAgentReady: vi.fn(), +})); + +vi.mock('drizzle-orm/d1', () => ({ drizzle: mocks.drizzle })); +vi.mock('../../../src/services/node-agent', () => ({ + waitForNodeAgentReady: mocks.waitForNodeAgentReady, +})); +vi.mock('../../../src/services/node-selector', () => ({ + selectNodeWithExplanation: mocks.selectNodeWithExplanation, +})); +vi.mock('../../../src/services/nodes', () => ({ + createNodeRecord: mocks.createNodeRecord, + provisionNode: mocks.provisionNode, +})); +vi.mock('../../../src/services/provider-credentials', () => ({ + resolveCredentialSource: mocks.resolveCredentialSource, +})); +vi.mock('../../../src/routes/workspaces/_helpers', () => ({ + scheduleWorkspaceCreateOnNode: mocks.scheduleWorkspaceCreateOnNode, +})); + +import { + completeManualWorkspacePlacement, + resolveManualWorkspacePlacement, +} from '../../../src/routes/workspaces/manual-placement'; + +function placement(outcome: PlacementExplanation['outcome'] = 'failed'): PlacementExplanation { + return { + schemaVersion: 2, + outcome, + selectionPath: 'manual', + selectedNodeId: null, + summary: 'Selected node was rejected.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + }; +} + +function resolveInput(db: unknown) { + return { + db, + env: {}, + userId: 'user-1', + projectId: 'project-1', + workspaceName: 'Workspace', + vmSize: 'medium' as const, + vmLocation: 'hel1', + activeNodeCount: 0, + maxNodesPerUser: 5, + heartbeatStaleAfterSeconds: 180, + now: '2026-08-11T00:00:00.000Z', + }; +} + +function insertDb() { + const inserted: Array> = []; + return { + inserted, + db: { + insert() { + return { + async values(value: Record) { + inserted.push(value); + }, + }; + }, + }, + }; +} + +function sqlValues(value: unknown, seen = new Set()): unknown[] { + if (value === null || typeof value !== 'object' || seen.has(value)) return []; + seen.add(value); + const record = value as Record; + const own = Object.hasOwn(record, 'value') ? [record.value] : []; + return own.concat( + Object.values(record).flatMap((child) => + Array.isArray(child) ? child.flatMap((item) => sqlValues(item, seen)) : sqlValues(child, seen) + ) + ); +} + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe('manual workspace placement persistence', () => { + it('persists a typed failed task when a selected node is rejected', async () => { + const harness = insertDb(); + mocks.selectNodeWithExplanation.mockResolvedValue({ node: null, explanation: placement() }); + + await expect( + resolveManualWorkspacePlacement({ + ...resolveInput(harness.db), + preferredNodeId: 'node-rejected', + } as never) + ).rejects.toThrow('Selected node is not eligible'); + + const stored = harness.inserted[0]; + expect(stored).toMatchObject({ status: 'failed', projectId: 'project-1' }); + expect(JSON.parse(stored?.placementExplanationJson as string)).toMatchObject({ + schemaVersion: 2, + outcome: 'failed', + }); + }); + + it.each([ + ['node-limit', { activeNodeCount: 5 }], + ['credentials-unavailable', {}], + ] as const)('persists %s when provisioning cannot start', async (reason, override) => { + const harness = insertDb(); + mocks.resolveCredentialSource.mockResolvedValue(null); + + await expect( + resolveManualWorkspacePlacement({ + ...resolveInput(harness.db), + ...override, + } as never) + ).rejects.toThrow(); + + const explanation = JSON.parse( + harness.inserted[0]?.placementExplanationJson as string + ) as PlacementExplanation; + expect(explanation.outcome).toBe('failed'); + expect(explanation.provisioningAttempts).toEqual([ + expect.objectContaining({ outcome: 'failed', failureReason: reason }), + ]); + }); + + it('updates both workspace and task when asynchronous provisioning fails', async () => { + const updates: Array<{ + target: 'workspace' | 'task'; + id: string; + value: Record; + }> = []; + const db = { + update(table: unknown) { + const target = table === schema.workspaces ? 'workspace' : 'task'; + return { + set(value: Record) { + return { + async where(condition: unknown) { + const id = target === 'workspace' ? 'workspace-1' : 'task-1'; + expect(sqlValues(condition)).toContain(id); + updates.push({ target, id, value }); + }, + }; + }, + }; + }, + select() { + return { from: () => ({ where: () => ({ limit: async () => [] }) }) }; + }, + }; + mocks.drizzle.mockReturnValue(db); + mocks.provisionNode.mockRejectedValue(new Error('CANARY_PROVIDER_DETAIL')); + + await completeManualWorkspacePlacement({ + env: { DATABASE: {} }, + explanation: placement('provisioned'), + mustProvisionNode: true, + nodeId: 'node-new', + workspaceId: 'workspace-1', + taskId: 'task-1', + userId: 'user-1', + repository: 'owner/repo', + branch: 'main', + project: { id: 'project-1' }, + vmSize: 'medium', + vmLocation: 'hel1', + } as never); + + expect(updates.map(({ target, id, value }) => ({ target, id, value }))).toEqual([ + { + target: 'workspace', + id: 'workspace-1', + value: expect.objectContaining({ placementExplanationJson: expect.any(String) }), + }, + { + target: 'task', + id: 'task-1', + value: expect.objectContaining({ placementExplanationJson: expect.any(String) }), + }, + { + target: 'workspace', + id: 'workspace-1', + value: expect.objectContaining({ + status: 'error', + errorMessage: 'Node provisioning failed', + }), + }, + { + target: 'task', + id: 'task-1', + value: expect.objectContaining({ + status: 'failed', + errorMessage: 'Node provisioning failed', + }), + }, + ]); + for (const update of updates.slice(0, 2)) { + expect(JSON.parse(update.value.placementExplanationJson as string)).toMatchObject({ + outcome: 'failed', + selectedNodeId: 'node-new', + provisioningAttempts: [ + expect.objectContaining({ outcome: 'failed', failureReason: 'provider-failed' }), + ], + }); + } + expect(JSON.stringify(updates)).not.toContain('CANARY_PROVIDER_DETAIL'); + expect(mocks.scheduleWorkspaceCreateOnNode).not.toHaveBeenCalled(); + }); + + it('finalizes a reused decision in both workspace and task before scheduling', async () => { + const updates: Array> = []; + const events: string[] = []; + mocks.drizzle.mockReturnValue({ + update(table: unknown) { + const target = table === schema.workspaces ? 'workspace' : 'task'; + return { + set(value: Record) { + updates.push(value); + return { + async where(condition: unknown) { + const expectedId = target === 'workspace' ? 'workspace-1' : 'task-1'; + expect(sqlValues(condition)).toContain(expectedId); + events.push(`${target}:persist`); + }, + }; + }, + }; + }, + }); + mocks.scheduleWorkspaceCreateOnNode.mockImplementation(async () => { + events.push('schedule'); + }); + const explanation = placement('reused'); + explanation.selectedNodeId = 'node-existing'; + + await completeManualWorkspacePlacement({ + env: { DATABASE: {} }, + explanation, + mustProvisionNode: false, + nodeId: 'node-existing', + workspaceId: 'workspace-1', + taskId: 'task-1', + userId: 'user-1', + repository: 'owner/repo', + branch: 'main', + project: { id: 'project-1' }, + vmSize: 'medium', + vmLocation: 'hel1', + } as never); + + const persisted = updates.filter((update) => 'placementExplanationJson' in update); + expect(persisted).toHaveLength(2); + expect( + persisted.map((update) => JSON.parse(update.placementExplanationJson as string)) + ).toEqual([ + expect.objectContaining({ outcome: 'reused', selectedNodeId: 'node-existing' }), + expect.objectContaining({ outcome: 'reused', selectedNodeId: 'node-existing' }), + ]); + expect(mocks.scheduleWorkspaceCreateOnNode).toHaveBeenCalledOnce(); + expect(events).toEqual(['workspace:persist', 'task:persist', 'schedule']); + }); + + it('persists started then succeeded provisioning in both records before scheduling', async () => { + const updates: Array> = []; + const events: string[] = []; + mocks.drizzle.mockReturnValue({ + update(table: unknown) { + const target = table === schema.workspaces ? 'workspace' : 'task'; + return { + set(value: Record) { + updates.push(value); + return { + async where(condition: unknown) { + const expectedId = target === 'workspace' ? 'workspace-1' : 'task-1'; + expect(sqlValues(condition)).toContain(expectedId); + events.push(`${target}:persist`); + }, + }; + }, + }; + }, + select() { + return { + from: () => ({ + where: () => ({ + limit: async () => [{ status: 'running', errorMessage: null }], + }), + }), + }; + }, + }); + mocks.provisionNode.mockResolvedValue(undefined); + mocks.waitForNodeAgentReady.mockResolvedValue(undefined); + mocks.scheduleWorkspaceCreateOnNode.mockImplementation(async () => { + events.push('schedule'); + }); + const explanation = placement('provisioned'); + explanation.selectionPath = 'provisioning'; + explanation.provisioningAttempts = [ + { vmSize: 'medium', vmLocation: 'hel1', outcome: 'started' }, + ]; + + await completeManualWorkspacePlacement({ + env: { DATABASE: {} }, + explanation, + mustProvisionNode: true, + nodeId: 'node-new', + workspaceId: 'workspace-1', + taskId: 'task-1', + userId: 'user-1', + repository: 'owner/repo', + branch: 'main', + project: { id: 'project-1' }, + vmSize: 'medium', + vmLocation: 'hel1', + } as never); + + const persisted = updates.filter((update) => 'placementExplanationJson' in update); + expect(persisted).toHaveLength(2); + for (const update of persisted) { + expect(JSON.parse(update.placementExplanationJson as string)).toMatchObject({ + outcome: 'provisioned', + selectedNodeId: 'node-new', + provisioningAttempts: [{ outcome: 'started' }, { outcome: 'succeeded' }], + }); + } + expect(mocks.provisionNode).toHaveBeenCalledWith('node-new', expect.anything()); + expect(mocks.waitForNodeAgentReady).toHaveBeenCalledWith('node-new', expect.anything()); + expect(mocks.scheduleWorkspaceCreateOnNode).toHaveBeenCalledOnce(); + expect(events).toEqual(['workspace:persist', 'task:persist', 'schedule']); + }); +}); diff --git a/apps/api/tests/unit/routes/mcp-workspace-placement.test.ts b/apps/api/tests/unit/routes/mcp-workspace-placement.test.ts new file mode 100644 index 0000000000..7027e14c13 --- /dev/null +++ b/apps/api/tests/unit/routes/mcp-workspace-placement.test.ts @@ -0,0 +1,148 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +const mocks = vi.hoisted(() => ({ + drizzle: vi.fn(), + fetchNodeAgent: vi.fn(), + signTerminalToken: vi.fn(), +})); + +vi.mock('drizzle-orm/d1', () => ({ drizzle: mocks.drizzle })); +vi.mock('../../../src/services/jwt', () => ({ + signNodeManagementToken: vi.fn(), + signPortAccessToken: vi.fn(), + signTerminalToken: mocks.signTerminalToken, +})); +vi.mock('../../../src/services/node-agent', () => ({ fetchNodeAgent: mocks.fetchNodeAgent })); + +import { + enrichWorkspaceInfoWithPlacement, + handleGetWorkspaceInfo, +} from '../../../src/routes/mcp/workspace-tools'; + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe('get_workspace_info placement enrichment', () => { + it('adds a concise summary and typed detail without exposing unrelated canaries', () => { + const canary = 'CANARY_SECRET'; + const raw = JSON.stringify({ + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: 'node-1', + summary: 'Reused a compatible node.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + rawAgentVersion: canary, + providerError: canary, + }); + + const result = enrichWorkspaceInfoWithPlacement({ id: 'workspace-1' }, raw); + const serialized = JSON.stringify(result); + + expect(result.placement).toMatchObject({ summary: 'Reused a compatible node.' }); + expect(serialized).not.toContain(canary); + expect(serialized).not.toContain('rawAgentVersion'); + expect(serialized).not.toContain('providerError'); + }); + + it('returns placement null for malformed records', () => { + expect(enrichWorkspaceInfoWithPlacement('legacy-vm-result', '{bad json')).toEqual({ + workspaceInfo: 'legacy-vm-result', + placement: null, + }); + }); + + it('reads D1 placement, proxies VM metadata, and returns one enriched MCP result', async () => { + const placementExplanationJson = JSON.stringify({ + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: 'node-1', + summary: 'Reused a compatible node.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + }); + const d1Rows = [ + { + id: 'workspace-1', + status: 'running', + nodeId: 'node-1', + projectId: 'project-1', + placementExplanationJson, + }, + ]; + mocks.drizzle.mockReturnValue({ + select: vi.fn(() => ({ + from: vi.fn(() => ({ + where: vi.fn(() => ({ limit: vi.fn(async () => d1Rows) })), + })), + })), + }); + mocks.signTerminalToken.mockResolvedValue({ token: 'terminal-token' }); + mocks.fetchNodeAgent.mockResolvedValue( + Response.json({ id: 'workspace-1', branch: 'main', vmSize: 'medium' }) + ); + + const response = await handleGetWorkspaceInfo( + 'request-1', + { + taskId: 'task-1', + projectId: 'project-1', + userId: 'user-1', + workspaceId: 'workspace-1', + createdAt: '2026-08-11T00:00:00.000Z', + }, + { + DATABASE: {}, + BASE_DOMAIN: 'example.invalid', + VM_AGENT_PROTOCOL: 'https', + VM_AGENT_PORT: '8443', + } as never + ); + const text = (response.result as { content: Array<{ text: string }> }).content[0]?.text; + const result = JSON.parse(text ?? '{}'); + + expect(result).toMatchObject({ + id: 'workspace-1', + branch: 'main', + placement: { + summary: 'Reused a compatible node.', + detail: { schemaVersion: 2, selectedNodeId: 'node-1' }, + }, + }); + expect(mocks.fetchNodeAgent).toHaveBeenCalledWith( + 'node-1', + expect.anything(), + 'https://node-1.vm.example.invalid:8443/workspaces/workspace-1/mcp/workspace-info', + expect.objectContaining({ + method: 'GET', + headers: { Authorization: 'Bearer terminal-token' }, + }), + 15_000 + ); + }); +}); diff --git a/apps/api/tests/unit/schemas/placement-inputs.test.ts b/apps/api/tests/unit/schemas/placement-inputs.test.ts new file mode 100644 index 0000000000..32e4b76719 --- /dev/null +++ b/apps/api/tests/unit/schemas/placement-inputs.test.ts @@ -0,0 +1,75 @@ +import { safeParse } from 'valibot'; +import { describe, expect, it } from 'vitest'; + +import { UpdateProjectSchema } from '../../../src/schemas/projects'; +import { RunTaskSchema, SubmitTaskSchema } from '../../../src/schemas/tasks'; +import { CreateWorkspaceSchema } from '../../../src/schemas/workspaces'; + +const validNodeId = '01KZR2JAP92AK3SKW951E4H21M'; + +describe('placement request boundaries', () => { + it.each([SubmitTaskSchema, RunTaskSchema])( + 'accepts canonical task placement inputs', + (schema) => { + expect( + safeParse(schema, { message: 'run', nodeId: validNodeId, vmLocation: 'us-central1-a' }) + .success + ).toBe(true); + } + ); + + it('accepts canonical manual workspace placement inputs', () => { + expect( + safeParse(CreateWorkspaceSchema, { + name: 'workspace', + projectId: 'project-1', + nodeId: validNodeId, + vmLocation: 'hel1', + }).success + ).toBe(true); + }); + + it.each(['not-a-node-id', `${validNodeId}CANARY_SECRET`, `01KZR2JAP92AK3SKW951E4H2\n`])( + 'rejects non-canonical node identifier %j', + (nodeId) => { + expect(safeParse(SubmitTaskSchema, { message: 'run', nodeId }).success).toBe(false); + expect( + safeParse(CreateWorkspaceSchema, { name: 'workspace', projectId: 'project-1', nodeId }) + .success + ).toBe(false); + } + ); + + it.each(['', 'hel1\nCANARY_SECRET', 'x'.repeat(65)])( + 'rejects unsafe VM location %j', + (vmLocation) => { + expect(safeParse(SubmitTaskSchema, { message: 'run', vmLocation }).success).toBe(false); + expect( + safeParse(CreateWorkspaceSchema, { + name: 'workspace', + projectId: 'project-1', + vmLocation, + }).success + ).toBe(false); + } + ); + + it('accepts large positive project limits without undocumented ceilings', () => { + expect( + safeParse(UpdateProjectSchema, { + maxWorkspacesPerNode: 20000, + nodeCpuThresholdPercent: 100, + nodeMemoryThresholdPercent: 0, + }).success + ).toBe(true); + }); + + it.each([ + { maxWorkspacesPerNode: 0 }, + { maxWorkspacesPerNode: 1.5 }, + { nodeCpuThresholdPercent: -1 }, + { nodeMemoryThresholdPercent: 101 }, + ])('rejects invalid project placement limit %j', (input) => { + expect(safeParse(UpdateProjectSchema, input).success).toBe(false); + }); +}); diff --git a/apps/api/tests/unit/services/configurable-limits.test.ts b/apps/api/tests/unit/services/configurable-limits.test.ts index 6a02c761aa..afea0f10f2 100644 --- a/apps/api/tests/unit/services/configurable-limits.test.ts +++ b/apps/api/tests/unit/services/configurable-limits.test.ts @@ -101,11 +101,15 @@ describe('getRuntimeLimits', () => { }); it('respects MAX_AGENT_SESSIONS_PER_WORKSPACE', () => { - expect(getRuntimeLimits({ MAX_AGENT_SESSIONS_PER_WORKSPACE: '5' }).maxAgentSessionsPerWorkspace).toBe(5); + expect( + getRuntimeLimits({ MAX_AGENT_SESSIONS_PER_WORKSPACE: '5' }).maxAgentSessionsPerWorkspace + ).toBe(5); }); it('respects NODE_HEARTBEAT_STALE_SECONDS', () => { - expect(getRuntimeLimits({ NODE_HEARTBEAT_STALE_SECONDS: '300' }).nodeHeartbeatStaleSeconds).toBe(300); + expect( + getRuntimeLimits({ NODE_HEARTBEAT_STALE_SECONDS: '300' }).nodeHeartbeatStaleSeconds + ).toBe(300); }); it('respects MAX_PROJECTS_PER_USER', () => { @@ -117,11 +121,15 @@ describe('getRuntimeLimits', () => { }); it('respects MAX_TASK_DEPENDENCIES_PER_TASK', () => { - expect(getRuntimeLimits({ MAX_TASK_DEPENDENCIES_PER_TASK: '100' }).maxTaskDependenciesPerTask).toBe(100); + expect( + getRuntimeLimits({ MAX_TASK_DEPENDENCIES_PER_TASK: '100' }).maxTaskDependenciesPerTask + ).toBe(100); }); it('respects TASK_LIST_DEFAULT_PAGE_SIZE', () => { - expect(getRuntimeLimits({ TASK_LIST_DEFAULT_PAGE_SIZE: '25' }).taskListDefaultPageSize).toBe(25); + expect(getRuntimeLimits({ TASK_LIST_DEFAULT_PAGE_SIZE: '25' }).taskListDefaultPageSize).toBe( + 25 + ); }); it('respects TASK_LIST_MAX_PAGE_SIZE', () => { @@ -129,31 +137,50 @@ describe('getRuntimeLimits', () => { }); it('respects MAX_PROJECT_RUNTIME_ENV_VARS_PER_PROJECT', () => { - expect(getRuntimeLimits({ MAX_PROJECT_RUNTIME_ENV_VARS_PER_PROJECT: '300' }).maxProjectRuntimeEnvVarsPerProject).toBe(300); + expect( + getRuntimeLimits({ MAX_PROJECT_RUNTIME_ENV_VARS_PER_PROJECT: '300' }) + .maxProjectRuntimeEnvVarsPerProject + ).toBe(300); }); it('respects MAX_PROJECT_RUNTIME_FILES_PER_PROJECT', () => { - expect(getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILES_PER_PROJECT: '100' }).maxProjectRuntimeFilesPerProject).toBe(100); + expect( + getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILES_PER_PROJECT: '100' }) + .maxProjectRuntimeFilesPerProject + ).toBe(100); }); it('respects MAX_PROJECT_RUNTIME_ENV_VALUE_BYTES', () => { - expect(getRuntimeLimits({ MAX_PROJECT_RUNTIME_ENV_VALUE_BYTES: '16384' }).maxProjectRuntimeEnvValueBytes).toBe(16384); + expect( + getRuntimeLimits({ MAX_PROJECT_RUNTIME_ENV_VALUE_BYTES: '16384' }) + .maxProjectRuntimeEnvValueBytes + ).toBe(16384); }); it('respects MAX_PROJECT_RUNTIME_FILE_CONTENT_BYTES', () => { - expect(getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILE_CONTENT_BYTES: '262144' }).maxProjectRuntimeFileContentBytes).toBe(262144); + expect( + getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILE_CONTENT_BYTES: '262144' }) + .maxProjectRuntimeFileContentBytes + ).toBe(262144); }); it('respects MAX_PROJECT_RUNTIME_FILE_PATH_LENGTH', () => { - expect(getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILE_PATH_LENGTH: '512' }).maxProjectRuntimeFilePathLength).toBe(512); + expect( + getRuntimeLimits({ MAX_PROJECT_RUNTIME_FILE_PATH_LENGTH: '512' }) + .maxProjectRuntimeFilePathLength + ).toBe(512); }); it('respects TASK_CALLBACK_TIMEOUT_MS', () => { - expect(getRuntimeLimits({ TASK_CALLBACK_TIMEOUT_MS: '30000' }).taskCallbackTimeoutMs).toBe(30000); + expect(getRuntimeLimits({ TASK_CALLBACK_TIMEOUT_MS: '30000' }).taskCallbackTimeoutMs).toBe( + 30000 + ); }); it('respects TASK_CALLBACK_RETRY_MAX_ATTEMPTS', () => { - expect(getRuntimeLimits({ TASK_CALLBACK_RETRY_MAX_ATTEMPTS: '5' }).taskCallbackRetryMaxAttempts).toBe(5); + expect( + getRuntimeLimits({ TASK_CALLBACK_RETRY_MAX_ATTEMPTS: '5' }).taskCallbackRetryMaxAttempts + ).toBe(5); }); }); @@ -163,7 +190,9 @@ describe('getRuntimeLimits', () => { describe('invalid env values use defaults', () => { it('ignores non-numeric string', () => { - expect(getRuntimeLimits({ MAX_PROJECTS_PER_USER: 'not-a-number' }).maxProjectsPerUser).toBe(100); + expect(getRuntimeLimits({ MAX_PROJECTS_PER_USER: 'not-a-number' }).maxProjectsPerUser).toBe( + 100 + ); }); it('ignores zero', () => { @@ -175,7 +204,9 @@ describe('getRuntimeLimits', () => { }); it('ignores empty string', () => { - expect(getRuntimeLimits({ NODE_HEARTBEAT_STALE_SECONDS: '' }).nodeHeartbeatStaleSeconds).toBe(180); + expect(getRuntimeLimits({ NODE_HEARTBEAT_STALE_SECONDS: '' }).nodeHeartbeatStaleSeconds).toBe( + 180 + ); }); }); }); @@ -211,10 +242,7 @@ describe('DEFAULT_RATE_LIMITS', () => { // ============================================================================= describe('task submit — configurable MAX_TASK_MESSAGE_LENGTH', () => { - const submitSource = readFileSync( - resolve(process.cwd(), 'src/routes/tasks/submit.ts'), - 'utf8' - ); + const submitSource = readFileSync(resolve(process.cwd(), 'src/routes/tasks/submit.ts'), 'utf8'); it('reads max message length from MAX_TASK_MESSAGE_LENGTH env var', () => { expect(submitSource).toContain('c.env.MAX_TASK_MESSAGE_LENGTH'); @@ -231,7 +259,9 @@ describe('task submit — configurable MAX_TASK_MESSAGE_LENGTH', () => { it('falls back to default when env var is absent', () => { // Uses parsePositiveInt helper for safe fallback - expect(submitSource).toContain('parsePositiveInt(c.env.MAX_TASK_MESSAGE_LENGTH, DEFAULT_MAX_MESSAGE_LENGTH)'); + expect(submitSource).toContain( + 'parsePositiveInt(c.env.MAX_TASK_MESSAGE_LENGTH, DEFAULT_MAX_MESSAGE_LENGTH)' + ); }); it('error message references the configurable limit variable', () => { @@ -283,7 +313,9 @@ describe('workspace messages — configurable MAX_MESSAGES_PAYLOAD_BYTES', () => it('defaults to 256*1024 (256 KB) when env var is absent', () => { expect(runtimeSource).toContain('DEFAULT_MAX_MESSAGES_PAYLOAD_BYTES = 256 * 1024'); - expect(runtimeSource).toContain('parsePositiveInt(\n c.env.MAX_MESSAGES_PAYLOAD_BYTES as string,\n DEFAULT_MAX_MESSAGES_PAYLOAD_BYTES\n )'); + expect(runtimeSource).toContain( + 'parsePositiveInt(\n c.env.MAX_MESSAGES_PAYLOAD_BYTES as string,\n DEFAULT_MAX_MESSAGES_PAYLOAD_BYTES\n )' + ); }); it('uses configurable maxPayloadBytes in the comparison', () => { @@ -310,7 +342,9 @@ describe('ACP sessions — configurable MAX_ACP_PROMPT_BYTES', () => { }); it('uses configurable maxPromptBytes in the comparison', () => { - expect(acpSource).toContain('new TextEncoder().encode(body.initialPrompt).length > maxPromptBytes'); + expect(acpSource).toContain( + 'new TextEncoder().encode(body.initialPrompt).length > maxPromptBytes' + ); }); it('error message interpolates the configurable limit', () => { @@ -339,11 +373,15 @@ describe('ACP sessions fork — configurable MAX_ACP_CONTEXT_BYTES', () => { }); it('uses configurable maxContextBytes in the comparison', () => { - expect(acpSource).toContain('new TextEncoder().encode(body.contextSummary).length > maxContextBytes'); + expect(acpSource).toContain( + 'new TextEncoder().encode(body.contextSummary).length > maxContextBytes' + ); }); it('error message interpolates the configurable limit', () => { - expect(acpSource).toContain('`contextSummary exceeds maximum size of ${maxContextBytes} bytes`'); + expect(acpSource).toContain( + '`contextSummary exceeds maximum size of ${maxContextBytes} bytes`' + ); }); }); @@ -410,7 +448,9 @@ describe('agent sessions — configurable MAX_AGENT_SESSION_LABEL_LENGTH', () => }); it('defaults label max length to 50 when env var is absent', () => { - expect(agentSessionsSource).toContain('parsePositiveInt(c.env.MAX_AGENT_SESSION_LABEL_LENGTH, 50)'); + expect(agentSessionsSource).toContain( + 'parsePositiveInt(c.env.MAX_AGENT_SESSION_LABEL_LENGTH, 50)' + ); }); it('uses configurable maxLabelLength in slice (not hardcoded 50)', () => { @@ -423,10 +463,7 @@ describe('agent sessions — configurable MAX_AGENT_SESSION_LABEL_LENGTH', () => // ============================================================================= describe('Env interface — new configurable limit env vars', () => { - const indexSource = readFileSync( - resolve(process.cwd(), 'src/env.ts'), - 'utf8' - ); + const indexSource = readFileSync(resolve(process.cwd(), 'src/env.ts'), 'utf8'); it('declares MAX_TASK_MESSAGE_LENGTH in Env', () => { expect(indexSource).toContain('MAX_TASK_MESSAGE_LENGTH'); @@ -474,10 +511,7 @@ describe('Env interface — new configurable limit env vars', () => { // ============================================================================= describe('workspace create — count limit removed', () => { - const crudSource = readFileSync( - resolve(process.cwd(), 'src/routes/workspaces/crud.ts'), - 'utf8' - ); + const crudSource = readFileSync(resolve(process.cwd(), 'src/routes/workspaces/crud.ts'), 'utf8'); it('still counts active workspaces per node (for telemetry)', () => { expect(crudSource).toContain('nodeWorkspaceCount'); @@ -490,7 +524,9 @@ describe('workspace create — count limit removed', () => { it('keeps the count query filtered to active statuses', () => { // Count query still filters by active statuses — just no longer used for enforcement - expect(crudSource).toContain("inArray(schema.workspaces.status, ['running', 'creating', 'recovery'])"); + expect(crudSource).toContain( + "inArray(schema.workspaces.status, ['running', 'creating', 'recovery'])" + ); }); }); @@ -498,7 +534,7 @@ describe('workspace create — count limit removed', () => { // Source contract: task-runner DO enforces workspace count limit // ============================================================================= -describe('task-runner DO — workspace count limit', () => { +describe('canonical placement — workspace count limit', () => { const doSource = [ 'index.ts', 'types.ts', @@ -508,23 +544,29 @@ describe('task-runner DO — workspace count limit', () => { 'agent-session-step.ts', 'state-machine.ts', 'helpers.ts', - ].map(f => readFileSync(resolve(process.cwd(), 'src/durable-objects/task-runner', f), 'utf8')).join('\n'); + ] + .map((f) => readFileSync(resolve(process.cwd(), 'src/durable-objects/task-runner', f), 'utf8')) + .join('\n'); + const placementSource = [ + readFileSync(resolve(process.cwd(), 'src/services/node-selector.ts'), 'utf8'), + readFileSync(resolve(process.cwd(), 'src/services/placement-explanation.ts'), 'utf8'), + ].join('\n'); it('references MAX_WORKSPACES_PER_NODE env var', () => { - expect(doSource).toContain('MAX_WORKSPACES_PER_NODE'); + expect(placementSource).toContain('MAX_WORKSPACES_PER_NODE'); }); it('references DEFAULT_MAX_WORKSPACES_PER_NODE constant', () => { - expect(doSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); + expect(placementSource).toContain('DEFAULT_MAX_WORKSPACES_PER_NODE'); }); it('still reads CPU and memory thresholds from env', () => { - expect(doSource).toContain('TASK_RUN_NODE_CPU_THRESHOLD_PERCENT'); - expect(doSource).toContain('TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT'); + expect(placementSource).toContain('TASK_RUN_NODE_CPU_THRESHOLD_PERCENT'); + expect(placementSource).toContain('TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT'); }); it('queries workspace count per node for limit enforcement', () => { - const section = doSource.slice(doSource.indexOf('findNodeWithCapacity')); - expect(section).toContain('>= maxWorkspaces'); + expect(placementSource).toContain('node.activeWorkspaceCount >= request.maxWorkspacesPerNode'); + expect(doSource).toContain('selectNodeWithExplanation'); }); }); diff --git a/apps/api/tests/unit/services/node-selector-user-scope.test.ts b/apps/api/tests/unit/services/node-selector-user-scope.test.ts index ca41d0f224..54857ac770 100644 --- a/apps/api/tests/unit/services/node-selector-user-scope.test.ts +++ b/apps/api/tests/unit/services/node-selector-user-scope.test.ts @@ -72,11 +72,18 @@ function seedHealthyNode(id: string, userId: string): void { sqlite ?.prepare( ` - INSERT INTO nodes (id, user_id, name, status, vm_size, vm_location, health_status, last_metrics, node_role, runtime, node_class) - VALUES (?, ?, ?, 'running', 'medium', 'nbg1', 'healthy', ?, 'workspace', 'vm', 'managed') + INSERT INTO nodes (id, user_id, name, status, vm_size, vm_location, health_status, last_metrics, last_heartbeat_at, agent_ready_at, node_role, runtime, node_class) + VALUES (?, ?, ?, 'running', 'medium', 'nbg1', 'healthy', ?, ?, ?, 'workspace', 'vm', 'managed') ` ) - .run(id, userId, `node-${id}`, JSON.stringify({ cpuLoadAvg1: 0.1, memoryPercent: 10 })); + .run( + id, + userId, + `node-${id}`, + JSON.stringify({ cpuLoadAvg1: 0.1, memoryPercent: 10 }), + new Date().toISOString(), + new Date().toISOString() + ); } // No warm-pool path: taskId omitted so step 0 is skipped and NODE_LIFECYCLE is never touched. diff --git a/apps/api/tests/unit/services/node-selector.test.ts b/apps/api/tests/unit/services/node-selector.test.ts index a6c39a84e4..fad1b6d9ff 100644 --- a/apps/api/tests/unit/services/node-selector.test.ts +++ b/apps/api/tests/unit/services/node-selector.test.ts @@ -17,6 +17,7 @@ import { nodeHasCapacity, scoreNodeLoad, selectNodeForTaskRun, + selectNodeWithExplanation, } from '../../../src/services/node-selector'; vi.mock('../../../src/services/node-lifecycle', () => ({ @@ -31,42 +32,52 @@ type MockNode = { vmSize: string; vmLocation: string; lastMetrics: string | null; - warmSince?: number | null; + runtime: string; + lastHeartbeatAt: string; + agentReadyAt: string; + agentVersion: string | null; + warmSince: string | null; }; function createMockDb({ nodes, warmNodes = [], workspaceCount = 0, + freshNodes, + freshWorkspaceCount, }: { nodes: MockNode[]; warmNodes?: MockNode[]; workspaceCount?: number; + freshNodes?: MockNode[]; + freshWorkspaceCount?: number; }) { + let nodeReadCount = 0; + let workspaceReadCount = 0; return { - select(selection?: Record) { + select(_selection?: Record) { return { from(table: unknown) { return { where() { if (table === schema.workspaces) { - return Promise.resolve([{ count: workspaceCount }]); + const count = + workspaceReadCount++ > 0 && freshWorkspaceCount !== undefined + ? freshWorkspaceCount + : workspaceCount; + return Promise.resolve( + Array.from({ length: count }, (_, index) => ({ + id: `workspace-${index}`, + nodeId: (nodes[0] ?? warmNodes[0])?.id ?? null, + })) + ); } if (table === schema.nodes) { - if (selection && 'warmSince' in selection && 'id' in selection) { - return Promise.resolve(warmNodes); - } - - if (selection && 'warmSince' in selection && 'status' in selection) { - return { - limit() { - return Promise.resolve([{ status: 'running', warmSince: Date.now() }]); - }, - }; - } - - return Promise.resolve(nodes); + const initialNodes = nodes.length > 0 ? nodes : warmNodes; + return Promise.resolve( + nodeReadCount++ > 0 && freshNodes !== undefined ? freshNodes : initialNodes + ); } return Promise.resolve([]); @@ -87,6 +98,10 @@ function node(overrides: Partial): MockNode { vmSize: 'medium', vmLocation: 'fsn1', lastMetrics: JSON.stringify({ cpuLoadAvg1: 5, memoryPercent: 10 }), + runtime: 'vm', + lastHeartbeatAt: new Date().toISOString(), + agentReadyAt: new Date().toISOString(), + agentVersion: null, warmSince: null, ...overrides, }; @@ -113,32 +128,24 @@ describe('scoreNodeLoad', () => { expect(scoreNodeLoad({ cpuLoadAvg1: 100, memoryPercent: 100 })).toBe(100); }); - it('applies 40% CPU + 60% memory weighting', () => { - // 50% CPU * 0.4 = 20, 80% mem * 0.6 = 48, total = 68 - expect(scoreNodeLoad({ cpuLoadAvg1: 50, memoryPercent: 80 })).toBe(68); + it('uses the most saturated resource', () => { + expect(scoreNodeLoad({ cpuLoadAvg1: 50, memoryPercent: 80 })).toBe(80); }); - it('weights memory higher than CPU', () => { - // High CPU, low memory + it('treats CPU and memory saturation symmetrically', () => { const cpuHeavy = scoreNodeLoad({ cpuLoadAvg1: 90, memoryPercent: 10 }); - // Low CPU, high memory const memHeavy = scoreNodeLoad({ cpuLoadAvg1: 10, memoryPercent: 90 }); - // 90*0.4 + 10*0.6 = 36+6 = 42 - expect(cpuHeavy).toBe(42); - // 10*0.4 + 90*0.6 = 4+54 = 58 - expect(memHeavy).toBe(58); - - // Memory-heavy node should score higher (more loaded) - expect(memHeavy).toBeGreaterThan(cpuHeavy!); + expect(cpuHeavy).toBe(90); + expect(memHeavy).toBe(90); }); it('treats missing cpuLoadAvg1 as 0', () => { - expect(scoreNodeLoad({ memoryPercent: 50 })).toBe(30); // 0*0.4 + 50*0.6 + expect(scoreNodeLoad({ memoryPercent: 50 })).toBe(50); }); it('treats missing memoryPercent as 0', () => { - expect(scoreNodeLoad({ cpuLoadAvg1: 50 })).toBe(20); // 50*0.4 + 0*0.6 + expect(scoreNodeLoad({ cpuLoadAvg1: 50 })).toBe(50); }); it('treats both missing as 0', () => { @@ -148,15 +155,13 @@ describe('scoreNodeLoad', () => { it('handles fractional values', () => { const score = scoreNodeLoad({ cpuLoadAvg1: 33.5, memoryPercent: 67.2 }); - // 33.5*0.4 + 67.2*0.6 = 13.4 + 40.32 = 53.72 - expect(score).toBeCloseTo(53.72, 2); + expect(score).toBeCloseTo(67.2, 2); }); it('handles values above 100 (overloaded node)', () => { // CPU load average can exceed 100% on multi-core systems const score = scoreNodeLoad({ cpuLoadAvg1: 200, memoryPercent: 95 }); - // 200*0.4 + 95*0.6 = 80 + 57 = 137 - expect(score).toBe(137); + expect(score).toBe(200); }); }); @@ -362,6 +367,46 @@ describe('parseMetrics via selectNodeForTaskRun.lastMetrics', () => { // ============================================================================= describe('selectNodeForTaskRun VM size minimum behavior', () => { + it('reuses a healthy compatible medium hel1 node with 2 of 5 workspaces and low load', async () => { + const requiredVersion = '2'.repeat(40); + const db = createMockDb({ + nodes: [ + node({ + id: '01KZR2JAP92AK3SKW951E4H21M', + vmSize: 'medium', + vmLocation: 'hel1', + agentVersion: requiredVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 3, memoryPercent: 18 }), + }), + ], + workspaceCount: 2, + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { VM_AGENT_REQUIRED_VERSION: requiredVersion }, + { + vmSize: 'medium', + vmLocation: 'hel1', + limits: { maxWorkspacesPerNode: 5 }, + } + ); + + expect(result.node?.id).toBe('01KZR2JAP92AK3SKW951E4H21M'); + expect(result.explanation).toMatchObject({ + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: '01KZR2JAP92AK3SKW951E4H21M', + evaluatedNodes: [ + { + accepted: true, + snapshot: { activeWorkspaceCount: 2, agentVersionCompatible: true }, + }, + ], + }); + }); + it('rejects smaller regular nodes for larger requested sizes', async () => { const db = createMockDb({ nodes: [ @@ -375,6 +420,274 @@ describe('selectNodeForTaskRun VM size minimum behavior', () => { expect(selected).toBeNull(); }); + it('records and excludes a warm node after a concurrent claim loss', async () => { + vi.mocked(nodeLifecycle.tryClaim).mockResolvedValue({ claimed: false } as never); + const db = createMockDb({ + nodes: [node({ id: 'warm-lost', warmSince: new Date().toISOString() })], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { NODE_LIFECYCLE: {} as DurableObjectNamespace }, + { vmSize: 'medium', vmLocation: 'fsn1', taskId: 'task-1' } + ); + + expect(result.node).toBeNull(); + expect(result.explanation.outcome).toBe('provisioned'); + expect( + result.explanation.evaluatedNodes.filter((evaluation) => + evaluation.rejectionReasons.includes('warm-claim-lost') + ) + ).toHaveLength(2); + }); + + it.each([ + ['status', { status: 'stopped' }, undefined, 'not-running', {}], + [ + 'heartbeat', + { lastHeartbeatAt: new Date(0).toISOString() }, + undefined, + 'heartbeat-stale', + { heartbeatAgeSeconds: expect.any(Number) }, + ], + [ + 'agent version', + { agentVersion: 'b'.repeat(40) }, + undefined, + 'agent-version-mismatch', + { agentVersionCompatible: false }, + ], + ['workspace count', {}, 5, 'workspace-limit', { activeWorkspaceCount: 5 }], + ] as const)( + 'does not consume the warm claim when the fresh %s check rejects the node', + async (_change, freshOverrides, freshWorkspaceCount, expectedReason, expectedSnapshot) => { + const requiredVersion = 'a'.repeat(40); + const initialNode = node({ + id: 'warm-changed', + warmSince: new Date().toISOString(), + agentVersion: requiredVersion, + }); + const db = createMockDb({ + nodes: [initialNode], + freshNodes: [node({ ...initialNode, ...freshOverrides })], + freshWorkspaceCount, + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { + VM_AGENT_REQUIRED_VERSION: requiredVersion, + NODE_LIFECYCLE: {} as DurableObjectNamespace, + }, + { + vmSize: 'medium', + vmLocation: 'fsn1', + taskId: 'task-1', + limits: { maxWorkspacesPerNode: 5 }, + } + ); + + expect(nodeLifecycle.tryClaim).not.toHaveBeenCalled(); + expect(result.node).toBeNull(); + expect(result.explanation.evaluatedNodes[0]?.rejectionReasons).toContain(expectedReason); + expect(result.explanation.evaluatedNodes[0]?.rejectionReasons).not.toContain( + 'warm-claim-lost' + ); + expect(result.explanation.evaluatedNodes[0]?.snapshot).toMatchObject(expectedSnapshot); + } + ); + + it('selects a compatible node over a lower-load incompatible node', async () => { + const requiredVersion = 'a'.repeat(40); + const db = createMockDb({ + nodes: [ + node({ + id: 'incompatible-idle', + agentVersion: 'b'.repeat(40), + lastMetrics: JSON.stringify({ cpuLoadAvg1: 0, memoryPercent: 0 }), + }), + node({ + id: 'compatible-busier', + agentVersion: requiredVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 30, memoryPercent: 30 }), + }), + ], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { VM_AGENT_REQUIRED_VERSION: requiredVersion }, + { vmSize: 'medium', vmLocation: 'fsn1' } + ); + + expect(result.node?.id).toBe('compatible-busier'); + expect( + result.explanation.evaluatedNodes.find((evaluation) => + evaluation.rejectionReasons.includes('agent-version-mismatch') + )?.rejectionReasons + ).toContain('agent-version-mismatch'); + expect(JSON.stringify(result.explanation)).not.toContain('incompatible-idle'); + expect(result.explanation.evaluatedNodes[0]?.nodeId).toBe('candidate-1'); + }); + + it('never persists a rejected request node ID or unsafe location canary', async () => { + const canary = 'CANARY_SECRET\n'.repeat(20); + const result = await selectNodeWithExplanation( + createMockDb({ nodes: [] }) as never, + 'user-1', + {}, + { + vmSize: 'medium', + vmLocation: canary, + preferredNodeId: canary, + preferredOnly: true, + } + ); + + const serialized = JSON.stringify(result.explanation); + expect(result.node).toBeNull(); + expect(result.explanation.summary).toBe('The preferred node was rejected.'); + expect(result.explanation.request.vmLocation).toBe('unknown'); + expect(result.explanation.evaluatedNodes[0]?.nodeId).toBe('candidate-1'); + expect(serialized).not.toContain('CANARY_SECRET'); + }); + + it.each([ + ['capacity', undefined], + ['trial', 'trial'], + ] as const)( + 'keeps only the selected real node ID on the %s path', + async (_path, selectionPath) => { + const db = createMockDb({ + nodes: [ + node({ id: 'selected-node', lastMetrics: JSON.stringify({ cpuLoadAvg1: 1 }) }), + node({ id: 'other-eligible-node', lastMetrics: JSON.stringify({ cpuLoadAvg1: 10 }) }), + ], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + {}, + { + vmSize: 'medium', + vmLocation: 'fsn1', + ...(selectionPath ? { selectionPath } : {}), + } + ); + + expect(result.node?.id).toBe('selected-node'); + expect(result.explanation.selectedNodeId).toBe('selected-node'); + expect(JSON.stringify(result.explanation)).not.toContain('other-eligible-node'); + expect(result.explanation.evaluatedNodes.map((evaluation) => evaluation.nodeId)).toEqual([ + 'selected-node', + 'candidate-1', + ]); + } + ); + + it('keeps only the claimed real node ID when several warm nodes qualify', async () => { + vi.mocked(nodeLifecycle.tryClaim).mockResolvedValue({ claimed: true }); + const warmSince = new Date().toISOString(); + const db = createMockDb({ + nodes: [ + node({ + id: 'selected-warm-node', + warmSince, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 1 }), + }), + node({ + id: 'other-eligible-warm-node', + warmSince, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 10 }), + }), + ], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { NODE_LIFECYCLE: {} as DurableObjectNamespace }, + { vmSize: 'medium', vmLocation: 'fsn1', taskId: 'task-1' } + ); + + expect(result.node?.id).toBe('selected-warm-node'); + expect(result.explanation.selectedNodeId).toBe('selected-warm-node'); + expect(JSON.stringify(result.explanation)).not.toContain('other-eligible-warm-node'); + expect(result.explanation.evaluatedNodes.map((evaluation) => evaluation.nodeId)).toEqual([ + 'selected-warm-node', + 'candidate-1', + ]); + }); + + it.each([ + ['preferred', { preferredNodeId: 'incompatible-node' }], + [ + 'manual', + { preferredNodeId: 'incompatible-node', preferredOnly: true, selectionPath: 'manual' }, + ], + ['trial', { selectionPath: 'trial' }], + ] as const)('records exact incompatibility on the %s selection path', async (path, options) => { + const requiredVersion = 'c'.repeat(40); + const db = createMockDb({ + nodes: [node({ id: 'incompatible-node', agentVersion: 'd'.repeat(40) })], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { VM_AGENT_REQUIRED_VERSION: requiredVersion }, + { vmSize: 'medium', vmLocation: 'hel1', ...options } + ); + + expect(result.node).toBeNull(); + expect(result.explanation.evaluatedNodes).toEqual([ + expect.objectContaining({ + path, + accepted: false, + rejectionReasons: expect.arrayContaining(['agent-version-mismatch']), + }), + ]); + }); + + it('records exact incompatibility on both warm and capacity evaluations', async () => { + const requiredVersion = 'e'.repeat(40); + const db = createMockDb({ + nodes: [ + node({ + id: 'incompatible-warm', + agentVersion: 'f'.repeat(40), + warmSince: new Date().toISOString(), + }), + ], + }); + + const result = await selectNodeWithExplanation( + db as never, + 'user-1', + { + VM_AGENT_REQUIRED_VERSION: requiredVersion, + NODE_LIFECYCLE: {} as DurableObjectNamespace, + }, + { vmSize: 'medium', vmLocation: 'hel1', taskId: 'task-incompatible' } + ); + + expect(nodeLifecycle.tryClaim).not.toHaveBeenCalled(); + expect(result.explanation.evaluatedNodes).toEqual([ + expect.objectContaining({ + path: 'warm', + rejectionReasons: expect.arrayContaining(['agent-version-mismatch']), + }), + expect.objectContaining({ + path: 'capacity', + rejectionReasons: expect.arrayContaining(['agent-version-mismatch']), + }), + ]); + }); + it('allows larger regular nodes to satisfy smaller requested sizes', async () => { const db = createMockDb({ nodes: [ @@ -418,8 +731,8 @@ describe('selectNodeForTaskRun VM size minimum behavior', () => { const db = createMockDb({ nodes: [], warmNodes: [ - node({ id: 'warm-small', vmSize: 'small', warmSince: Date.now() }), - node({ id: 'warm-medium', vmSize: 'medium', warmSince: Date.now() }), + node({ id: 'warm-small', vmSize: 'small', warmSince: new Date().toISOString() }), + node({ id: 'warm-medium', vmSize: 'medium', warmSince: new Date().toISOString() }), ], }); @@ -440,7 +753,7 @@ describe('selectNodeForTaskRun VM size minimum behavior', () => { vi.mocked(nodeLifecycle.tryClaim).mockResolvedValue({ claimed: true }); const db = createMockDb({ nodes: [], - warmNodes: [node({ id: 'warm-large', vmSize: 'large', warmSince: Date.now() })], + warmNodes: [node({ id: 'warm-large', vmSize: 'large', warmSince: new Date().toISOString() })], }); const selected = await selectNodeForTaskRun( diff --git a/apps/api/tests/unit/services/placement-explanation.test.ts b/apps/api/tests/unit/services/placement-explanation.test.ts new file mode 100644 index 0000000000..f76f0a5254 --- /dev/null +++ b/apps/api/tests/unit/services/placement-explanation.test.ts @@ -0,0 +1,193 @@ +import { describe, expect, it } from 'vitest'; + +import { + appendProvisioningAttempt, + createPlacementExplanation, + evaluatePlacementNode, + resolvePlacementRequest, + sanitizePlacementExplanation, +} from '../../../src/services/placement-explanation'; + +const nowMs = Date.parse('2026-08-11T12:00:00.000Z'); +const requiredVersion = 'a'.repeat(40); +const request = resolvePlacementRequest( + { + VM_AGENT_REQUIRED_VERSION: requiredVersion, + MAX_WORKSPACES_PER_NODE: '5', + TASK_RUN_NODE_CPU_THRESHOLD_PERCENT: '50', + TASK_RUN_NODE_MEMORY_THRESHOLD_PERCENT: '60', + NODE_HEARTBEAT_STALE_SECONDS: '180', + }, + 'medium', + 'hel1' +); + +function node(overrides: Record = {}) { + return { + id: 'node-1', + status: 'running', + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + lastHeartbeatAt: '2026-08-11T11:59:30.000Z', + agentReadyAt: '2026-08-11T11:59:00.000Z', + agentVersion: requiredVersion, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 8, memoryPercent: 20 }), + activeWorkspaceCount: 2, + warmSince: '2026-08-11T11:00:00.000Z', + ...overrides, + } as Parameters[0]; +} + +describe('typed placement evaluation', () => { + it('accepts the production-like healthy medium/hel1 node at 2/5 and low load', () => { + const evaluation = evaluatePlacementNode( + node({ id: '01KZR2JAP92AK3SKW951E4H21M' }), + request, + 'capacity', + requiredVersion, + nowMs + ); + + expect(evaluation.accepted).toBe(true); + expect(evaluation.rejectionReasons).toEqual([]); + expect(evaluation.snapshot).toMatchObject({ + activeWorkspaceCount: 2, + cpuLoadAvg1: 8, + memoryPercent: 20, + agentVersionCompatible: true, + }); + }); + + it.each([ + ['not-running', { status: 'stopped' }], + ['wrong-runtime', { runtime: 'cf-container' }], + ['unhealthy', { healthStatus: 'unhealthy' }], + ['heartbeat-missing', { lastHeartbeatAt: null }], + ['heartbeat-stale', { lastHeartbeatAt: '2026-08-11T11:00:00.000Z' }], + ['agent-not-ready', { agentReadyAt: null }], + ['agent-version-mismatch', { agentVersion: 'b'.repeat(40) }], + ['undersized', { vmSize: 'small' }], + ['workspace-limit', { activeWorkspaceCount: 5 }], + ['cpu-threshold', { lastMetrics: JSON.stringify({ cpuLoadAvg1: 50, memoryPercent: 1 }) }], + ['memory-threshold', { lastMetrics: JSON.stringify({ cpuLoadAvg1: 1, memoryPercent: 60 }) }], + ['not-warm', { warmSince: null }], + ])('records %s without raw diagnostic data', (reason, overrides) => { + const evaluation = evaluatePlacementNode( + node(overrides), + request, + reason === 'not-warm' ? 'warm' : 'capacity', + requiredVersion, + nowMs + ); + + expect(evaluation.accepted).toBe(false); + expect(evaluation.rejectionReasons).toContain(reason); + }); + + it('keeps exact compatibility and excludes raw versions, metrics fields, and canaries', () => { + const canary = 'CANARY_SECRET_DO_NOT_PERSIST'; + const evaluation = evaluatePlacementNode( + node({ + agentVersion: `wrong-${canary}`, + lastMetrics: JSON.stringify({ + cpuLoadAvg1: 10, + memoryPercent: 20, + rawProcessList: canary, + providerError: canary, + }), + }), + request, + 'manual', + requiredVersion, + nowMs + ); + const serialized = JSON.stringify(evaluation); + + expect(evaluation.rejectionReasons).toContain('agent-version-mismatch'); + expect(serialized).not.toContain(canary); + expect(serialized).not.toContain(requiredVersion); + expect(serialized).not.toContain('rawProcessList'); + expect(serialized).not.toContain('providerError'); + }); + + it('records provisioning outcomes using only typed failure reasons', () => { + const explanation = appendProvisioningAttempt( + createPlacementExplanation(request, 'provisioning', '2026-08-11T12:00:00.000Z'), + { + vmSize: 'medium', + vmLocation: 'hel1', + outcome: 'failed', + failureReason: 'provider-failed', + }, + 'node-new', + '2026-08-11T12:01:00.000Z' + ); + + expect(explanation).toMatchObject({ + schemaVersion: 2, + outcome: 'failed', + selectedNodeId: 'node-new', + provisioningAttempts: [{ failureReason: 'provider-failed' }], + }); + }); + + it('preserves configured positive limits without hidden ceilings', () => { + const configured = resolvePlacementRequest( + { + MAX_WORKSPACES_PER_NODE: '20000', + NODE_HEARTBEAT_STALE_SECONDS: '172800', + }, + 'large', + 'us-central1-a' + ); + + expect(configured).toMatchObject({ + maxWorkspacesPerNode: 20000, + heartbeatStaleSeconds: 172800, + }); + }); + + it('keeps every candidate evaluation while aliasing non-selected node identifiers', () => { + const explanation = createPlacementExplanation(request, 'capacity'); + explanation.selectedNodeId = 'node-300'; + explanation.evaluatedNodes = Array.from({ length: 300 }, (_, index) => ({ + ...evaluatePlacementNode( + node({ id: `node-${index + 1}` }), + request, + 'capacity', + requiredVersion, + nowMs + ), + accepted: index === 299, + })); + + const sanitized = sanitizePlacementExplanation(explanation); + + expect(sanitized.evaluatedNodes).toHaveLength(300); + expect(sanitized.evaluatedNodes[0]?.nodeId).toBe('candidate-1'); + expect(sanitized.evaluatedNodes[299]?.nodeId).toBe('node-300'); + }); + + it('clears a rejected provisional node before trying a fallback size', () => { + const started = appendProvisioningAttempt( + createPlacementExplanation(request, 'provisioning', '2026-08-11T12:00:00.000Z'), + { vmSize: 'large', vmLocation: 'hel1', outcome: 'started' }, + 'node-rejected' + ); + const rejected = appendProvisioningAttempt( + started, + { + vmSize: 'large', + vmLocation: 'hel1', + outcome: 'capacity-rejected', + failureReason: 'capacity-unavailable', + }, + null + ); + + expect(rejected.selectedNodeId).toBeNull(); + expect(rejected.provisioningAttempts).toHaveLength(2); + }); +}); diff --git a/apps/api/tests/unit/task-runner-node-selection-runtime-guard.test.ts b/apps/api/tests/unit/task-runner-node-selection-runtime-guard.test.ts index c29bfc9831..8198a61821 100644 --- a/apps/api/tests/unit/task-runner-node-selection-runtime-guard.test.ts +++ b/apps/api/tests/unit/task-runner-node-selection-runtime-guard.test.ts @@ -1,54 +1,60 @@ -import { describe, expect, it, vi } from 'vitest'; +import { describe, expect, it } from 'vitest'; -import { handleNodeSelection } from '../../src/durable-objects/task-runner/node-steps'; -import type { TaskRunnerContext, TaskRunnerState } from '../../src/durable-objects/task-runner/types'; +import * as schema from '../../src/db/schema'; +import { selectNodeWithExplanation } from '../../src/services/node-selector'; // Task-runner node reuse must never select cf-container (instant-session) // nodes: the standalone vm-agent hosts exactly one lightweight workspace and // rejects task-runner re-dispatch (no `lightweight` flag) with a 409 profile -// conflict — the same class node-lifecycle.ts already guards against. These -// tests capture the SQL the selection step actually issues to D1 and assert -// both reuse queries carry the runtime exclusion. +// conflict. The canonical evaluator records this as a typed rejection on +// every reuse path, even if the row is present in the candidate query. -function makeCapturingDb(issuedSql: string[]) { +function makeDb() { + const now = new Date().toISOString(); + const cfContainerNode = { + id: 'instant-node', + status: 'running', + runtime: 'cf-container', + vmSize: 'small', + vmLocation: 'fsn1', + healthStatus: 'healthy', + lastHeartbeatAt: now, + agentReadyAt: now, + agentVersion: null, + lastMetrics: JSON.stringify({ cpuLoadAvg1: 1, memoryPercent: 1 }), + warmSince: now, + }; return { - prepare: (sql: string) => { - issuedSql.push(sql); - return { - bind: () => ({ - all: async () => ({ results: [] }), - first: async () => null, - }), - }; - }, + select: () => ({ + from: (table: unknown) => ({ + where: async () => (table === schema.nodes ? [cfContainerNode] : []), + }), + }), }; } -describe('handleNodeSelection runtime guards', () => { - it('excludes cf-container nodes from both warm-pool and capacity reuse queries', async () => { - const issuedSql: string[] = []; - const rc = { - env: { DATABASE: makeCapturingDb(issuedSql), NODE_LIFECYCLE: {} }, - updateD1ExecutionStep: vi.fn().mockResolvedValue(undefined), - advanceToStep: vi.fn().mockResolvedValue(undefined), - } as unknown as TaskRunnerContext; - const state = { - taskId: 'task-1', - userId: 'user-1', - config: { vmSize: 'small', vmLocation: 'fsn1' }, - stepResults: {}, - } as unknown as TaskRunnerState; - - await handleNodeSelection(state, rc); - - const warmSql = issuedSql.find((sql) => sql.includes('warm_since IS NOT NULL')); - const capacitySql = issuedSql.find((sql) => sql.includes("health_status != 'unhealthy'")); - expect(warmSql, 'warm-pool query was not issued').toBeDefined(); - expect(capacitySql, 'capacity reuse query was not issued').toBeDefined(); - expect(warmSql).toContain("runtime IS NULL OR runtime != 'cf-container'"); - expect(capacitySql).toContain("runtime IS NULL OR runtime != 'cf-container'"); +describe('TaskRunner runtime guards', () => { + it('rejects cf-container nodes on both warm and capacity reuse paths', async () => { + const result = await selectNodeWithExplanation( + makeDb() as never, + 'user-1', + { NODE_LIFECYCLE: {} as DurableObjectNamespace }, + { vmSize: 'small', vmLocation: 'fsn1', taskId: 'task-1' } + ); - // With no eligible nodes the step falls through to provisioning. - expect(rc.advanceToStep).toHaveBeenCalledWith(state, 'node_provisioning'); + expect(result.node).toBeNull(); + expect(result.explanation.outcome).toBe('provisioned'); + expect(result.explanation.evaluatedNodes).toEqual([ + expect.objectContaining({ + path: 'warm', + accepted: false, + rejectionReasons: expect.arrayContaining(['wrong-runtime']), + }), + expect.objectContaining({ + path: 'capacity', + accepted: false, + rejectionReasons: expect.arrayContaining(['wrong-runtime']), + }), + ]); }); }); diff --git a/apps/api/tests/unit/task-runner-provisioning-timeout.test.ts b/apps/api/tests/unit/task-runner-provisioning-timeout.test.ts index 4b817e6a9c..f1c3f2db0e 100644 --- a/apps/api/tests/unit/task-runner-provisioning-timeout.test.ts +++ b/apps/api/tests/unit/task-runner-provisioning-timeout.test.ts @@ -11,11 +11,20 @@ * Bug 2 (Suspenders): updateD1ExecutionStep refreshed updated_at on every poll cycle * even when the step hadn't changed, defeating the stuck-tasks cron's staleness detection. */ -import { DEFAULT_TASK_RUNNER_PROVISION_TIMEOUT_MS } from '@simple-agent-manager/shared'; +import { + DEFAULT_TASK_RUNNER_PROVISION_TIMEOUT_MS, + type PlacementExplanation, +} from '@simple-agent-manager/shared'; import { describe, expect, it, vi } from 'vitest'; -import { handleNodeAgentReady, handleNodeProvisioning } from '../../src/durable-objects/task-runner/node-steps'; -import type { TaskRunnerContext, TaskRunnerState } from '../../src/durable-objects/task-runner/types'; +import { + handleNodeAgentReady, + handleNodeProvisioning, +} from '../../src/durable-objects/task-runner/node-steps'; +import type { + TaskRunnerContext, + TaskRunnerState, +} from '../../src/durable-objects/task-runner/types'; // --------------------------------------------------------------------------- // Helpers @@ -79,6 +88,30 @@ function makeState(overrides: Partial = {}): TaskRunnerState { }; } +function makeProvisioningExplanation(): PlacementExplanation { + const now = new Date().toISOString(); + return { + schemaVersion: 2, + outcome: 'provisioned', + selectionPath: 'provisioning', + selectedNodeId: null, + summary: 'No reusable node was eligible; provisioning is required.', + request: { + runtime: 'vm', + vmSize: 'small', + vmLocation: 'fsn1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 80, + memoryThresholdPercent: 85, + heartbeatStaleSeconds: 120, + }, + evaluatedNodes: [], + provisioningAttempts: [{ vmSize: 'small', vmLocation: 'fsn1', outcome: 'started' }], + decidedAt: now, + updatedAt: now, + }; +} + function makeContext(overrides: Partial = {}): TaskRunnerContext { return { env: { @@ -119,6 +152,7 @@ describe('handleNodeProvisioning — timeout', () => { it('classifies a missing claimed node before the generic provisioning timeout', async () => { const state = makeState({ provisioningStartedAt: Date.now() - DEFAULT_TASK_RUNNER_PROVISION_TIMEOUT_MS - 1_000, + placementExplanation: makeProvisioningExplanation(), stepResults: { ...makeState().stepResults, nodeId: 'node-deleted-during-provisioning', @@ -132,6 +166,10 @@ describe('handleNodeProvisioning — timeout', () => { permanent: true, }); expect(state.stepResults.autoProvisioned).toBe(false); + expect(state.placementExplanation?.provisioningAttempts.at(-1)).toMatchObject({ + outcome: 'failed', + failureReason: 'node-unavailable', + }); expect(rc.ctx.storage.put).toHaveBeenCalledWith('state', state); expect(rc.ctx.storage.setAlarm).not.toHaveBeenCalled(); }); @@ -178,6 +216,7 @@ describe('handleNodeProvisioning — timeout', () => { const timeoutMs = 900_000; // 15 minutes const state = makeState({ provisioningStartedAt: Date.now() - timeoutMs - 1000, // past timeout + placementExplanation: makeProvisioningExplanation(), stepResults: { ...makeState().stepResults, nodeId: 'node-1' }, }); @@ -192,6 +231,10 @@ describe('handleNodeProvisioning — timeout', () => { await expect(handleNodeProvisioning(state, rc)).rejects.toThrow( /Node provisioning timed out after 15 minutes/ ); + expect(state.placementExplanation?.provisioningAttempts.at(-1)).toMatchObject({ + outcome: 'failed', + failureReason: 'provisioning-timeout', + }); }); it('timeout is configurable via context', async () => { @@ -264,7 +307,11 @@ describe('handleNodeProvisioning — timeout', () => { const rc = makeContext(); (rc.env.DATABASE.prepare as ReturnType).mockReturnValue({ bind: vi.fn().mockReturnValue({ - first: vi.fn().mockResolvedValue({ id: 'node-1', status: 'error', error_message: 'Server creation failed' }), + first: vi.fn().mockResolvedValue({ + id: 'node-1', + status: 'error', + error_message: 'Server creation failed', + }), run: vi.fn().mockResolvedValue({ meta: { changes: 1 } }), }), }); @@ -281,7 +328,11 @@ describe('handleNodeProvisioning — timeout', () => { const rc = makeContext(); (rc.env.DATABASE.prepare as ReturnType).mockReturnValue({ bind: vi.fn().mockReturnValue({ - first: vi.fn().mockResolvedValue({ id: 'node-1', status: 'error', error_message: 'Server creation failed' }), + first: vi.fn().mockResolvedValue({ + id: 'node-1', + status: 'error', + error_message: 'Server creation failed', + }), run: vi.fn().mockResolvedValue({ meta: { changes: 1 } }), }), }); @@ -326,6 +377,7 @@ describe('timeout parity — node_agent_ready vs node_provisioning', () => { const state = makeState({ currentStep: 'node_agent_ready', agentReadyStartedAt: Date.now() - 1_000_000, + placementExplanation: makeProvisioningExplanation(), stepResults: { ...makeState().stepResults, nodeId: 'node-deleted-during-readiness', @@ -342,6 +394,7 @@ describe('timeout parity — node_agent_ready vs node_provisioning', () => { agent_version: null, status: 'deleted', }), + run: vi.fn().mockResolvedValue({ meta: { changes: 1 } }), }), }); @@ -350,6 +403,10 @@ describe('timeout parity — node_agent_ready vs node_provisioning', () => { permanent: true, }); expect(state.stepResults.autoProvisioned).toBe(false); + expect(state.placementExplanation?.provisioningAttempts.at(-1)).toMatchObject({ + outcome: 'failed', + failureReason: 'node-unavailable', + }); expect(rc.ctx.storage.put).toHaveBeenCalledWith('state', state); expect(rc.ctx.storage.setAlarm).not.toHaveBeenCalled(); }); @@ -358,6 +415,7 @@ describe('timeout parity — node_agent_ready vs node_provisioning', () => { const state = makeState({ currentStep: 'node_agent_ready', agentReadyStartedAt: Date.now() - 1_000_000, // way past 15 min timeout + placementExplanation: makeProvisioningExplanation(), stepResults: { ...makeState().stepResults, nodeId: 'node-1' }, }); @@ -371,10 +429,17 @@ describe('timeout parity — node_agent_ready vs node_provisioning', () => { agent_version: null, status: 'running', }), + run: vi.fn().mockResolvedValue({ meta: { changes: 1 } }), }), }); await expect(handleNodeAgentReady(state, rc)).rejects.toThrow(/Node agent not ready within/); + expect(state.placementExplanation?.provisioningAttempts.at(-1)).toMatchObject({ + outcome: 'failed', + failureReason: 'readiness-timeout', + }); + expect(rc.ctx.storage.put).toHaveBeenCalledWith('state', state); + expect(rc.ctx.storage.setAlarm).not.toHaveBeenCalled(); }); it('handleNodeProvisioning throws after timeout (matching pattern)', async () => { diff --git a/apps/web/src/components/WorkspaceSidebar.tsx b/apps/web/src/components/WorkspaceSidebar.tsx index 44dab1b53c..7aeae6be30 100644 --- a/apps/web/src/components/WorkspaceSidebar.tsx +++ b/apps/web/src/components/WorkspaceSidebar.tsx @@ -1,21 +1,20 @@ import type { TokenUsage } from '@simple-agent-manager/acp-client'; -import type { AgentSession } from '@simple-agent-manager/shared'; -import type { DetectedPort, Event, WorkspaceResponse } from '@simple-agent-manager/shared'; -import { VM_LOCATIONS, VM_SIZE_LABELS } from '@simple-agent-manager/shared'; +import type { + AgentSession, + DetectedPort, + Event, + WorkspaceResponse, +} from '@simple-agent-manager/shared'; import { Button } from '@simple-agent-manager/ui'; -import { ExternalLink, GitBranch, Globe, Play, Trash2 } from 'lucide-react'; -import { type FC, useEffect, useMemo, useState } from 'react'; -import { Link } from 'react-router'; +import { GitBranch } from 'lucide-react'; +import { type FC, useMemo } from 'react'; -import { useNodeSystemInfo } from '../hooks/useNodeSystemInfo'; import type { GitStatusData } from '../lib/api'; -import { getPortAccessUrl } from '../lib/api'; -import { formatFileSize } from '../lib/file-utils'; -import { sanitizeUrl } from '../lib/url-utils'; import { CollapsibleSection } from './CollapsibleSection'; -import { ResourceBar } from './node/ResourceBar'; +import { WorkspaceSidebarInfrastructure } from './WorkspaceSidebarInfrastructure'; +import { type SidebarTab, WorkspaceSidebarSessions } from './WorkspaceSidebarSessions'; -// ─── Types ─────────────────────────────────────────────────── +export type { SidebarTab } from './WorkspaceSidebarSessions'; export interface SessionTokenUsage { sessionId: string; @@ -23,153 +22,38 @@ export interface SessionTokenUsage { usage: TokenUsage; } -export interface SidebarTab { - id: string; - kind: 'terminal' | 'chat'; - sessionId: string; - title: string; - status: string; - hostStatus?: string | null; - viewerCount?: number | null; -} - interface WorkspaceSidebarProps { workspace: WorkspaceResponse | null; isRunning: boolean; isMobile: boolean; - - // Lifecycle actions actionLoading: boolean; onStop: () => void; onRestart: () => void; onRebuild: () => void; - - // Rename displayNameInput: string; onDisplayNameChange: (value: string) => void; onRename: () => void; renaming: boolean; - - // Sessions workspaceTabs: SidebarTab[]; activeTabId: string | null; onSelectTab: (tab: SidebarTab) => void; onStopSession?: (sessionId: string) => void; - - // Session history (suspended/stopped sessions) historySessions?: AgentSession[]; onResumeSession?: (sessionId: string) => void; onDeleteSession?: (sessionId: string) => void; - - // Git gitStatus: GitStatusData | null; onOpenGitChanges: () => void; - - // Token usage (aggregated from ChatSession callbacks) sessionTokenUsages: SessionTokenUsage[]; - - // Detected ports detectedPorts: DetectedPort[]; - - // Events workspaceEvents: Event[]; } -// ─── Helpers ───────────────────────────────────────────────── - -function formatTokens(n: number): string { - if (n >= 1_000_000) return `${(n / 1_000_000).toFixed(1)}M`; - if (n >= 1_000) return `${(n / 1_000).toFixed(1)}K`; - return String(n); -} - -// VM display helpers using shared provider-agnostic constants -function vmSizeLabel(size: string): string { - const config = VM_SIZE_LABELS[size as keyof typeof VM_SIZE_LABELS]; - return config ? `${config.label} (${config.shortDescription})` : size; -} - -function vmLocationLabel(location: string): string { - const config = VM_LOCATIONS[location]; - return config ? `${config.name}, ${config.country}` : location; -} - -function useRelativeTime(isoDate: string | null | undefined): string { - const [now, setNow] = useState(Date.now()); - - useEffect(() => { - if (!isoDate) return; - const interval = setInterval(() => setNow(Date.now()), 1000); - return () => clearInterval(interval); - }, [isoDate]); - - if (!isoDate) return '-'; - - const ms = now - new Date(isoDate).getTime(); - if (ms < 0) return 'just now'; - const seconds = Math.floor(ms / 1000); - if (seconds < 60) return `${seconds}s`; - const minutes = Math.floor(seconds / 60); - if (minutes < 60) return `${minutes}m ${seconds % 60}s`; - const hours = Math.floor(minutes / 60); - return `${hours}h ${minutes % 60}m`; +function formatTokens(value: number): string { + if (value >= 1_000_000) return `${(value / 1_000_000).toFixed(1)}M`; + if (value >= 1_000) return `${(value / 1_000).toFixed(1)}K`; + return String(value); } -function sessionStatusColor(status: string, hostStatus?: string | null): string { - // Use live hostStatus for finer-grained colors when available - if (hostStatus) { - switch (hostStatus) { - case 'prompting': - return 'var(--sam-workspace-purple-fg)'; // purple: actively working - case 'ready': - return 'var(--sam-workspace-success-fg)'; // green: ready for prompts - case 'starting': - return 'var(--sam-workspace-warning-fg)'; // amber: initializing - case 'idle': - return 'var(--sam-workspace-tab-muted)'; // dim: no agent selected - case 'stopped': - return 'var(--sam-workspace-muted-dot)'; // dimmer: stopped - case 'error': - return 'var(--sam-workspace-danger-fg)'; // red - } - } - - switch (status) { - case 'connected': - case 'running': - return 'var(--sam-workspace-success-fg)'; - case 'connecting': - case 'reconnecting': - return 'var(--sam-workspace-warning-fg)'; - case 'error': - return 'var(--sam-workspace-danger-fg)'; - default: - return 'var(--sam-workspace-tab-muted)'; - } -} - -/** Human-readable label for agent host status */ -function hostStatusLabel(hostStatus: string): string { - switch (hostStatus) { - case 'prompting': - return 'working'; - case 'ready': - return 'ready'; - case 'starting': - return 'starting'; - case 'idle': - return 'idle'; - case 'stopped': - return 'stopped'; - case 'error': - return 'error'; - default: - return hostStatus; - } -} - -// ─── Component ─────────────────────────────────────────────── - export const WorkspaceSidebar: FC = ({ workspace, isRunning, @@ -195,33 +79,21 @@ export const WorkspaceSidebar: FC = ({ detectedPorts, workspaceEvents, }) => { - const uptime = useRelativeTime(workspace?.createdAt); - - // Node resource polling — only when workspace is running - const { systemInfo, error: systemInfoError } = useNodeSystemInfo( - workspace?.nodeId ?? undefined, - isRunning ? 'running' : undefined - ); - const gitTotal = gitStatus ? gitStatus.staged.length + gitStatus.unstaged.length + gitStatus.untracked.length : 0; - const totalUsage = useMemo(() => { const totals = { inputTokens: 0, outputTokens: 0, totalTokens: 0 }; - for (const s of sessionTokenUsages) { - totals.inputTokens += s.usage.inputTokens; - totals.outputTokens += s.usage.outputTokens; - totals.totalTokens += s.usage.totalTokens; + for (const session of sessionTokenUsages) { + totals.inputTokens += session.usage.inputTokens; + totals.outputTokens += session.usage.outputTokens; + totals.totalTokens += session.usage.totalTokens; } return totals; }, [sessionTokenUsages]); - const repoUrl = workspace?.repository ? `https://github.com/${workspace.repository}` : null; - return (
- {/* ── Header: name + lifecycle ── */}
= ({ }} placeholder="Workspace name" className="flex-1 rounded-sm border border-border-default bg-canvas text-fg-primary min-w-0" - style={{ - padding: '5px 8px', - fontSize: 'var(--sam-type-caption-size)', - }} + style={{ padding: '5px 8px', fontSize: 'var(--sam-type-caption-size)' }} />
- {/* Lifecycle buttons */}
{isRunning && ( <> @@ -286,331 +154,23 @@ export const WorkspaceSidebar: FC = ({
- {/* ── Scrollable sections ── */} -
- {/* Workspace Info */} - -
- {/* Repository */} - {workspace?.repository && ( - - {repoUrl ? ( - - {workspace.repository} - - - ) : ( - {workspace.repository} - )} - - )} - - {/* Branch */} - {workspace?.branch && ( - - - - {workspace.branch} - - - )} - - {/* VM */} - {workspace?.vmSize && ( - - {vmSizeLabel(workspace.vmSize)} - {workspace.vmLocation ? ` \u00B7 ${vmLocationLabel(workspace.vmLocation)}` : ''} - - )} - - {/* Node */} - {workspace?.nodeId && ( - - - {workspace.nodeId.slice(0, 8)} - - - - )} - - {/* Uptime */} - {uptime} -
- - - {/* Node Resources */} - {isRunning && workspace?.nodeId && ( - - {systemInfo ? ( -
- - - -
- ) : systemInfoError ? ( - - Unable to load resource data - - ) : ( - - Loading... - - )} - - )} - - {/* Active Ports */} - {isRunning && detectedPorts.length > 0 && ( - -
- {detectedPorts - .slice() - .sort((a, b) => a.port - b.port) - .map((p) => ( - - - {p.port} - {p.label} - {p.address === '127.0.0.1' && ( - (local) - )} - - - ))} -
- - )} - - {/* Sessions */} - {workspaceTabs.length > 0 && ( - -
- {workspaceTabs.map((tab) => { - const active = activeTabId === tab.id; - const isChat = tab.kind === 'chat'; - const canStop = isChat && onStopSession && tab.status === 'running'; - return ( -
- - {/* Stop button for chat sessions */} - {canStop && ( - - )} -
- ); - })} -
- - )} +
+ + - {/* Session History (suspended/stopped) */} - {historySessions.length > 0 && ( - -
- {historySessions.map((session) => ( -
- {/* Status dot */} - - {/* Label + last prompt */} -
-
- {session.label || `Chat ${session.id.slice(-6)}`} -
- {session.lastPrompt && ( -
- {session.lastPrompt} -
- )} -
- {session.status === 'suspended' ? 'suspended' : 'stopped'} - {session.suspendedAt && - ` \u00B7 ${new Date(session.suspendedAt).toLocaleTimeString()}`} - {!session.suspendedAt && - session.stoppedAt && - ` \u00B7 ${new Date(session.stoppedAt).toLocaleTimeString()}`} -
-
- {/* Action buttons */} -
- {session.status === 'suspended' && onResumeSession && ( - - )} - {onDeleteSession && ( - - )} -
-
- ))} -
- - )} - - {/* Git Summary */} {isRunning && ( = ({ > {gitStatus ? (
-
+
{gitStatus.staged.length} @@ -642,60 +199,49 @@ export const WorkspaceSidebar: FC = ({
) : ( - - Loading... - + Loading... )} )} - {/* Token Usage */} {sessionTokenUsages.length > 0 && totalUsage.totalTokens > 0 && ( -
+
{sessionTokenUsages - .filter((s) => s.usage.totalTokens > 0) - .map((s) => ( -
- - {s.label} - - - {formatTokens(s.usage.inputTokens)} in / {formatTokens(s.usage.outputTokens)}{' '} - out + .filter((session) => session.usage.totalTokens > 0) + .map((session) => ( +
+ {session.label} + + {formatTokens(session.usage.inputTokens)} in /{' '} + {formatTokens(session.usage.outputTokens)} out
))} - {sessionTokenUsages.filter((s) => s.usage.totalTokens > 0).length > 1 && ( - <> -
- Total - - {formatTokens(totalUsage.inputTokens)} in /{' '} - {formatTokens(totalUsage.outputTokens)} out - -
- + {sessionTokenUsages.filter((session) => session.usage.totalTokens > 0).length > 1 && ( +
+ Total + + {formatTokens(totalUsage.inputTokens)} in /{' '} + {formatTokens(totalUsage.outputTokens)} out + +
)}
)} - {/* Workspace Events (demoted — collapsed by default) */} = ({ storageKey="sam-sidebar-events" > {workspaceEvents.length === 0 ? ( - - No events yet. - + No events yet. ) : ( -
+
{workspaceEvents.map((event) => ( -
-
- {event.type} +
+
+ {event.type} {new Date(event.createdAt).toLocaleTimeString()}
-
{event.message}
+
{event.message}
))}
@@ -726,19 +270,3 @@ export const WorkspaceSidebar: FC = ({
); }; - -// ─── Sub-components ────────────────────────────────────────── - -const InfoRow: FC<{ label: string; children: React.ReactNode }> = ({ label, children }) => ( -
- - {label} - - - {children} - -
-); diff --git a/apps/web/src/components/WorkspaceSidebarInfrastructure.tsx b/apps/web/src/components/WorkspaceSidebarInfrastructure.tsx new file mode 100644 index 0000000000..2da1e6e4c4 --- /dev/null +++ b/apps/web/src/components/WorkspaceSidebarInfrastructure.tsx @@ -0,0 +1,324 @@ +import type { + DetectedPort, + LegacyPlacementExplanation, + PlacementExplanation, + PlacementRejectionReason, + WorkspaceResponse, +} from '@simple-agent-manager/shared'; +import { + isPlacementExplanationV2, + VM_LOCATIONS, + VM_SIZE_LABELS, +} from '@simple-agent-manager/shared'; +import { ExternalLink, GitBranch, Globe } from 'lucide-react'; +import { useEffect, useState } from 'react'; +import { Link } from 'react-router'; + +import { useNodeSystemInfo } from '../hooks/useNodeSystemInfo'; +import { getPortAccessUrl } from '../lib/api'; +import { formatFileSize } from '../lib/file-utils'; +import { sanitizeUrl } from '../lib/url-utils'; +import { CollapsibleSection } from './CollapsibleSection'; +import { ResourceBar } from './node/ResourceBar'; + +const REJECTION_LABELS: Record = { + 'node-not-found': 'Node was not found', + 'not-running': 'Node is not running', + 'wrong-runtime': 'Runtime cannot host VM workspaces', + unhealthy: 'Health check is not healthy', + 'heartbeat-missing': 'Heartbeat is missing', + 'heartbeat-stale': 'Heartbeat is stale', + 'agent-not-ready': 'VM agent is not ready', + 'agent-version-mismatch': 'VM agent build is incompatible', + undersized: 'VM is smaller than requested', + 'workspace-limit': 'Workspace limit reached', + 'cpu-threshold': 'CPU threshold reached', + 'memory-threshold': 'Memory threshold reached', + 'not-warm': 'Node is not in the warm pool', + 'warm-claim-lost': 'Another task claimed the warm node', +}; + +const PROVISIONING_OUTCOME_LABELS: Record< + PlacementExplanation['provisioningAttempts'][number]['outcome'], + string +> = { + started: 'Started', + succeeded: 'Succeeded', + 'capacity-rejected': 'Capacity unavailable', + failed: 'Failed', +}; + +const PROVISIONING_FAILURE_LABELS: Record< + NonNullable, + string +> = { + 'capacity-unavailable': 'Provider capacity unavailable', + 'node-limit': 'Node limit reached', + 'quota-exceeded': 'Compute quota exceeded', + 'credentials-unavailable': 'Cloud credentials unavailable', + 'provider-failed': 'Cloud provider failed', + 'provisioning-timeout': 'VM provisioning timed out', + 'readiness-timeout': 'VM agent readiness timed out', + 'node-unavailable': 'Provisioned node unavailable', +}; + +function useRelativeTime(isoDate: string | null | undefined): string { + const [now, setNow] = useState(Date.now()); + + useEffect(() => { + if (!isoDate) return; + const interval = setInterval(() => setNow(Date.now()), 1000); + return () => clearInterval(interval); + }, [isoDate]); + + if (!isoDate) return '-'; + const ms = now - new Date(isoDate).getTime(); + if (ms < 0) return 'just now'; + const seconds = Math.floor(ms / 1000); + if (seconds < 60) return `${seconds}s`; + const minutes = Math.floor(seconds / 60); + if (minutes < 60) return `${minutes}m ${seconds % 60}s`; + const hours = Math.floor(minutes / 60); + return `${hours}h ${minutes % 60}m`; +} + +function vmSizeLabel(size: string): string { + const config = VM_SIZE_LABELS[size as keyof typeof VM_SIZE_LABELS]; + return config ? `${config.label} (${config.shortDescription})` : size; +} + +function vmLocationLabel(location: string): string { + const config = VM_LOCATIONS[location]; + return config ? `${config.name}, ${config.country}` : location; +} + +function InfoRow({ label, children }: Readonly<{ label: string; children: React.ReactNode }>) { + return ( +
+ + {label} + + + {children} + +
+ ); +} + +function PlacementDetail({ + explanation, +}: Readonly<{ + explanation: PlacementExplanation | LegacyPlacementExplanation | null | undefined; +}>) { + if (!explanation) { + return No placement explanation was recorded.; + } + if (!isPlacementExplanationV2(explanation)) { + return ( +
+

{explanation.reason}

+ {explanation.selectedVmSize} + {explanation.vmSizeSource} +
+ ); + } + + const rejected = explanation.evaluatedNodes.filter((evaluation) => !evaluation.accepted); + return ( +
+

{explanation.summary}

+
+ {explanation.outcome} + {explanation.selectionPath} + + {explanation.request.vmSize} · {explanation.request.vmLocation} + + {explanation.selectedNodeId && ( + + {explanation.selectedNodeId} + + )} +
+ + {rejected.length > 0 && ( +
+ Rejected candidates + {rejected.map((evaluation, index) => ( +
+
{evaluation.nodeId}
+
+ {evaluation.rejectionReasons.map((reason) => REJECTION_LABELS[reason]).join(' · ')} +
+
+ ))} +
+ )} + + {explanation.provisioningAttempts.length > 0 && ( +
+ Provisioning attempts + {explanation.provisioningAttempts.map((attempt, index) => ( +
+ {attempt.vmSize} · {attempt.vmLocation} ·{' '} + {PROVISIONING_OUTCOME_LABELS[attempt.outcome]} + {attempt.failureReason + ? ` · ${PROVISIONING_FAILURE_LABELS[attempt.failureReason]}` + : ''} +
+ ))} +
+ )} +
+ ); +} + +export function WorkspaceSidebarInfrastructure({ + workspace, + isRunning, + detectedPorts, +}: Readonly<{ + workspace: WorkspaceResponse | null; + isRunning: boolean; + detectedPorts: DetectedPort[]; +}>) { + const uptime = useRelativeTime(workspace?.createdAt); + const { systemInfo, error: systemInfoError } = useNodeSystemInfo( + workspace?.nodeId ?? undefined, + isRunning ? 'running' : undefined + ); + const repoUrl = workspace?.repository ? `https://github.com/${workspace.repository}` : null; + + return ( + <> + +
+ {workspace?.repository && ( + + {repoUrl ? ( + + {workspace.repository} + + + ) : ( + {workspace.repository} + )} + + )} + {workspace?.branch && ( + + + + {workspace.branch} + + + )} + {workspace?.vmSize && ( + + {vmSizeLabel(workspace.vmSize)} + {workspace.vmLocation ? ` · ${vmLocationLabel(workspace.vmLocation)}` : ''} + + )} + {workspace?.nodeId && ( + + + {workspace.nodeId.slice(0, 8)} + + + + )} + {uptime} +
+ + + + + + + {isRunning && workspace?.nodeId && ( + + {systemInfo ? ( +
+ + + +
+ ) : ( + + {systemInfoError ? 'Unable to load resource data' : 'Loading...'} + + )} + + )} + + {isRunning && detectedPorts.length > 0 && ( + +
+ {detectedPorts + .slice() + .sort((a, b) => a.port - b.port) + .map((port) => ( + + + {port.port} + {port.label} + {port.address === '127.0.0.1' && ( + (local) + )} + + + ))} +
+ + )} + + ); +} diff --git a/apps/web/src/components/WorkspaceSidebarSessions.tsx b/apps/web/src/components/WorkspaceSidebarSessions.tsx new file mode 100644 index 0000000000..d231c49d54 --- /dev/null +++ b/apps/web/src/components/WorkspaceSidebarSessions.tsx @@ -0,0 +1,204 @@ +import type { AgentSession } from '@simple-agent-manager/shared'; +import { Play, Trash2 } from 'lucide-react'; + +import { CollapsibleSection } from './CollapsibleSection'; + +export interface SidebarTab { + id: string; + kind: 'terminal' | 'chat'; + sessionId: string; + title: string; + status: string; + hostStatus?: string | null; + viewerCount?: number | null; +} + +function sessionStatusColor(status: string, hostStatus?: string | null): string { + if (hostStatus) { + const colors: Record = { + prompting: 'var(--sam-workspace-purple-fg)', + ready: 'var(--sam-workspace-success-fg)', + starting: 'var(--sam-workspace-warning-fg)', + idle: 'var(--sam-workspace-tab-muted)', + stopped: 'var(--sam-workspace-muted-dot)', + error: 'var(--sam-workspace-danger-fg)', + }; + if (colors[hostStatus]) return colors[hostStatus]; + } + if (status === 'connected' || status === 'running') return 'var(--sam-workspace-success-fg)'; + if (status === 'connecting' || status === 'reconnecting') + return 'var(--sam-workspace-warning-fg)'; + if (status === 'error') return 'var(--sam-workspace-danger-fg)'; + return 'var(--sam-workspace-tab-muted)'; +} + +function hostStatusLabel(status: string): string { + return status === 'prompting' ? 'working' : status; +} + +export function WorkspaceSidebarSessions({ + workspaceTabs, + activeTabId, + onSelectTab, + onStopSession, + historySessions, + onResumeSession, + onDeleteSession, + isMobile, +}: Readonly<{ + workspaceTabs: SidebarTab[]; + activeTabId: string | null; + onSelectTab: (tab: SidebarTab) => void; + onStopSession?: (sessionId: string) => void; + historySessions: AgentSession[]; + onResumeSession?: (sessionId: string) => void; + onDeleteSession?: (sessionId: string) => void; + isMobile: boolean; +}>) { + return ( + <> + {workspaceTabs.length > 0 && ( + +
+ {workspaceTabs.map((tab) => { + const active = activeTabId === tab.id; + const isChat = tab.kind === 'chat'; + const canStop = isChat && onStopSession && tab.status === 'running'; + return ( +
+ + {canStop && ( + + )} +
+ ); + })} +
+ + )} + + {historySessions.length > 0 && ( + +
+ {historySessions.map((session) => ( +
+ +
+
+ {session.label || `Chat ${session.id.slice(-6)}`} +
+ {session.lastPrompt && ( +
+ {session.lastPrompt} +
+ )} +
+ {session.status === 'suspended' ? 'suspended' : 'stopped'} +
+
+
+ {session.status === 'suspended' && onResumeSession && ( + + )} + {onDeleteSession && ( + + )} +
+
+ ))} +
+ + )} + + ); +} diff --git a/apps/web/tests/playwright/workspace-placement-audit.spec.ts b/apps/web/tests/playwright/workspace-placement-audit.spec.ts new file mode 100644 index 0000000000..5654c81650 --- /dev/null +++ b/apps/web/tests/playwright/workspace-placement-audit.spec.ts @@ -0,0 +1,226 @@ +import { expect, type Page, type Route, test } from '@playwright/test'; +import type { PlacementExplanation } from '@simple-agent-manager/shared'; + +import { + assertNoClippedOverflow, + assertNoOverflow, + makeMockUser, + screenshot, + seedTheme, +} from './audit-helpers'; + +const MOCK_USER = makeMockUser({ + email: 'placement@example.com', + name: 'Placement Auditor', + sessionId: 'session-placement-1', + userId: 'user-placement-1', +}); + +const BASE_PLACEMENT: PlacementExplanation = { + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: '01NODE_SELECTED', + summary: 'Reused a healthy compatible node with available capacity.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [ + { + nodeId: '01NODE_SELECTED', + path: 'capacity', + accepted: true, + rejectionReasons: [], + snapshot: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + agentVersionCompatible: true, + heartbeatAgeSeconds: 12, + activeWorkspaceCount: 2, + cpuLoadAvg1: 8, + memoryPercent: 20, + }, + }, + ], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', +}; + +const STRESS_PLACEMENT: PlacementExplanation = { + ...BASE_PLACEMENT, + outcome: 'failed', + selectionPath: 'provisioning', + selectedNodeId: null, + summary: + `No reusable node qualified — ${'long placement evidence '.repeat(14)}` + + '& é🚀.', + evaluatedNodes: Array.from({ length: 32 }, (_, index) => ({ + ...BASE_PLACEMENT.evaluatedNodes[0]!, + nodeId: `node-${index}-é🚀--${'x'.repeat(120)}`, + accepted: false, + rejectionReasons: ['agent-version-mismatch', 'workspace-limit', 'heartbeat-stale'] as const, + })), + provisioningAttempts: [ + { + vmSize: 'large', + vmLocation: 'hel1', + outcome: 'capacity-rejected', + failureReason: 'capacity-unavailable', + }, + { + vmSize: 'medium', + vmLocation: 'hel1', + outcome: 'failed', + failureReason: 'provider-failed', + }, + ], +}; + +const LEGACY_PLACEMENT = { + selectedVmSize: 'medium', + vmSizeSource: 'project', + reservation: { + cpuMillis: 1000, + memoryMb: 2048, + diskMb: 4096, + exclusiveNode: false, + maxCoTenants: 5, + source: 'project', + sourceId: 'p', + version: 1, + }, + reason: 'L', + decidedAt: '2026-08-10T00:00:00.000Z', +}; + +function fulfill(route: Route, body: unknown, status = 200) { + return route.fulfill({ status, contentType: 'application/json', body: JSON.stringify(body) }); +} + +async function setupMocks(page: Page, placementExplanation: unknown) { + const workspace = { + id: 'ws-placement-1', + name: 'placement-audit', + displayName: 'Placement Audit Workspace', + status: 'stopped', + nodeId: 'node-placement-1', + projectId: 'project-placement-1', + userId: 'user-placement-1', + repository: 'owner/repository', + branch: 'main', + vmSize: 'medium', + vmLocation: 'hel1', + workspaceProfile: 'full', + url: 'http://localhost:4173', + placementExplanation, + errorMessage: null, + chatSessionId: null, + createdAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:05:00.000Z', + }; + const project = { + id: 'project-placement-1', + name: 'Placement Project', + repository: 'owner/repository', + defaultBranch: 'main', + userId: 'user-placement-1', + }; + + await page.route('**/api/**', async (route) => { + const path = new URL(route.request().url()).pathname; + if (path.includes('/api/auth/')) return fulfill(route, MOCK_USER); + if (path === '/api/workspaces/ws-placement-1') return fulfill(route, workspace); + if (path === '/api/workspaces/ws-placement-1/agent-sessions') return fulfill(route, []); + if (path === '/api/workspaces/ws-placement-1/events') { + return fulfill(route, { events: [], nextCursor: null }); + } + if (path === '/api/workspaces') return fulfill(route, [workspace]); + if (path === '/api/projects/project-placement-1') return fulfill(route, project); + if (path === '/api/projects') return fulfill(route, { projects: [project], nextCursor: null }); + if (path.startsWith('/api/notifications')) { + return fulfill(route, { notifications: [], unreadCount: 0 }); + } + if (path === '/api/providers/catalog') return fulfill(route, { catalogs: [] }); + if (path === '/api/trial/status') return fulfill(route, { available: false }); + if (path === '/api/agents') return fulfill(route, { agents: [] }); + if (path === '/api/github/installations') return fulfill(route, []); + return fulfill(route, {}); + }); +} + +async function openPlacement(page: Page): Promise { + await page.goto('/workspaces/ws-placement-1'); + await expect(page.getByText('Placement Audit Workspace').first()).toBeVisible(); + if (page.viewportSize()!.width < 768) { + await page.getByRole('button', { name: 'Open workspace menu' }).click(); + await expect(page.getByRole('dialog', { name: 'Workspace menu' })).toBeVisible(); + } + const placementToggle = page.getByRole('button', { name: /placement/i }); + await expect(placementToggle).toBeVisible(); + await placementToggle.click(); + await expect(placementToggle).toHaveAttribute('aria-expanded', 'true'); +} + +async function auditLayout(page: Page): Promise { + await assertNoOverflow(page); + await assertNoClippedOverflow(page); + const sidebar = + page.viewportSize()!.width < 768 + ? page.getByRole('dialog', { name: 'Workspace menu' }) + : page.locator('aside'); + const dimensions = await sidebar.evaluate((element) => ({ + clientWidth: element.clientWidth, + scrollWidth: element.scrollWidth, + left: element.getBoundingClientRect().left, + right: element.getBoundingClientRect().right, + })); + expect(dimensions.scrollWidth).toBeLessThanOrEqual(dimensions.clientWidth + 1); + expect(dimensions.left).toBeGreaterThanOrEqual(0); + expect(dimensions.right).toBeLessThanOrEqual(page.viewportSize()!.width + 1); +} + +const scenarios = [ + { name: 'normal', value: BASE_PLACEMENT, expected: BASE_PLACEMENT.summary }, + { name: 'empty', value: null, expected: 'No placement explanation was recorded.' }, + { name: 'legacy-single-character', value: LEGACY_PLACEMENT, expected: 'L' }, + { name: 'stress-failed-special', value: STRESS_PLACEMENT, expected: 'Rejected candidates' }, +] as const; + +for (const theme of ['dark', 'light'] as const) { + test.describe(`Workspace placement — ${theme}`, () => { + for (const scenario of scenarios) { + test(scenario.name, async ({ page }) => { + await seedTheme(page, theme); + await setupMocks(page, scenario.value); + await openPlacement(page); + await expect( + page.getByText(scenario.expected, { exact: scenario.expected === 'L' }) + ).toBeVisible(); + const injectedScript = await page + .locator('script') + .evaluateAll((elements) => + elements.some((element) => element.textContent?.includes('alert(')) + ); + expect(injectedScript).toBe(false); + await auditLayout(page); + + if (scenario.name === 'stress-failed-special') { + const lastCandidate = page.getByText(/node-31-é🚀-/); + await lastCandidate.scrollIntoViewIfNeeded(); + await expect(lastCandidate).toBeVisible(); + await auditLayout(page); + } + await screenshot(page, `workspace-placement-${scenario.name}-${theme}`); + }); + } + }); +} diff --git a/apps/web/tests/unit/components/workspace-placement-section.test.tsx b/apps/web/tests/unit/components/workspace-placement-section.test.tsx new file mode 100644 index 0000000000..48a0797c69 --- /dev/null +++ b/apps/web/tests/unit/components/workspace-placement-section.test.tsx @@ -0,0 +1,168 @@ +import type { PlacementExplanation, WorkspaceResponse } from '@simple-agent-manager/shared'; +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { MemoryRouter } from 'react-router'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { WorkspaceSidebarInfrastructure } from '../../../src/components/WorkspaceSidebarInfrastructure'; + +vi.mock('../../../src/hooks/useNodeSystemInfo', () => ({ + useNodeSystemInfo: () => ({ systemInfo: null, error: null }), +})); + +const basePlacement: PlacementExplanation = { + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: '01NODE_SELECTED', + summary: 'Reused a healthy compatible node with available capacity.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [ + { + nodeId: '01NODE_REJECTED', + path: 'capacity', + accepted: false, + rejectionReasons: ['agent-version-mismatch', 'workspace-limit'], + snapshot: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + healthStatus: 'healthy', + agentVersionCompatible: false, + heartbeatAgeSeconds: 12, + activeWorkspaceCount: 5, + cpuLoadAvg1: 8, + memoryPercent: 20, + }, + }, + ], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', +}; + +function workspace( + placementExplanation: WorkspaceResponse['placementExplanation'] = basePlacement +): WorkspaceResponse { + return { + id: 'workspace-1', + nodeId: '01NODE_SELECTED', + projectId: 'project-1', + name: 'Workspace', + displayName: 'Workspace', + repository: 'owner/repo', + branch: 'main', + status: 'running', + vmSize: 'medium', + vmLocation: 'hel1', + placementExplanation, + vmIp: null, + lastActivityAt: null, + errorMessage: null, + createdAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', + }; +} + +function renderInfrastructure(value: WorkspaceResponse): void { + render( + + + + ); +} + +beforeEach(() => localStorage.clear()); + +describe('workspace placement sidebar section', () => { + it('starts collapsed, then exposes concise outcome and typed rejection reasons', async () => { + renderInfrastructure(workspace()); + + const toggle = screen.getByRole('button', { name: /placement/i }); + expect(toggle).toHaveAttribute('aria-expanded', 'false'); + expect(screen.queryByText(basePlacement.summary)).not.toBeInTheDocument(); + + await userEvent.click(toggle); + + expect(toggle).toHaveAttribute('aria-expanded', 'true'); + expect(screen.getByText(basePlacement.summary)).toBeInTheDocument(); + expect( + screen.getByText('VM agent build is incompatible · Workspace limit reached') + ).toBeInTheDocument(); + expect(screen.getByText('01NODE_SELECTED')).toBeInTheDocument(); + }); + + it('renders empty and legacy records without crashing', async () => { + const { unmount } = render( + + + + ); + await userEvent.click(screen.getByRole('button', { name: /placement/i })); + expect(screen.getByText('No placement explanation was recorded.')).toBeInTheDocument(); + unmount(); + localStorage.clear(); + + renderInfrastructure( + workspace({ + selectedVmSize: 'medium', + vmSizeSource: 'project', + reservation: { + cpuMillis: 1000, + memoryMb: 2048, + diskMb: 4096, + exclusiveNode: false, + maxCoTenants: 5, + source: 'project', + sourceId: 'project-1', + version: 1, + }, + reason: 'Legacy placement record', + decidedAt: '2026-08-10T00:00:00.000Z', + }) + ); + await userEvent.click(screen.getByRole('button', { name: /placement/i })); + expect(screen.getByText('Legacy placement record')).toBeInTheDocument(); + }); + + it('wraps long, numerous, unicode, and markup-like evidence as text', async () => { + const manyRejected = Array.from({ length: 30 }, (_, index) => ({ + ...basePlacement.evaluatedNodes[0]!, + nodeId: `node-${index}-é🚀--${'x'.repeat(180)}`, + })); + renderInfrastructure( + workspace({ + ...basePlacement, + outcome: 'failed', + summary: `No reusable node qualified — ${'long evidence '.repeat(20)} & `, + evaluatedNodes: manyRejected, + provisioningAttempts: [ + { + vmSize: 'medium', + vmLocation: 'hel1', + outcome: 'failed', + failureReason: 'provider-failed', + }, + ], + }) + ); + await userEvent.click(screen.getByRole('button', { name: /placement/i })); + + expect(screen.getAllByText(/node-\d+-é🚀-/)).toHaveLength(30); + expect(document.querySelector('script')).toBeNull(); + expect(screen.getByTestId('placement-detail')).toHaveClass('min-w-0'); + expect(screen.getByText(/Cloud provider failed/)).toBeInTheDocument(); + }); +}); diff --git a/apps/www/src/content/docs/docs/architecture/overview.md b/apps/www/src/content/docs/docs/architecture/overview.md index 317fb3d651..375ed17f50 100644 --- a/apps/www/src/content/docs/docs/architecture/overview.md +++ b/apps/www/src/content/docs/docs/architecture/overview.md @@ -259,6 +259,28 @@ graph LR Cross-DO coordination with NodeLifecycle (for warm node claims) and ProjectData (for session linkage). Exponential backoff on transient errors. +### Reusable VM placement evidence + +Every reusable-VM decision goes through one selector for preferred, warm-pool, +capacity, trial, and manually selected nodes. The selector keeps the exact +published VM-agent build check: a node with an unknown or different build is +not eligible for new work. It also evaluates runtime, VM size, health, +heartbeat freshness, agent readiness, workspace count, and configured CPU and +memory thresholds before choosing a node. + +SAM stores a versioned placement explanation on the task or trial as soon as +the decision is made, appends provisioning attempts and typed failure reasons, +and copies the final record to the workspace. Workspace and task APIs, +`get_workspace_info`, and the workspace sidebar expose this evidence. The +record contains only bounded metric snapshots, identifiers, and typed outcomes; +it deliberately excludes raw agent versions, raw metrics, provider errors, +credentials, prompts, repositories, and environment values. + +The selected node ID remains visible because it is already part of the workspace +contract. Every rejected or otherwise unselected candidate is persisted as a stable +`candidate-N` alias, so one tenant's placement evidence cannot reveal another trial's +or project's host identifiers. + ## ACP Session Lifecycle Agent sessions are managed by the ProjectData DO with this state machine: @@ -302,16 +324,27 @@ graph TD P1 -.- P1D["D1, KV, R2, DNS records"] P2["Phase 2: Configuration"] --> P3 P2 -.- P2D["Sync wrangler.toml, read security keys"] - P3["Phase 3: Application"] --> P4 - P3 -.- P3D["Build → Bake vm-agent into container image → Deploy Worker → Deploy Pages → Migrations → Secrets"] - P4["Phase 4: VM Agent"] --> P5 - P4 -.- P4D["Build Go (multi-arch) → Upload to R2"] + P3["Phase 3: Application + artifacts"] --> P4 + P3 -.- P3D["Build apps → Bake container artifact → Migrate D1 → Publish changed reusable-VM binaries → Deploy Worker/Pages"] + P4["Phase 4: Secrets"] --> P5 + P4 -.- P4D["Apply deployment-managed Worker secrets"] P5["Phase 5: Validation"] P5 -.- P5D["Health check polling"] ``` CI runs lint, typecheck, tests, and build on pull requests and on canonical-repository `main` pushes. In the canonical repository, Deploy Production runs after successful `main` CI and re-verifies that the completed CI SHA is still the current `main` tip after entering the serialized deployment queue. In self-host forks, `main` push CI is intentionally skipped, so operators update their instance by manually running **Deploy Production** against the exact commit SHA from the fork's synced `main` branch. The production GitHub Environment must separately restrict deployments to the selected `main` branch so other refs cannot access its secrets with modified workflow code. +Reusable-VM agent releases have their own identity inside that pipeline. A +deterministic fingerprint covers the tracked `packages/vm-agent` build inputs +and an explicit compatibility marker. An unrelated application deploy carries +forward the last actually published required build and skips the reusable-VM +binary upload. Changed inputs or an explicit compatibility bump publish the new +binaries before the Worker begins requiring that exact deployment SHA. A +requested agent skip fails closed when compatibility changed or no prior +published release can be proven. The separately baked Cloudflare Container +image remains part of the Worker/container rollback boundary and does not share +the reusable VM release identity. + ## Key Design Decisions | Decision | Rationale | diff --git a/apps/www/src/content/docs/docs/guides/agents.md b/apps/www/src/content/docs/docs/guides/agents.md index 57498d4410..e616c856d0 100644 --- a/apps/www/src/content/docs/docs/guides/agents.md +++ b/apps/www/src/content/docs/docs/guides/agents.md @@ -193,9 +193,14 @@ Running agents have access to project-aware MCP tools: | `search_messages` | Search messages by keyword — uses FTS5 full-text search for completed sessions; keyword matching for active sessions | | `update_task_status` | Report progress | | `get_task_details` | Inspect task state, persisted output fields, PR/error details, session id, and bounded recent assistant diagnostics | +| `get_workspace_info` | Read VM-local workspace metadata plus the D1-backed, non-sensitive node-placement summary and typed detail | | `complete_task` | Mark current work as done, optionally with structured completion evidence | | `request_human_input` | Record a user decision request and notify the user; the tool call itself is non-blocking | +`get_workspace_info` returns `placement: null | { summary, detail }`. `summary` is a +concise non-sensitive outcome, while `detail` contains the typed placement explanation +when one was persisted. + `get_task_details` keeps `outputSummary` and `completionEvidence` as the canonical persisted completion fields. When a task has a linked chat session, it can also include a bounded `recentAssistantMessages` array with up to five recent assistant messages, each capped to 2,000 characters, so orchestrators can recover useful final output when the persisted summary is sparse. The SAM session (Anthropic tool) variant returns a single `finalAssistantMessage` (the latest assistant message, content capped to 2,000 characters) instead of the full array. If session diagnostics are unavailable, task details still return and the diagnostic fields are empty/null. Claude Code and Codex get these tools on both the VM and [Instant](/docs/guides/instant-sessions/) runtimes. If a Codex session is handed an MCP server without a usable token, it fails to start with an explicit error rather than launching a tool-less agent. diff --git a/apps/www/src/content/docs/docs/guides/self-hosting.mdx b/apps/www/src/content/docs/docs/guides/self-hosting.mdx index c7f7455864..5221fbca72 100644 --- a/apps/www/src/content/docs/docs/guides/self-hosting.mdx +++ b/apps/www/src/content/docs/docs/guides/self-hosting.mdx @@ -241,10 +241,23 @@ The workflow: 1. Validates configuration 2. Provisions infrastructure via Pulumi (D1, KV, R2, DNS), including prefix-scoped R2 retention for session snapshots, private VM diagnostic evidence, one-day temporary uploads, and thirty-day TTS cache objects (`infra/resources/storage.ts:r2BucketLifecycle`) -3. Deploys API Worker and Web UI +3. Resolves the reusable-VM agent release from a deterministic fingerprint of `packages/vm-agent/**` plus the explicit compatibility marker 4. Runs database migrations -5. Builds and uploads VM Agent binaries -6. Runs health check +5. Builds and uploads VM Agent binaries on a first deploy, when those inputs or the compatibility marker changed, or when legacy release metadata cannot prove equivalence; otherwise it carries the published release forward +6. Deploys the API Worker and Web UI with the exact published VM-agent requirement +7. Runs health checks + +Unrelated application deploys carry forward the last actually published VM-agent +version and skip the binary build/upload. A change to VM-agent build inputs—or to +`scripts/deploy/vm-agent-compatibility-version.txt` when a protocol change requires a +new exact build—publishes the binaries before the Worker begins enforcing that version. +Legacy or otherwise unproven equivalence also publishes a fresh release when agent +building is enabled. The workflow stops if Worker settings cannot be read, the settings +response is malformed, publication fails, or `skip_agent` attempts to bypass a first, +changed, or unproven release. Empty or malformed release hashes are never carried and +force a fresh publication when building is enabled. `VM_AGENT_REQUIRED_VERSION` and +`VM_AGENT_BUILD_FINGERPRINT` are generated release metadata; self-hosters should not set +or rotate them manually. Before deploying the API Worker, the workflow reads its applied Durable Object migration tag. A fresh installation creates every namespace with SQLite storage; an existing installation retains its already-applied namespace history and storage backends. This is automatic—do not edit historical migration entries or create namespaces manually. diff --git a/apps/www/src/content/docs/docs/reference/api.md b/apps/www/src/content/docs/docs/reference/api.md index d2c3f4a7b2..4a6a06ef70 100644 --- a/apps/www/src/content/docs/docs/reference/api.md +++ b/apps/www/src/content/docs/docs/reference/api.md @@ -47,7 +47,15 @@ List all workspaces for the authenticated user. ### `GET /api/workspaces/:id` -Get workspace details including status, node info, and URLs. +Get workspace details including status, node info, URLs, and a safely parsed +`placementExplanation` when placement evidence was recorded. Version 2 records +describe whether SAM reused or provisioned a node, the selection path, typed +candidate rejection reasons, bounded request/metric snapshots, and +provisioning attempts. Historical workspaces may return a legacy placement +shape or `null`. Only the selected node retains its real ID; rejected and +eligible-but-unselected candidates use stable `candidate-N` aliases. Provisioning +failures use allowlisted reasons such as `provider-failed`, `provisioning-timeout`, +`readiness-timeout`, and `node-unavailable` rather than raw provider messages. ### `POST /api/workspaces/:id/stop` @@ -161,6 +169,15 @@ Create a task record. } ``` +Task responses retain `placementExplanationJson` for compatibility and also +return its validated `placementExplanation` form. Invalid or unknown stored +shapes are returned as `null` in the parsed field instead of being reflected to +clients. + +The generated CLI OpenAPI contract describes both the version 2 and legacy parsed +placement shapes. Task responses also retain the nullable raw +`placementExplanationJson` string for backward compatibility. + ## Deployment Releases ### `POST /api/projects/:projectId/environments/:envId/releases` diff --git a/apps/www/src/content/docs/docs/reference/configuration.md b/apps/www/src/content/docs/docs/reference/configuration.md index 8f102fb77a..9fa4462bfb 100644 --- a/apps/www/src/content/docs/docs/reference/configuration.md +++ b/apps/www/src/content/docs/docs/reference/configuration.md @@ -572,14 +572,15 @@ Webhook damping uses Cloudflare KV's eventually consistent read-update-write beh ## Node & Workspace Readiness -| Variable | Default | Description | -| ---------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `NODE_AGENT_READY_TIMEOUT_MS` | `900000` (15 min) | Wait for VM agent to report ready | -| `NODE_AGENT_READY_POLL_INTERVAL_MS` | `5000` | Poll interval for agent readiness | -| `VM_AGENT_REQUIRED_VERSION` | _(deploy-generated)_ | Required vm-agent build for reusable VM nodes. Official deploys derive this from the Git commit SHA after publishing matching binaries; leave unset only for local/manual development or skip-agent deploys. | -| `TASK_RUNNER_WORKSPACE_READY_TIMEOUT_MS` | `1800000` (30 min) | Max wait for workspace-ready callback | -| `PROVISIONING_TIMEOUT_MS` | `1800000` (30 min) | Cron marks stuck workspaces as error | -| `NODE_HEARTBEAT_STALE_SECONDS` | `180` | Seconds without a heartbeat before a node is treated as stale | +| Variable | Default | Description | +| ---------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `NODE_AGENT_READY_TIMEOUT_MS` | `900000` (15 min) | Wait for VM agent to report ready | +| `NODE_AGENT_READY_POLL_INTERVAL_MS` | `5000` | Poll interval for agent readiness | +| `VM_AGENT_REQUIRED_VERSION` | _(deploy-generated)_ | Exact published vm-agent build required for reusable VM nodes. Official deploys carry the last published SHA across unrelated changes and advance it only after compatible binaries are uploaded; leave unset only for local/manual development. | +| `VM_AGENT_BUILD_FINGERPRINT` | _(deploy-generated)_ | Deterministic fingerprint of vm-agent source, Go dependency/toolchain inputs, build scripts, and the explicit compatibility marker. Used with the required version to decide whether an official deploy can carry a release forward. | +| `TASK_RUNNER_WORKSPACE_READY_TIMEOUT_MS` | `1800000` (30 min) | Max wait for workspace-ready callback | +| `PROVISIONING_TIMEOUT_MS` | `1800000` (30 min) | Cron marks stuck workspaces as error | +| `NODE_HEARTBEAT_STALE_SECONDS` | `180` | Seconds without a heartbeat before a node is treated as stale | ## App Deployment Routing diff --git a/packages/shared/src/constants/index.ts b/packages/shared/src/constants/index.ts index 92bfb608a5..e94ffaeb1e 100644 --- a/packages/shared/src/constants/index.ts +++ b/packages/shared/src/constants/index.ts @@ -26,6 +26,15 @@ export { VM_LOCATIONS, } from './providers'; +// Placement evidence and request boundaries +export { + isSafeVmLocationId, + SAM_NODE_ID_LENGTH, + SAM_NODE_ID_REGEX, + VM_LOCATION_ID_MAX_LENGTH, + VM_LOCATION_ID_REGEX, +} from './placement'; + // Status export { STATUS_COLORS, STATUS_LABELS } from './status'; diff --git a/packages/shared/src/constants/placement.ts b/packages/shared/src/constants/placement.ts new file mode 100644 index 0000000000..99ee2096ce --- /dev/null +++ b/packages/shared/src/constants/placement.ts @@ -0,0 +1,11 @@ +/** Canonical SAM node identifiers are ULIDs (uppercase Crockford Base32). */ +export const SAM_NODE_ID_LENGTH = 26; +export const SAM_NODE_ID_REGEX = /^[0-9A-HJKMNP-TV-Z]{26}$/; + +/** Provider location identifiers are bounded, printable slug-like values. */ +export const VM_LOCATION_ID_MAX_LENGTH = 64; +export const VM_LOCATION_ID_REGEX = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; + +export function isSafeVmLocationId(value: string): boolean { + return value.length <= VM_LOCATION_ID_MAX_LENGTH && VM_LOCATION_ID_REGEX.test(value); +} diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 1629968feb..e0b71dd08e 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -39,3 +39,6 @@ export * from './failure-classification'; // Runtime validation (dependency-free "is this a plain JSON object" predicate) export * from './runtime-validation'; + +// Versioned, safely parsed placement audit records +export * from './placement'; diff --git a/packages/shared/src/placement.ts b/packages/shared/src/placement.ts new file mode 100644 index 0000000000..08e0a2a4f4 --- /dev/null +++ b/packages/shared/src/placement.ts @@ -0,0 +1,318 @@ +import type { + LegacyPlacementExplanation, + PlacementExplanation, + PlacementNodeEvaluation, + PlacementNodeSnapshot, + PlacementProvisioningAttempt, + PlacementRejectionReason, + PlacementRequestSnapshot, + PlacementSelectionPath, + ResourceRequirementsSource, +} from './types/resource'; +import type { VMSize } from './types/workspace'; + +const VM_SIZES = new Set(['small', 'medium', 'large']); +const PATHS = new Set([ + 'preferred', + 'warm', + 'capacity', + 'trial', + 'manual', + 'provisioning', +]); +const REJECTIONS = new Set([ + 'node-not-found', + 'not-running', + 'wrong-runtime', + 'unhealthy', + 'heartbeat-missing', + 'heartbeat-stale', + 'agent-not-ready', + 'agent-version-mismatch', + 'undersized', + 'workspace-limit', + 'cpu-threshold', + 'memory-threshold', + 'not-warm', + 'warm-claim-lost', +]); +const PROVISIONING_OUTCOMES = new Set([ + 'started', + 'succeeded', + 'capacity-rejected', + 'failed', +]); +const PROVISIONING_FAILURES = new Set>([ + 'capacity-unavailable', + 'node-limit', + 'quota-exceeded', + 'credentials-unavailable', + 'provider-failed', + 'provisioning-timeout', + 'readiness-timeout', + 'node-unavailable', +]); +const RESOURCE_SOURCES = new Set([ + 'task', + 'trigger', + 'skill', + 'agent-profile', + 'project', + 'user', + 'platform', +]); +const VM_SIZE_SOURCES = new Set([ + ...RESOURCE_SOURCES, + 'explicit', +]); + +function record(value: unknown): Record | null { + return typeof value === 'object' && value !== null && !Array.isArray(value) + ? (value as Record) + : null; +} + +function finiteNumber(value: unknown, min = 0, max = Number.MAX_SAFE_INTEGER): number | null { + return typeof value === 'number' && Number.isFinite(value) && value >= min && value <= max + ? value + : null; +} + +function stringValue(value: unknown, maxLength = 256): string | null { + return typeof value === 'string' && value.length > 0 && value.length <= maxLength ? value : null; +} + +function vmSize(value: unknown): VMSize | null { + return typeof value === 'string' && VM_SIZES.has(value as VMSize) ? (value as VMSize) : null; +} + +function parseRequest(value: unknown): PlacementRequestSnapshot | null { + const raw = record(value); + const size = vmSize(raw?.vmSize); + const location = stringValue(raw?.vmLocation, 64); + const maxWorkspaces = finiteNumber(raw?.maxWorkspacesPerNode, 1); + const cpu = finiteNumber(raw?.cpuThresholdPercent, 0, 100); + const memory = finiteNumber(raw?.memoryThresholdPercent, 0, 100); + const stale = finiteNumber(raw?.heartbeatStaleSeconds, 1); + if ( + raw?.runtime !== 'vm' || + !size || + !location || + maxWorkspaces === null || + cpu === null || + memory === null || + stale === null + ) { + return null; + } + return { + runtime: 'vm', + vmSize: size, + vmLocation: location, + maxWorkspacesPerNode: maxWorkspaces, + cpuThresholdPercent: cpu, + memoryThresholdPercent: memory, + heartbeatStaleSeconds: stale, + }; +} + +function parseSnapshot(value: unknown): PlacementNodeSnapshot | null { + const raw = record(value); + const size = stringValue(raw?.vmSize, 32); + const location = stringValue(raw?.vmLocation, 64); + const active = finiteNumber(raw?.activeWorkspaceCount); + const heartbeat = + raw?.heartbeatAgeSeconds === null ? null : finiteNumber(raw?.heartbeatAgeSeconds); + const cpu = raw?.cpuLoadAvg1 === null ? null : finiteNumber(raw?.cpuLoadAvg1, 0, 100); + const memory = raw?.memoryPercent === null ? null : finiteNumber(raw?.memoryPercent, 0, 100); + if ( + (raw?.runtime !== 'vm' && raw?.runtime !== 'other') || + !size || + !location || + !['healthy', 'stale', 'unhealthy', 'unknown'].includes(String(raw?.healthStatus)) || + typeof raw?.agentVersionCompatible !== 'boolean' || + active === null || + (raw?.heartbeatAgeSeconds !== null && heartbeat === null) || + (raw?.cpuLoadAvg1 !== null && cpu === null) || + (raw?.memoryPercent !== null && memory === null) + ) { + return null; + } + return { + runtime: raw.runtime, + vmSize: size, + vmLocation: location, + healthStatus: raw.healthStatus as PlacementNodeSnapshot['healthStatus'], + agentVersionCompatible: raw.agentVersionCompatible, + heartbeatAgeSeconds: heartbeat, + activeWorkspaceCount: active, + cpuLoadAvg1: cpu, + memoryPercent: memory, + }; +} + +function parseEvaluation(value: unknown): PlacementNodeEvaluation | null { + const raw = record(value); + const nodeId = stringValue(raw?.nodeId, 128); + const path = raw?.path; + const snapshot = parseSnapshot(raw?.snapshot); + if ( + !nodeId || + typeof path !== 'string' || + path === 'provisioning' || + !PATHS.has(path as PlacementSelectionPath) || + typeof raw?.accepted !== 'boolean' || + !Array.isArray(raw?.rejectionReasons) || + !raw.rejectionReasons.every( + (reason) => typeof reason === 'string' && REJECTIONS.has(reason as PlacementRejectionReason) + ) || + !snapshot + ) { + return null; + } + return { + nodeId, + path: path as PlacementNodeEvaluation['path'], + accepted: raw.accepted, + rejectionReasons: raw.rejectionReasons as PlacementRejectionReason[], + snapshot, + }; +} + +function parseAttempt(value: unknown): PlacementProvisioningAttempt | null { + const raw = record(value); + const size = vmSize(raw?.vmSize); + const location = stringValue(raw?.vmLocation, 64); + const outcome = raw?.outcome; + const failureReason = raw?.failureReason; + if ( + !size || + !location || + typeof outcome !== 'string' || + !PROVISIONING_OUTCOMES.has(outcome as PlacementProvisioningAttempt['outcome']) || + (failureReason !== undefined && + (typeof failureReason !== 'string' || + !PROVISIONING_FAILURES.has( + failureReason as NonNullable + ))) + ) { + return null; + } + return { + vmSize: size, + vmLocation: location, + outcome: outcome as PlacementProvisioningAttempt['outcome'], + ...(failureReason + ? { failureReason: failureReason as PlacementProvisioningAttempt['failureReason'] } + : {}), + }; +} + +function parseV2(value: unknown): PlacementExplanation | null { + const raw = record(value); + const path = raw?.selectionPath; + const selectedNodeId = raw?.selectedNodeId; + const summary = stringValue(raw?.summary, 300); + const decidedAt = stringValue(raw?.decidedAt, 64); + const updatedAt = stringValue(raw?.updatedAt, 64); + const request = parseRequest(raw?.request); + if ( + raw?.schemaVersion !== 2 || + !['reused', 'provisioned', 'failed'].includes(String(raw?.outcome)) || + typeof path !== 'string' || + !PATHS.has(path as PlacementSelectionPath) || + (selectedNodeId !== null && !stringValue(selectedNodeId, 128)) || + !summary || + !request || + !Array.isArray(raw?.evaluatedNodes) || + !Array.isArray(raw?.provisioningAttempts) || + !decidedAt || + !updatedAt + ) { + return null; + } + const evaluatedNodes = raw.evaluatedNodes.map(parseEvaluation); + const provisioningAttempts = raw.provisioningAttempts.map(parseAttempt); + if (evaluatedNodes.some((item) => !item) || provisioningAttempts.some((item) => !item)) { + return null; + } + return { + schemaVersion: 2, + outcome: raw.outcome as PlacementExplanation['outcome'], + selectionPath: path as PlacementSelectionPath, + selectedNodeId: selectedNodeId as string | null, + summary, + request, + evaluatedNodes: evaluatedNodes as PlacementNodeEvaluation[], + provisioningAttempts: provisioningAttempts as PlacementProvisioningAttempt[], + decidedAt, + updatedAt, + }; +} + +function parseLegacy(value: unknown): LegacyPlacementExplanation | null { + const raw = record(value); + const selectedVmSize = vmSize(raw?.selectedVmSize); + const vmSizeSource = raw?.vmSizeSource; + const reason = stringValue(raw?.reason, 500); + const decidedAt = stringValue(raw?.decidedAt, 64); + const reservation = record(raw?.reservation); + if ( + !selectedVmSize || + typeof vmSizeSource !== 'string' || + !VM_SIZE_SOURCES.has(vmSizeSource as ResourceRequirementsSource | 'explicit') || + !reason || + !decidedAt || + !reservation || + finiteNumber(reservation.cpuMillis) === null || + finiteNumber(reservation.memoryMb) === null || + finiteNumber(reservation.diskMb) === null || + typeof reservation.exclusiveNode !== 'boolean' || + finiteNumber(reservation.maxCoTenants, 1) === null || + typeof reservation.source !== 'string' || + !RESOURCE_SOURCES.has(reservation.source as ResourceRequirementsSource) || + !stringValue(reservation.sourceId, 128) || + finiteNumber(reservation.version, 1) === null + ) { + return null; + } + return { + selectedVmSize, + vmSizeSource: vmSizeSource as LegacyPlacementExplanation['vmSizeSource'], + reservation: { + cpuMillis: reservation.cpuMillis as number, + memoryMb: reservation.memoryMb as number, + diskMb: reservation.diskMb as number, + exclusiveNode: reservation.exclusiveNode, + maxCoTenants: reservation.maxCoTenants as number, + source: reservation.source as ResourceRequirementsSource, + sourceId: reservation.sourceId as string, + version: reservation.version as number, + }, + reason, + decidedAt, + }; +} + +export function parsePlacementExplanation( + value: unknown +): PlacementExplanation | LegacyPlacementExplanation | null { + return parseV2(value) ?? parseLegacy(value); +} + +export function parsePlacementExplanationJson( + raw: string | null | undefined +): PlacementExplanation | LegacyPlacementExplanation | null { + if (!raw) return null; + try { + return parsePlacementExplanation(JSON.parse(raw)); + } catch { + return null; + } +} + +export function isPlacementExplanationV2( + value: PlacementExplanation | LegacyPlacementExplanation +): value is PlacementExplanation { + return 'schemaVersion' in value && value.schemaVersion === 2; +} diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 6d438d3274..40d6f88462 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -629,7 +629,16 @@ export type { // Resource Requirements & Reservations export type { + LegacyPlacementExplanation, PlacementExplanation, + PlacementNodeEvaluation, + PlacementNodeSnapshot, + PlacementOutcome, + PlacementProvisioningAttempt, + PlacementProvisioningFailureReason, + PlacementRejectionReason, + PlacementRequestSnapshot, + PlacementSelectionPath, ResolvedResourceReservation, ResourceRequirements, ResourceRequirementsSource, diff --git a/packages/shared/src/types/resource.ts b/packages/shared/src/types/resource.ts index 7e8587e3e6..001ea764e1 100644 --- a/packages/shared/src/types/resource.ts +++ b/packages/shared/src/types/resource.ts @@ -67,8 +67,8 @@ export interface ResolvedResourceReservation { // Placement Explanation (audit trail) // ============================================================================= -/** Audit record explaining why a task was placed on a particular node/VM size. */ -export interface PlacementExplanation { +/** Legacy, unversioned audit record retained for safe parsing of historical rows. */ +export interface LegacyPlacementExplanation { /** The VM size that was selected. */ selectedVmSize: VMSize; /** Where the VM size came from. */ @@ -81,6 +81,97 @@ export interface PlacementExplanation { decidedAt: string; } +export type PlacementOutcome = 'reused' | 'provisioned' | 'failed'; + +export type PlacementSelectionPath = + | 'preferred' + | 'warm' + | 'capacity' + | 'trial' + | 'manual' + | 'provisioning'; + +export type PlacementRejectionReason = + | 'node-not-found' + | 'not-running' + | 'wrong-runtime' + | 'unhealthy' + | 'heartbeat-missing' + | 'heartbeat-stale' + | 'agent-not-ready' + | 'agent-version-mismatch' + | 'undersized' + | 'workspace-limit' + | 'cpu-threshold' + | 'memory-threshold' + | 'not-warm' + | 'warm-claim-lost'; + +export type PlacementProvisioningFailureReason = + | 'capacity-unavailable' + | 'node-limit' + | 'quota-exceeded' + | 'credentials-unavailable' + | 'provider-failed' + | 'provisioning-timeout' + | 'readiness-timeout' + | 'node-unavailable'; + +export interface PlacementRequestSnapshot { + runtime: 'vm'; + vmSize: VMSize; + vmLocation: string; + maxWorkspacesPerNode: number; + cpuThresholdPercent: number; + memoryThresholdPercent: number; + heartbeatStaleSeconds: number; +} + +export interface PlacementNodeSnapshot { + runtime: 'vm' | 'other'; + vmSize: string; + vmLocation: string; + healthStatus: 'healthy' | 'stale' | 'unhealthy' | 'unknown'; + agentVersionCompatible: boolean; + heartbeatAgeSeconds: number | null; + activeWorkspaceCount: number; + cpuLoadAvg1: number | null; + memoryPercent: number | null; +} + +export interface PlacementNodeEvaluation { + nodeId: string; + path: Exclude; + accepted: boolean; + rejectionReasons: PlacementRejectionReason[]; + snapshot: PlacementNodeSnapshot; +} + +export interface PlacementProvisioningAttempt { + vmSize: VMSize; + vmLocation: string; + outcome: 'started' | 'succeeded' | 'capacity-rejected' | 'failed'; + failureReason?: PlacementProvisioningFailureReason; +} + +/** + * Versioned, non-sensitive audit record for reusable-node selection and fallback provisioning. + * Raw agent versions, raw metrics, provider errors, credentials, prompts, and repository data + * are intentionally excluded from this contract. + */ +export interface PlacementExplanation { + schemaVersion: 2; + outcome: PlacementOutcome; + selectionPath: PlacementSelectionPath; + selectedNodeId: string | null; + summary: string; + request: PlacementRequestSnapshot; + evaluatedNodes: PlacementNodeEvaluation[]; + provisioningAttempts: PlacementProvisioningAttempt[]; + decidedAt: string; + updatedAt: string; +} + // ============================================================================= // Resolution Input (collector for the precedence chain) // ============================================================================= diff --git a/packages/shared/src/types/task.ts b/packages/shared/src/types/task.ts index 02eee07335..4e57b9f9c1 100644 --- a/packages/shared/src/types/task.ts +++ b/packages/shared/src/types/task.ts @@ -375,6 +375,11 @@ export interface Task { resolvedReservationJson: string | null; /** JSON snapshot of the PlacementExplanation. */ placementExplanationJson: string | null; + /** Safely parsed placement explanation for API consumers. */ + placementExplanation?: + | import('./resource').PlacementExplanation + | import('./resource').LegacyPlacementExplanation + | null; startedAt: string | null; completedAt: string | null; errorMessage: string | null; diff --git a/packages/shared/src/types/workspace.ts b/packages/shared/src/types/workspace.ts index 328b3ff939..5ab2d58747 100644 --- a/packages/shared/src/types/workspace.ts +++ b/packages/shared/src/types/workspace.ts @@ -239,6 +239,11 @@ export interface WorkspaceResponse { status: WorkspaceStatus; vmSize: VMSize; vmLocation: VMLocation; + /** Safely parsed reusable-node placement audit record. */ + placementExplanation?: + | import('./resource').PlacementExplanation + | import('./resource').LegacyPlacementExplanation + | null; workspaceProfile?: WorkspaceProfile | null; /** Selected devcontainer config name (subdirectory under .devcontainer/). null = auto-discover default. */ devcontainerConfigName?: string | null; diff --git a/packages/shared/tests/placement.test.ts b/packages/shared/tests/placement.test.ts new file mode 100644 index 0000000000..17fc8b5b0f --- /dev/null +++ b/packages/shared/tests/placement.test.ts @@ -0,0 +1,120 @@ +import { describe, expect, it } from 'vitest'; + +import { isPlacementExplanationV2, parsePlacementExplanationJson } from '../src/placement'; + +const validV2 = { + schemaVersion: 2, + outcome: 'reused', + selectionPath: 'capacity', + selectedNodeId: 'node-1', + summary: 'Reused node node-1 through the capacity path.', + request: { + runtime: 'vm', + vmSize: 'medium', + vmLocation: 'hel1', + maxWorkspacesPerNode: 5, + cpuThresholdPercent: 50, + memoryThresholdPercent: 50, + heartbeatStaleSeconds: 180, + }, + evaluatedNodes: [], + provisioningAttempts: [], + decidedAt: '2026-08-11T00:00:00.000Z', + updatedAt: '2026-08-11T00:00:00.000Z', +}; + +describe('parsePlacementExplanationJson', () => { + it('parses a valid v2 placement record', () => { + const parsed = parsePlacementExplanationJson(JSON.stringify(validV2)); + expect(parsed && isPlacementExplanationV2(parsed)).toBe(true); + expect(parsed).toEqual(validV2); + }); + + it('round-trips high configured limits and typed provisioning timeouts', () => { + const configured = { + ...validV2, + request: { + ...validV2.request, + maxWorkspacesPerNode: 20000, + heartbeatStaleSeconds: 172800, + }, + provisioningAttempts: [ + { + vmSize: 'medium', + vmLocation: 'hel1', + outcome: 'failed', + failureReason: 'provisioning-timeout', + }, + ], + }; + + expect(parsePlacementExplanationJson(JSON.stringify(configured))).toEqual(configured); + }); + + it('parses a legacy placement record', () => { + const canary = 'CANARY_LEGACY_SECRET'; + const parsed = parsePlacementExplanationJson( + JSON.stringify({ + selectedVmSize: 'medium', + vmSizeSource: 'project', + reservation: { + cpuMillis: 1000, + memoryMb: 2048, + diskMb: 4096, + exclusiveNode: false, + maxCoTenants: 5, + source: 'project', + sourceId: 'project-1', + version: 1, + providerError: canary, + }, + reason: 'Legacy record', + decidedAt: '2026-08-10T00:00:00.000Z', + rawAgentVersion: canary, + }) + ); + expect(parsed).toMatchObject({ reason: 'Legacy record' }); + expect(JSON.stringify(parsed)).not.toContain(canary); + expect(parsed).not.toHaveProperty('rawAgentVersion'); + expect(parsed).not.toHaveProperty('reservation.providerError'); + }); + + it('allows explicit VM-size selection but rejects it as a reservation source', () => { + const legacy = { + selectedVmSize: 'medium', + vmSizeSource: 'explicit', + reservation: { + cpuMillis: 1000, + memoryMb: 2048, + diskMb: 4096, + exclusiveNode: false, + maxCoTenants: 5, + source: 'project', + sourceId: 'project-1', + version: 1, + }, + reason: 'Legacy explicit size', + decidedAt: '2026-08-10T00:00:00.000Z', + }; + + expect(parsePlacementExplanationJson(JSON.stringify(legacy))).toMatchObject({ + vmSizeSource: 'explicit', + reservation: { source: 'project' }, + }); + expect( + parsePlacementExplanationJson( + JSON.stringify({ ...legacy, reservation: { ...legacy.reservation, source: 'explicit' } }) + ) + ).toBeNull(); + }); + + it.each([ + null, + '', + '{bad json', + JSON.stringify({ ...validV2, schemaVersion: 99 }), + JSON.stringify({ ...validV2, evaluatedNodes: [{ raw: 'unvalidated' }] }), + ])('rejects absent or malformed data safely', (raw) => { + expect(parsePlacementExplanationJson(raw)).toBeNull(); + }); +}); diff --git a/scripts/deploy/resolve-vm-agent-release.ts b/scripts/deploy/resolve-vm-agent-release.ts new file mode 100644 index 0000000000..3e5a664807 --- /dev/null +++ b/scripts/deploy/resolve-vm-agent-release.ts @@ -0,0 +1,91 @@ +#!/usr/bin/env tsx + +import { appendFileSync } from 'node:fs'; +import { resolve } from 'node:path'; + +import { DEPLOYMENT_CONFIG } from './config.js'; +import { + computeVmAgentBuildFingerprint, + createGitBuildInputReader, + getDeployedVmAgentRelease, + isValidVmAgentReleaseVersion, + resolveVmAgentRelease, +} from './vm-agent-release.js'; + +function requireEnv(name: string): string { + const value = process.env[name]?.trim(); + if (!value) throw new Error(`${name} is required to resolve the VM-agent release`); + return value; +} + +async function main(): Promise { + const repositoryRoot = resolve(import.meta.dirname, '../..'); + const targetVersion = requireEnv('DEPLOY_SHA').toLowerCase(); + const stack = requireEnv('PULUMI_STACK'); + const accountId = requireEnv('CF_ACCOUNT_ID'); + const apiToken = process.env.CF_API_TOKEN?.trim() || process.env.CLOUDFLARE_API_TOKEN?.trim(); + if (!apiToken) { + throw new Error('CF_API_TOKEN or CLOUDFLARE_API_TOKEN is required'); + } + + const reader = createGitBuildInputReader(repositoryRoot); + const targetFingerprint = computeVmAgentBuildFingerprint(targetVersion, reader); + const workerName = DEPLOYMENT_CONFIG.resources.workerName(stack); + const deployed = await getDeployedVmAgentRelease({ + accountId, + workerName, + apiToken, + }); + + let inferredPriorFingerprint: string | null = null; + if ( + deployed?.requiredVersion && + !deployed.fingerprint && + isValidVmAgentReleaseVersion(deployed.requiredVersion) + ) { + try { + inferredPriorFingerprint = computeVmAgentBuildFingerprint(deployed.requiredVersion, reader, { + allowLegacyMissingMarker: true, + }); + } catch { + console.warn( + 'Unable to prove the legacy VM-agent fingerprint from the deployed release; a fresh release will be published.' + ); + } + } + + const resolution = resolveVmAgentRelease({ + targetVersion, + targetFingerprint, + deployed, + inferredPriorFingerprint, + skipAgent: process.env.SKIP_AGENT === 'true', + }); + + const githubOutput = requireEnv('GITHUB_OUTPUT'); + appendFileSync( + githubOutput, + [ + `build_agent=${String(resolution.buildAgent)}`, + `required_version=${resolution.requiredVersion}`, + `fingerprint=${resolution.fingerprint}`, + `reason=${resolution.reason}`, + '', + ].join('\n') + ); + + console.log( + JSON.stringify({ + workerName, + buildAgent: resolution.buildAgent, + requiredVersion: resolution.requiredVersion, + fingerprint: resolution.fingerprint, + reason: resolution.reason, + }) + ); +} + +main().catch((error: unknown) => { + console.error(error instanceof Error ? error.message : 'VM-agent release resolution failed'); + process.exitCode = 1; +}); diff --git a/scripts/deploy/sync-wrangler-config.ts b/scripts/deploy/sync-wrangler-config.ts index 951e130b0f..c944ba051c 100644 --- a/scripts/deploy/sync-wrangler-config.ts +++ b/scripts/deploy/sync-wrangler-config.ts @@ -558,6 +558,7 @@ function getApiWorkerVars( 'SETUP_SESSION_SWEEP_MAX_CANDIDATES', 'POOL_LEASE_BUFFER_MS', 'VM_AGENT_REQUIRED_VERSION', + 'VM_AGENT_BUILD_FINGERPRINT', ]), // AI Gateway ID matches the resource prefix (created by configure-ai-gateway.sh) AI_GATEWAY_ID: DEPLOYMENT_CONFIG.prefix, diff --git a/scripts/deploy/vm-agent-compatibility-version.txt b/scripts/deploy/vm-agent-compatibility-version.txt new file mode 100644 index 0000000000..d00491fd7e --- /dev/null +++ b/scripts/deploy/vm-agent-compatibility-version.txt @@ -0,0 +1 @@ +1 diff --git a/scripts/deploy/vm-agent-release.ts b/scripts/deploy/vm-agent-release.ts new file mode 100644 index 0000000000..0966ce949a --- /dev/null +++ b/scripts/deploy/vm-agent-release.ts @@ -0,0 +1,256 @@ +import { execFileSync, type ExecFileSyncOptionsWithStringEncoding } from 'node:child_process'; +import { createHash } from 'node:crypto'; + +export const VM_AGENT_REQUIRED_VERSION_BINDING = 'VM_AGENT_REQUIRED_VERSION'; +export const VM_AGENT_BUILD_FINGERPRINT_BINDING = 'VM_AGENT_BUILD_FINGERPRINT'; +export const VM_AGENT_COMPATIBILITY_MARKER_PATH = + 'scripts/deploy/vm-agent-compatibility-version.txt'; +export const LEGACY_VM_AGENT_COMPATIBILITY_VERSION = '1'; + +const CLOUDFLARE_API_BASE_URL = 'https://api.cloudflare.com/client/v4'; +const VM_AGENT_SOURCE_PATH = 'packages/vm-agent'; +const FINGERPRINT_SCHEMA_VERSION = 1; +const GIT_SHA_PATTERN = /^[0-9a-f]{40}$/; +const FINGERPRINT_PATTERN = /^[0-9a-f]{64}$/; + +export type VmAgentReleaseReason = + | 'first-deploy' + | 'inputs-changed' + | 'inputs-unchanged' + | 'legacy-fingerprint-match' + | 'legacy-fingerprint-unproven'; + +export interface DeployedVmAgentRelease { + requiredVersion: string | null; + fingerprint: string | null; +} + +export interface VmAgentReleaseResolution { + buildAgent: boolean; + requiredVersion: string; + fingerprint: string; + reason: VmAgentReleaseReason; +} + +export interface VmAgentBuildInputReader { + listTrackedInputs(ref: string): string; + readCompatibilityVersion(ref: string): string | null; +} + +function normalizeGitSha(value: string | null | undefined): string | null { + const normalized = value?.trim().toLowerCase(); + return normalized && GIT_SHA_PATTERN.test(normalized) ? normalized : null; +} + +function normalizeFingerprint(value: string | null | undefined): string | null { + const normalized = value?.trim().toLowerCase(); + return normalized && FINGERPRINT_PATTERN.test(normalized) ? normalized : null; +} + +function requireGitCommitSha(ref: string): string { + const commitSha = normalizeGitSha(ref); + if (!commitSha) { + throw new Error('VM-agent build inputs must be read from a valid 40-character commit SHA'); + } + return commitSha; +} + +function runGit(repositoryRoot: string, args: string[]): string { + const options: ExecFileSyncOptionsWithStringEncoding = { + cwd: repositoryRoot, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }; + // Security: every dynamic ref in the private callers is a strict hex SHA, + // arguments never enter a shell, and Git receives an option terminator. + // Sonar cannot infer the custom validator, so suppress that false positive. + return execFileSync('git', args, options); // NOSONAR +} + +export function createGitBuildInputReader(repositoryRoot: string): VmAgentBuildInputReader { + return { + listTrackedInputs(ref) { + const commitSha = requireGitCommitSha(ref); + return runGit(repositoryRoot, [ + 'ls-tree', + '-r', + '-z', + '--end-of-options', + commitSha, + '--', + VM_AGENT_SOURCE_PATH, + ]); + }, + readCompatibilityVersion(ref) { + const commitSha = requireGitCommitSha(ref); + try { + return runGit(repositoryRoot, [ + 'show', + '--end-of-options', + `${commitSha}:${VM_AGENT_COMPATIBILITY_MARKER_PATH}`, + ]); + } catch { + return null; + } + }, + }; +} + +export function computeVmAgentBuildFingerprint( + ref: string, + reader: VmAgentBuildInputReader, + options: { allowLegacyMissingMarker?: boolean } = {} +): string { + const trackedInputs = reader.listTrackedInputs(ref); + if (!trackedInputs) { + throw new Error(`No tracked VM-agent build inputs found at ${ref}`); + } + + const marker = reader.readCompatibilityVersion(ref)?.trim(); + const compatibilityVersion = + marker || (options.allowLegacyMissingMarker ? LEGACY_VM_AGENT_COMPATIBILITY_VERSION : null); + if (!compatibilityVersion) { + throw new Error( + `Missing VM-agent compatibility marker at ${ref}:${VM_AGENT_COMPATIBILITY_MARKER_PATH}` + ); + } + + return createHash('sha256') + .update( + JSON.stringify({ + schemaVersion: FINGERPRINT_SCHEMA_VERSION, + trackedInputs, + compatibilityVersion, + }) + ) + .digest('hex'); +} + +export function resolveVmAgentRelease(input: { + targetVersion: string; + targetFingerprint: string; + deployed: DeployedVmAgentRelease | null; + inferredPriorFingerprint?: string | null; + skipAgent: boolean; +}): VmAgentReleaseResolution { + const targetVersion = normalizeGitSha(input.targetVersion); + const targetFingerprint = normalizeFingerprint(input.targetFingerprint); + if (!targetVersion || !targetFingerprint) { + throw new Error('Target VM-agent version and fingerprint must be valid lowercase hashes'); + } + + const priorVersion = normalizeGitSha(input.deployed?.requiredVersion); + const recordedFingerprint = normalizeFingerprint(input.deployed?.fingerprint); + const inferredFingerprint = normalizeFingerprint(input.inferredPriorFingerprint); + const priorFingerprint = recordedFingerprint ?? inferredFingerprint; + + if (!priorVersion) { + if (input.skipAgent) { + throw new Error('skip_agent cannot be used before a published VM-agent release is available'); + } + return { + buildAgent: true, + requiredVersion: targetVersion, + fingerprint: targetFingerprint, + reason: 'first-deploy', + }; + } + + if (priorFingerprint === targetFingerprint) { + return { + buildAgent: false, + requiredVersion: priorVersion, + fingerprint: targetFingerprint, + reason: recordedFingerprint ? 'inputs-unchanged' : 'legacy-fingerprint-match', + }; + } + + if (input.skipAgent) { + throw new Error( + 'skip_agent cannot bypass changed or unproven VM-agent build inputs; publish the agent release' + ); + } + + return { + buildAgent: true, + requiredVersion: targetVersion, + fingerprint: targetFingerprint, + reason: priorFingerprint ? 'inputs-changed' : 'legacy-fingerprint-unproven', + }; +} + +function readPlainTextBinding(bindings: unknown[], name: string): string | null { + const matches = bindings.filter( + (binding): binding is Record => + typeof binding === 'object' && + binding !== null && + !Array.isArray(binding) && + 'name' in binding && + binding.name === name + ); + if (matches.length > 1) { + throw new Error(`Cloudflare returned duplicate ${name} bindings`); + } + const binding = matches[0]; + if (!binding || binding.type !== 'plain_text' || typeof binding.text !== 'string') { + return null; + } + return binding.text; +} + +export async function getDeployedVmAgentRelease(input: { + accountId: string; + workerName: string; + apiToken: string; + fetchImpl?: typeof fetch; +}): Promise { + const fetchImpl = input.fetchImpl ?? fetch; + const url = + `${CLOUDFLARE_API_BASE_URL}/accounts/${encodeURIComponent(input.accountId)}` + + `/workers/scripts/${encodeURIComponent(input.workerName)}/settings`; + const response = await fetchImpl(url, { + headers: { Authorization: `Bearer ${input.apiToken}` }, + }); + if (response.status === 404) { + return null; + } + if (!response.ok) { + throw new Error( + `Failed to read deployed VM-agent release metadata for Worker "${input.workerName}" (HTTP ${response.status})` + ); + } + + let payload: unknown; + try { + payload = await response.json(); + } catch { + throw new Error('Cloudflare returned invalid JSON for Worker release metadata'); + } + if ( + typeof payload !== 'object' || + payload === null || + !('success' in payload) || + payload.success !== true || + !('result' in payload) || + typeof payload.result !== 'object' || + payload.result === null || + !('bindings' in payload.result) || + !Array.isArray(payload.result.bindings) + ) { + throw new Error('Cloudflare returned an invalid Worker settings response'); + } + + // SECURITY: the response also contains all other plaintext Worker bindings. + // Return only the two allowlisted release fields and never log the payload. + return { + requiredVersion: readPlainTextBinding( + payload.result.bindings, + VM_AGENT_REQUIRED_VERSION_BINDING + ), + fingerprint: readPlainTextBinding(payload.result.bindings, VM_AGENT_BUILD_FINGERPRINT_BINDING), + }; +} + +export function isValidVmAgentReleaseVersion(value: string | null | undefined): boolean { + return normalizeGitSha(value) !== null; +} diff --git a/scripts/quality/deploy-reusable-workflow.test.ts b/scripts/quality/deploy-reusable-workflow.test.ts index 808ddba6d8..e9229400a4 100644 --- a/scripts/quality/deploy-reusable-workflow.test.ts +++ b/scripts/quality/deploy-reusable-workflow.test.ts @@ -118,7 +118,7 @@ describe('deploy reusable workflow', () => { expect(block).not.toContain('npx wrangler'); }); - it('behaviorally verifies the checked-out SHA and skip-agent output', () => { + it('behaviorally verifies the checked-out SHA without deciding the agent release', () => { const tmp = mkdtempSync(join(tmpdir(), 'sam-deploy-sha-')); const script = stepRunScript('Resolve and Verify Deployment SHA'); @@ -140,36 +140,19 @@ describe('deploy reusable workflow', () => { env: { ...process.env, EXPECTED_DEPLOY_SHA: head, - SKIP_AGENT: 'false', GITHUB_OUTPUT: normalOutput, }, encoding: 'utf8', }); expect(normal.status).toBe(0); expect(readFileSync(normalOutput, 'utf8')).toContain(`value=${head}`); - expect(readFileSync(normalOutput, 'utf8')).toContain(`agent_version=${head}`); - - const skippedOutput = join(tmp, 'skipped-output.txt'); - const skipped = spawnSync('bash', ['-c', script], { - cwd: tmp, - env: { - ...process.env, - EXPECTED_DEPLOY_SHA: head, - SKIP_AGENT: 'true', - GITHUB_OUTPUT: skippedOutput, - }, - encoding: 'utf8', - }); - expect(skipped.status).toBe(0); - expect(readFileSync(skippedOutput, 'utf8')).toContain(`value=${head}`); - expect(readFileSync(skippedOutput, 'utf8')).toMatch(/agent_version=\n/); + expect(readFileSync(normalOutput, 'utf8')).not.toContain('agent_version='); const mismatch = spawnSync('bash', ['-c', script], { cwd: tmp, env: { ...process.env, EXPECTED_DEPLOY_SHA: 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb', - SKIP_AGENT: 'false', GITHUB_OUTPUT: join(tmp, 'mismatch-output.txt'), }, encoding: 'utf8', @@ -216,7 +199,10 @@ describe('deploy reusable workflow', () => { expect(initialSync).toContain('BASE_DOMAIN: ${{ vars.BASE_DOMAIN }}'); expect(initialSync).toContain('RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }}'); expect(initialSync).toContain( - 'VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }}' + 'VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}' + ); + expect(initialSync).toContain( + 'VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }}' ); expect(initialSync).toContain( 'ARTIFACTS_BINDING_ENABLED: ${{ vars.ARTIFACTS_BINDING_ENABLED }}' @@ -226,7 +212,10 @@ describe('deploy reusable workflow', () => { expect(firstDeployResync).toContain('BASE_DOMAIN: ${{ vars.BASE_DOMAIN }}'); expect(firstDeployResync).toContain('RESOURCE_PREFIX: ${{ steps.prefix.outputs.value }}'); expect(firstDeployResync).toContain( - 'VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }}' + 'VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}' + ); + expect(firstDeployResync).toContain( + 'VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }}' ); expect(firstDeployResync).toContain( 'ARTIFACTS_BINDING_ENABLED: ${{ vars.ARTIFACTS_BINDING_ENABLED }}' @@ -245,8 +234,12 @@ describe('deploy reusable workflow', () => { expect(firstDeployResync).toContain(mapping); } + const generatedReleaseVars = new Set([ + 'VM_AGENT_REQUIRED_VERSION', + 'VM_AGENT_BUILD_FINGERPRINT', + ]); for (const envVar of extractOptionalWorkerEnvVars().filter( - (name) => name !== 'VM_AGENT_REQUIRED_VERSION' + (name) => !generatedReleaseVars.has(name) )) { const mapping = `${envVar}: \${{ vars.${envVar} }}`; expect(initialSync).toContain(mapping); @@ -394,14 +387,32 @@ describe('deploy reusable workflow', () => { expect(sync).toContain('POOL_LEASE_BUFFER_MS: ${{ vars.POOL_LEASE_BUFFER_MS }}'); }); - it('versions the R2 vm-agent binaries with the same commit SHA as the container binary', () => { + it('builds reusable-VM binaries only for the resolved published release', () => { const build = stepBlock('Build VM Agent'); - // Both the container-baked binary and the R2-uploaded binaries must report - // the deploy commit SHA so a running agent can be correlated to its artifact. expect(build).toContain('make -C packages/vm-agent build-all'); - expect(build).toContain('VERSION="$DEPLOY_SHA"'); - expect(build).toContain('DEPLOY_SHA: ${{ steps.deploy-sha.outputs.value }}'); + expect(build).toContain('VERSION="$AGENT_VERSION"'); + expect(build).toContain("steps.vm-agent-release.outputs.build_agent == 'true'"); + expect(build).toContain( + 'AGENT_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}' + ); + }); + + it('resolves release metadata before Wrangler sync and artifact publication', () => { + const resolutionIndex = workflow.indexOf('- name: Resolve VM Agent Release'); + const syncIndex = workflow.indexOf('- name: Sync Wrangler Config (API + Tail Worker)'); + const buildIndex = workflow.indexOf('- name: Build VM Agent'); + const uploadIndex = workflow.indexOf('- name: Upload VM Agent Binaries'); + const deployIndex = workflow.indexOf('- name: Deploy API Worker'); + + expect(resolutionIndex).toBeGreaterThan(-1); + expect(syncIndex).toBeGreaterThan(resolutionIndex); + expect(buildIndex).toBeGreaterThan(syncIndex); + expect(uploadIndex).toBeGreaterThan(buildIndex); + expect(deployIndex).toBeGreaterThan(uploadIndex); + expect(stepBlock('Resolve VM Agent Release')).toContain( + 'pnpm tsx scripts/deploy/resolve-vm-agent-release.ts' + ); }); it('continues deployment when workers.dev subdomain setup succeeds', () => { diff --git a/scripts/quality/deploy-safety.test.ts b/scripts/quality/deploy-safety.test.ts index e015970e7a..e26a4f5aa6 100644 --- a/scripts/quality/deploy-safety.test.ts +++ b/scripts/quality/deploy-safety.test.ts @@ -1,10 +1,11 @@ +import { execFileSync, spawnSync } from 'node:child_process'; import { existsSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { execFileSync, spawnSync } from 'node:child_process'; import { describe, expect, it, vi } from 'vitest'; +import { validatePulumiOutputs } from '../deploy/sync-wrangler-config.js'; import { normalizeSha, selectSuccessfulCiRun, @@ -12,7 +13,6 @@ import { validateEmergencyOverrideReason, validateProductionDispatch, } from '../deploy/validate-production-dispatch.js'; -import { validatePulumiOutputs } from '../deploy/sync-wrangler-config.js'; const greenSha = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'; const redSha = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'; @@ -291,12 +291,16 @@ describe('deployment workflow safety wiring', () => { expect(reusable).toContain('target_commit_sha:'); expect(reusable).toContain('ref: ${{ inputs.target_commit_sha || github.sha }}'); + expect(reusable).toContain('fetch-depth: 0'); expect(reusable).toContain('- name: Resolve and Verify Deployment SHA'); expect(reusable).toContain('ACTUAL_DEPLOY_SHA=$(git rev-parse HEAD)'); - expect(reusable).toContain('echo "agent_version=" >> "$GITHUB_OUTPUT"'); - expect(reusable).toContain('echo "agent_version=$ACTUAL_DEPLOY_SHA" >> "$GITHUB_OUTPUT"'); + expect(reusable).toContain('- name: Resolve VM Agent Release'); + expect(reusable).toContain('scripts/deploy/resolve-vm-agent-release.ts'); + expect(reusable).toContain( + 'VM_AGENT_REQUIRED_VERSION: ${{ steps.vm-agent-release.outputs.required_version }}' + ); expect(reusable).toContain( - 'VM_AGENT_REQUIRED_VERSION: ${{ steps.deploy-sha.outputs.agent_version }}' + 'VM_AGENT_BUILD_FINGERPRINT: ${{ steps.vm-agent-release.outputs.fingerprint }}' ); expect(reusable).toContain('steps.deploy-sha.outputs.value'); }); diff --git a/scripts/quality/sync-wrangler-config.test.ts b/scripts/quality/sync-wrangler-config.test.ts index 96ecf2e0f5..507e96cfa5 100644 --- a/scripts/quality/sync-wrangler-config.test.ts +++ b/scripts/quality/sync-wrangler-config.test.ts @@ -189,6 +189,7 @@ describe('sync wrangler config', () => { it('generates Cloudflare container max_instances with unchanged safe defaults', () => { vi.stubEnv('RESOURCE_PREFIX', 's123abc'); vi.stubEnv('VM_AGENT_REQUIRED_VERSION', 'deploy-sha'); + vi.stubEnv('VM_AGENT_BUILD_FINGERPRINT', 'fingerprint-sha256'); const containers = [ { @@ -208,6 +209,7 @@ describe('sync wrangler config', () => { const envConfig = generateApiWorkerEnv({ containers }, outputs, 'prod', false, false, null); expect(envConfig.vars?.VM_AGENT_REQUIRED_VERSION).toBe('deploy-sha'); + expect(envConfig.vars?.VM_AGENT_BUILD_FINGERPRINT).toBe('fingerprint-sha256'); expect(envConfig.containers).toEqual([ { diff --git a/scripts/quality/vm-agent-release.test.ts b/scripts/quality/vm-agent-release.test.ts new file mode 100644 index 0000000000..7ed36cc1e7 --- /dev/null +++ b/scripts/quality/vm-agent-release.test.ts @@ -0,0 +1,321 @@ +import { execFileSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { describe, expect, it, vi } from 'vitest'; + +import { + computeVmAgentBuildFingerprint, + createGitBuildInputReader, + type DeployedVmAgentRelease, + getDeployedVmAgentRelease, + LEGACY_VM_AGENT_COMPATIBILITY_VERSION, + resolveVmAgentRelease, + type VmAgentBuildInputReader, +} from '../deploy/vm-agent-release'; + +const OLD_VERSION = '1'.repeat(40); +const TARGET_VERSION = '2'.repeat(40); +const TARGET_FINGERPRINT = 'a'.repeat(64); +const OTHER_FINGERPRINT = 'b'.repeat(64); + +function deployed(fingerprint: string | null = TARGET_FINGERPRINT): DeployedVmAgentRelease { + return { requiredVersion: OLD_VERSION, fingerprint }; +} + +function reader(input: string, marker: string | null = '1'): VmAgentBuildInputReader { + return { + listTrackedInputs: vi.fn(() => input), + readCompatibilityVersion: vi.fn(() => marker), + }; +} + +describe('computeVmAgentBuildFingerprint', () => { + it('is deterministic for the tracked package tree and compatibility marker', () => { + const first = computeVmAgentBuildFingerprint('ref-a', reader('tree-a')); + const second = computeVmAgentBuildFingerprint('ref-b', reader('tree-a')); + expect(first).toBe(second); + expect(first).toMatch(/^[0-9a-f]{64}$/); + }); + + it('changes for source, dependency, toolchain, or Makefile tree changes', () => { + const baseline = computeVmAgentBuildFingerprint('ref', reader('tree-a')); + expect(computeVmAgentBuildFingerprint('ref', reader('tree-b'))).not.toBe(baseline); + }); + + it('changes when the explicit compatibility marker changes', () => { + const baseline = computeVmAgentBuildFingerprint('ref', reader('tree-a', '1')); + expect(computeVmAgentBuildFingerprint('ref', reader('tree-a', '2'))).not.toBe(baseline); + }); + + it('uses the documented legacy marker only for first-rollout inference', () => { + const explicit = computeVmAgentBuildFingerprint( + 'target', + reader('tree-a', LEGACY_VM_AGENT_COMPATIBILITY_VERSION) + ); + const inferred = computeVmAgentBuildFingerprint('prior', reader('tree-a', null), { + allowLegacyMissingMarker: true, + }); + expect(inferred).toBe(explicit); + expect(() => computeVmAgentBuildFingerprint('target', reader('tree-a', null))).toThrow( + 'Missing VM-agent compatibility marker' + ); + }); + + it('tracks the real VM-agent git tree and marker while ignoring unrelated files', () => { + const repositoryRoot = mkdtempSync(join(tmpdir(), 'sam-vm-agent-release-')); + const git = (...args: string[]) => + execFileSync('git', args, { + cwd: repositoryRoot, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); + const commit = (message: string) => { + git('add', '.'); + git('commit', '-m', message); + return git('rev-parse', 'HEAD'); + }; + + try { + git('init'); + git('config', 'user.name', 'SAM test'); + git('config', 'user.email', 'sam-test@example.invalid'); + mkdirSync(join(repositoryRoot, 'packages/vm-agent'), { recursive: true }); + mkdirSync(join(repositoryRoot, 'scripts/deploy'), { recursive: true }); + mkdirSync(join(repositoryRoot, 'apps/api'), { recursive: true }); + const vmAgentFiles = ['main.go', 'go.mod', 'go.sum', 'Makefile']; + for (const filename of vmAgentFiles) { + writeFileSync(join(repositoryRoot, 'packages/vm-agent', filename), `${filename} v1\n`); + } + writeFileSync( + join(repositoryRoot, 'scripts/deploy/vm-agent-compatibility-version.txt'), + '1\n' + ); + writeFileSync(join(repositoryRoot, 'apps/api/index.ts'), 'unrelated v1\n'); + + const reader = createGitBuildInputReader(repositoryRoot); + const baselineRef = commit('baseline'); + const baseline = computeVmAgentBuildFingerprint(baselineRef, reader); + + writeFileSync(join(repositoryRoot, 'apps/api/index.ts'), 'unrelated v2\n'); + const unrelatedRef = commit('unrelated API change'); + expect(computeVmAgentBuildFingerprint(unrelatedRef, reader)).toBe(baseline); + + let previous = baseline; + for (const filename of vmAgentFiles) { + writeFileSync(join(repositoryRoot, 'packages/vm-agent', filename), `${filename} v2\n`); + const ref = commit(`change ${filename}`); + const fingerprint = computeVmAgentBuildFingerprint(ref, reader); + expect(fingerprint).not.toBe(previous); + previous = fingerprint; + } + + writeFileSync( + join(repositoryRoot, 'scripts/deploy/vm-agent-compatibility-version.txt'), + '2\n' + ); + const markerRef = commit('change compatibility marker'); + expect(computeVmAgentBuildFingerprint(markerRef, reader)).not.toBe(previous); + } finally { + rmSync(repositoryRoot, { recursive: true, force: true }); + } + }); +}); + +describe('createGitBuildInputReader', () => { + it('rejects non-commit refs before invoking Git build-input reads', () => { + const reader = createGitBuildInputReader(process.cwd()); + + expect(() => reader.listTrackedInputs('--help')).toThrow( + 'VM-agent build inputs must be read from a valid 40-character commit SHA' + ); + expect(() => reader.readCompatibilityVersion('HEAD^{tree}')).toThrow( + 'VM-agent build inputs must be read from a valid 40-character commit SHA' + ); + }); +}); + +describe('resolveVmAgentRelease', () => { + it('carries the last published version and skips build for unchanged inputs', () => { + expect( + resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: deployed(), + skipAgent: false, + }) + ).toEqual({ + buildAgent: false, + requiredVersion: OLD_VERSION, + fingerprint: TARGET_FINGERPRINT, + reason: 'inputs-unchanged', + }); + }); + + it('publishes and advances when tracked inputs changed', () => { + expect( + resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: deployed(OTHER_FINGERPRINT), + skipAgent: false, + }) + ).toEqual({ + buildAgent: true, + requiredVersion: TARGET_VERSION, + fingerprint: TARGET_FINGERPRINT, + reason: 'inputs-changed', + }); + }); + + it('publishes on first deployment', () => { + const result = resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: null, + skipAgent: false, + }); + expect(result.buildAgent).toBe(true); + expect(result.requiredVersion).toBe(TARGET_VERSION); + expect(result.reason).toBe('first-deploy'); + }); + + it('carries a legacy release when its inferred fingerprint matches', () => { + const result = resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: deployed(null), + inferredPriorFingerprint: TARGET_FINGERPRINT, + skipAgent: false, + }); + expect(result).toMatchObject({ + buildAgent: false, + requiredVersion: OLD_VERSION, + reason: 'legacy-fingerprint-match', + }); + }); + + it('publishes when legacy equivalence cannot be proven', () => { + const result = resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: deployed(null), + inferredPriorFingerprint: null, + skipAgent: false, + }); + expect(result).toMatchObject({ + buildAgent: true, + requiredVersion: TARGET_VERSION, + reason: 'legacy-fingerprint-unproven', + }); + }); + + it('preserves the published version for unchanged skip_agent', () => { + const result = resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: deployed(), + skipAgent: true, + }); + expect(result.buildAgent).toBe(false); + expect(result.requiredVersion).toBe(OLD_VERSION); + }); + + it('rejects first-deploy, changed-input, and unproven skip_agent', () => { + for (const prior of [null, deployed(OTHER_FINGERPRINT), deployed(null)]) { + expect(() => + resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: prior, + skipAgent: true, + }) + ).toThrow(/skip_agent/); + } + }); + + it('never carries an empty or malformed deployed required version', () => { + for (const requiredVersion of [null, '', 'not-a-sha']) { + const result = resolveVmAgentRelease({ + targetVersion: TARGET_VERSION, + targetFingerprint: TARGET_FINGERPRINT, + deployed: { requiredVersion, fingerprint: TARGET_FINGERPRINT }, + skipAgent: false, + }); + expect(result.buildAgent).toBe(true); + expect(result.requiredVersion).toBe(TARGET_VERSION); + } + }); +}); + +describe('getDeployedVmAgentRelease', () => { + it('returns null only for a confirmed missing Worker', async () => { + const fetchImpl = vi.fn(async () => new Response(null, { status: 404 })); + await expect( + getDeployedVmAgentRelease({ + accountId: 'account', + workerName: 'worker', + apiToken: 'token', + fetchImpl, + }) + ).resolves.toBeNull(); + }); + + it('allowlists release bindings without returning other plaintext values', async () => { + const canary = 'CANARY_SUPER_SECRET_DO_NOT_COPY'; + const fetchImpl = vi.fn(async () => + Response.json({ + success: true, + result: { + bindings: [ + { name: 'SETUP_TOKEN', type: 'plain_text', text: canary }, + { + name: 'VM_AGENT_REQUIRED_VERSION', + type: 'plain_text', + text: OLD_VERSION, + }, + { + name: 'VM_AGENT_BUILD_FINGERPRINT', + type: 'plain_text', + text: TARGET_FINGERPRINT, + }, + ], + }, + }) + ); + const result = await getDeployedVmAgentRelease({ + accountId: 'account', + workerName: 'worker', + apiToken: 'token', + fetchImpl, + }); + expect(result).toEqual({ + requiredVersion: OLD_VERSION, + fingerprint: TARGET_FINGERPRINT, + }); + expect(JSON.stringify(result)).not.toContain(canary); + }); + + it('fails closed on unreadable or malformed settings state', async () => { + const forbidden = vi.fn(async () => new Response(null, { status: 403 })); + await expect( + getDeployedVmAgentRelease({ + accountId: 'account', + workerName: 'worker', + apiToken: 'token', + fetchImpl: forbidden, + }) + ).rejects.toThrow('HTTP 403'); + + const malformed = vi.fn(async () => Response.json({ success: true, result: {} })); + await expect( + getDeployedVmAgentRelease({ + accountId: 'account', + workerName: 'worker', + apiToken: 'token', + fetchImpl: malformed, + }) + ).rejects.toThrow('invalid Worker settings response'); + }); +}); diff --git a/tasks/active/2026-08-11-vm-agent-release-placement-explanation.md b/tasks/active/2026-08-11-vm-agent-release-placement-explanation.md new file mode 100644 index 0000000000..a785c5543a --- /dev/null +++ b/tasks/active/2026-08-11-vm-agent-release-placement-explanation.md @@ -0,0 +1,264 @@ +# Preserve VM-agent releases and explain reusable-node placement + +## Problem Statement + +Production deploy run `31492102025` set `VM_AGENT_REQUIRED_VERSION` to +`fc1e394217248c3bd004b2e6619cf2344eade7e3` even though `packages/vm-agent` had not +changed since the build reported by healthy node `01KZR2JAP92AK3SKW951E4H21M` +(`23e7adc23954d2b3a231b942edbb3195a6442301`). The exact compatibility gate correctly +rejected that node, so task `01KZRHQ4PVD1V55YP18H3BWKBF` provisioned another VM even +though the old medium/hel1 node had only 2/5 workspaces and low load. Both the task and +workspace had null `placement_explanation_json`, so the reason was not directly +observable. + +The fix must separate a VM-agent release identity from an unrelated application deploy +identity without weakening exact compatibility, and it must make every reusable-node +decision durable and explainable. + +## Hard-Gate Review + +The plan from SAM Idea `01KZRMP0J6ZEXT5KEGCSKFSPMH` is technically sound and is approved +for implementation with these safety-preserving refinements: + +1. A `skip_agent` request is allowed to carry a release only when the deterministic + build fingerprint is unchanged. If inputs changed or no prior release is provable, + the deploy fails closed instead of deploying controller changes against an + unpublished agent. +2. The fingerprint is stored as deploy-owned Worker metadata next to the required + release. Its first rollout attempts to recompute the prior fingerprint from the + deployed commit SHA, treating the initial compatibility marker value as the legacy + baseline; an ambiguous state publishes a fresh release. +3. Trials have no task row. To persist failed trial placement before workspace creation, + `trials.placement_explanation_json` is required in addition to the existing task and + workspace columns. +4. Placement evidence is built only from allowlisted identifiers, enums, booleans, and + bounded numeric snapshots. Raw metrics JSON, agent versions, provider errors, + prompts, repository data, environment values, and secrets are never copied. +5. Only the selected node retains its real identifier. Rejected and + eligible-but-unselected candidates are persisted as stable `candidate-N` aliases so + trial and cross-project host IDs cannot leak through placement evidence. + +The normal `/do` task-file push to `main` is intentionally not used: pushing `main` can +trigger a production deployment, while this task explicitly prohibits every deployment. +The task file and all implementation changes stay on the SAM-provided output branch and +will be reviewed in one draft PR. + +## Research Findings + +1. `.github/workflows/deploy-reusable.yml:Resolve and Verify Deployment SHA` assigns the + deployment SHA to `agent_version` on every normal run and emits an empty value for + `skip_agent`; build/upload conditions are based only on `skip_agent`. This directly + causes unrelated release churn and can clear enforcement. Addressed by checklist A. +2. `scripts/deploy/sync-wrangler-config.ts:getApiWorkerVars` copies + `VM_AGENT_REQUIRED_VERSION` only when non-empty. The deployment already has a + fail-closed Cloudflare settings-read precedent in + `scripts/deploy/durable-object-migrations.ts`; the release resolver must similarly + avoid logging the settings payload because it contains plaintext bindings such as + `SETUP_TOKEN`. Addressed by checklist A and security tests. +3. `packages/vm-agent/**` contains the Go module/toolchain declaration, dependency lock, + source, and Makefile. Using `go-version-file: packages/vm-agent/go.mod` and hashing the + tracked package tree plus an explicit compatibility marker creates a historical, + deterministic input contract. Addressed by checklist A. +4. `isNodeAgentVersionCompatible` is the exact gate shared by preferred, warm, + capacity, trial, manual, readiness, and cleanup paths. It intentionally permits an + unset requirement for local/manual development. The implementation must keep this + function's semantics unchanged and ensure official deployments never emit an empty + requirement. Addressed by checklists A, B, and tests. +5. Reusable selection is duplicated between + `apps/api/src/durable-objects/task-runner/node-selection.ts`, + `apps/api/src/services/node-selector.ts`, the trial orchestrator, and manual workspace + creation. Several SQL filters discard candidates before any typed rejection can be + recorded, and TaskRunner verifies heartbeat only after choosing one candidate. + Addressed by checklist B with a central typed evaluator/selector. +6. `PlacementExplanation` exists in `packages/shared/src/types/resource.ts`, and D1 + already has task/workspace JSON columns from migration 0056, but no production path + writes them. Task responses expose only raw JSON and workspace responses omit it. + Addressed by checklists B and C. +7. TaskRunner must persist a provisioning decision immediately after reusable selection, + then append size-fallback/provider-attempt outcomes and copy the finalized explanation + into the workspace row. `failTask` must preserve a placement failure only when the + failure belongs to selection/provisioning/readiness, not overwrite a successful + placement when a later agent step fails. Addressed by checklist B. +8. Trials explicitly have no task row, and manual workspace creation creates its + conversation task only after selecting/creating the node. Trial persistence therefore + needs a D1 column; manual persistence can write the same explanation into the new + task/workspace rows. Addressed by checklist B and migration tests. +9. `get_workspace_info` currently proxies only VM-agent-local metadata. It can safely + enrich that result from the project-scoped D1 workspace row, while REST mappers can + expose safely parsed structured data. Addressed by checklist C. +10. `WorkspaceSidebar` is an existing compact workspace detail surface. A default- + collapsed placement section satisfies the UI objective without a new top-level page, + but requires unit/accessibility coverage and local Playwright audits at 375px and + 1280px. Addressed by checklists C and D. +11. Prior rollout incidents show that artifact publication must precede Worker + enforcement and that busy incompatible VMs, active provisioning claims, and fresh + unversioned nodes must remain protected. Existing cleanup/readiness behavior is not + being redesigned. Addressed by checklist A and regression suite D. +12. Existing selection tests contain many source-string contracts alongside behavioral + tests. Refactoring must replace brittle contracts that encode duplication with + behavioral coverage of typed decisions and persisted vertical slices. Addressed by + checklist D. +13. The deterministic historical fingerprint is identical for production SHAs + `23e7adc23954d2b3a231b942edbb3195a6442301` and + `fc1e394217248c3bd004b2e6619cf2344eade7e3` + (`af2265ffb952310437913fbaae702cec303d5c1dbb2016bec2ee63c7d870609a`), proving + the resolver classifies that incident's controller-only deploy as unchanged. + +## Implementation Checklist + +### A. Deterministic VM-agent release resolution + +- [x] Add a versioned compatibility marker with a documented legacy baseline. +- [x] Add a tested release-resolution module that deterministically fingerprints the + tracked VM-agent package inputs at a Git ref and emits `build_agent`, + `required_version`, `fingerprint`, and a machine-readable reason. +- [x] Read only the allowlisted deployed Worker bindings needed for resolution; treat + missing Worker state as first install and fail closed on unreadable/invalid state. +- [x] Infer an absent first-rollout fingerprint from the prior required Git SHA when + possible; publish when equivalence cannot be proven. +- [x] Carry the prior required release and skip R2 build/upload when the fingerprint is + unchanged. +- [x] Publish and advance to the target deployment SHA when inputs or the explicit marker + changed. +- [x] Preserve the prior release for unchanged `skip_agent`; reject changed-input or + first-deploy `skip_agent`. +- [x] Use the release outputs for every Wrangler sync invocation and ensure official + deployed environments never receive an empty requirement. +- [x] Keep the separate Cloudflare Container image build/version boundary intact. + +### B. Typed placement evaluation and persistence + +- [x] Evolve the shared placement model to a versioned v2 shape with outcome, selection + path, request/limit snapshot, evaluated candidates, typed rejection reasons, + provisioning attempts, timestamps, and concise summary. +- [x] Add safe parsing for v2 and the legacy unversioned shape. +- [x] Centralize reusable-node evaluation for preferred, warm, capacity, trial, manual, + and compatibility-wrapper paths while retaining exact agent-version comparison. +- [x] Record deterministic typed reasons including agent mismatch, unhealthy, stale + heartbeat, wrong runtime, undersized VM, workspace limit, CPU/memory thresholds, + and lost warm claim. +- [x] Persist TaskRunner selection immediately, update provisioning/fallback attempts, + preserve selection/provisioning failures, and copy/finalize the explanation on + workspace creation. +- [x] Add a safe migration and schema field for trial placement, and persist trial reuse, + provisioning, and failure decisions. +- [x] Persist manual selected-node and provision-new decisions on its conversation task + and workspace. +- [x] Emit structured placement log events with only the allowlisted explanation. + +### C. API, MCP, and compact UI exposure + +- [x] Return parsed placement data from task and workspace mappers while retaining the + legacy raw task JSON field for compatibility. +- [x] Enrich `get_workspace_info` with a concise D1-backed placement summary/detail. +- [x] Add a default-collapsed placement section to the existing workspace sidebar, + including concise outcome text and typed rejection/attempt detail. +- [x] Update public configuration/architecture documentation, env references, and + `.claude/rules/54-vm-agent-rollout-compatibility.md` with release carry-forward + semantics and explicit compatibility bump guidance. + +### D. Verification + +- [x] Add release-resolution tests for unchanged inputs, source/toolchain/build-script + changes, compatibility-marker changes, first install, first-rollout inference, + changed/unchanged `skip_agent`, invalid metadata, and no empty enforcement. +- [x] Update workflow/sync safety tests for release outputs, step ordering, and + conditional R2 build/upload. +- [x] Add behavioral selector tests covering reuse, exact incompatibility on every path, + deterministic reasons/metrics, healthy-versus-better-incompatible ranking, and + concurrent warm claim loss. +- [x] Add TaskRunner/trial/manual vertical slices proving reused, provisioned, and failed + decisions persist and copy to workspaces. +- [x] Seed canary secrets in raw metrics, agent version/provider errors, and unrelated DB + fields; prove stored, REST, log, and MCP placement payloads exclude them. +- [x] Add mapper/MCP/UI tests and run local Playwright visual/accessibility audits with + normal, long, empty/legacy, many-rejection, error, and special-character scenarios + at mobile and desktop widths with no horizontal overflow. +- [x] Run focused suites, `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build`. +- [x] Run Cloudflare, environment, security, UI/UX, documentation, constitution, and + test-engineering specialist reviews; address all findings. +- [x] Run the mandatory task-completion validator after this final task/evidence update. +- [x] Push the output branch, open a draft PR explicitly stating “not deployed to + staging” and “do not merge”, wait for applicable CI, and leave it open/unmerged. + +## Local Validation Evidence + +- `pnpm lint` — passed with only the repository's existing warnings. +- `pnpm typecheck` — passed; the WWW package reported its expected baseline-only note. +- `pnpm build` — passed with the existing CSS minification and chunk-size warnings. +- `pnpm test` — passed all 21 Turbo tasks; Web 255 files / 3,069 tests passed. The + final post-review API suite separately passed 539 files / 7,114 tests. +- `pnpm format:check` — passed the repository format ratchet. +- Release-resolution and workflow focused suites — 94/94 passed, including rejection + of option/revision-shaped Git refs before either build-input subprocess executes. +- Final manual-placement review suite — 41/41 API tests; shared placement parser — + 9/9; broader placement/persistence/API/MCP focused suites — 200+; UI placement unit + tests — 3/3. +- TaskRunner placement vertical slice — 3/3 passed against real in-memory SQLite, + covering reused and provision-new selection through workspace persistence/dispatch, + write-before-advance ordering, and a typed terminal provisioning timeout. +- Local Playwright UI audit — 16/16 dark/light mobile/desktop scenarios passed with + explicit horizontal-overflow and accessibility checks. No environment was deployed. +- Draft PR #1808 is open and unmerged with the required “not deployed to staging” and + “do not merge” notices. All applicable CI passed on implementation commit `7798a3fef`, + including Playwright Visual Tests, SonarCloud, the full test/build/lint/typecheck jobs, + deploy-script validation, smoke tests, benchmarks, and specialist evidence checks. + +## Specialist Review Evidence + +| Reviewer | Status | Outcome | +| ------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| Cloudflare specialist | PASS | Release metadata, Worker settings, migration ordering, and R2 publication behavior verified after fixes through `935379670`. | +| Environment validator | PASS | Generated deployment-owned bindings and documentation remain consistent; no new manual secret prerequisite. | +| Security auditor | ADDRESSED / PASS | Tenant-safe placement passed; strict SHA/argv/option guards justify the documented Sonar sink suppression. | +| UI/UX specialist | PASS | Default-collapsed sidebar, accessibility, responsive behavior, and 16 local Playwright scenarios passed. | +| Documentation validator | ADDRESSED / PASS | Release ordering/fail-closed semantics, MCP shape, OpenAPI, parser contract, and public docs passed at `abcc1ddac`. | +| Constitution validator | ADDRESSED / PASS | Removed hidden ceilings/truncation/weights and duplicated defaults; Principle XI review passed at `839267430`. | +| Test engineer | ADDRESSED / PASS | Real route/SQLite reuse and provision slices plus exact failure/timeout persistence passed at `26ac0c615`. | +| Task-completion validator | PASS | Final branch/task review passed at `7798a3fef`; every implementation, test, privacy, and documentation criterion is covered. | + +## Acceptance Criteria + +- An unrelated deployment with unchanged build inputs carries forward the last actually + published required version and does not build/upload reusable-VM binaries. +- Any tracked VM-agent input or compatibility-marker change publishes and advances the + exact required version; `skip_agent` cannot bypass that transition. +- First-deploy or unreadable/invalid prior state never empties compatibility enforcement. +- The compatibility predicate remains exact when a deployed requirement exists; busy + incompatible nodes retain work but receive no new work. +- A healthy matching medium/hel1 node with 2/5 workspaces and low load is selected, and + both task/workspace placement records say it was reused. +- Every preferred, warm, capacity, trial, and manual reusable-node evaluation records + typed rejection reasons and allowlisted limit/metric snapshots. +- Provisioning and failure paths retain a versioned explanation, including warm-claim + loss and size-fallback attempts where applicable. +- REST task/workspace responses, `get_workspace_info`, and the compact workspace detail + UI expose the placement explanation without credentials, env values, prompts, + repositories, process data, raw provider errors, raw agent versions, or seeded canary + secrets. +- Local quality gates, applicable specialist reviews, and CI complete without deploying + to any environment. The final PR remains draft and unmerged for human review. + +## References + +- SAM Idea `01KZRMP0J6ZEXT5KEGCSKFSPMH` +- `.claude/rules/54-vm-agent-rollout-compatibility.md` +- `.claude/rules/07-env-and-urls.md` +- `.claude/rules/10-e2e-verification.md` +- `.claude/rules/17-ui-visual-testing.md` +- `tasks/archive/2026-08-06-fix-node-reaping-orphan-reconciliation.md` +- `tasks/archive/2026-08-07-fix-provisioning-node-cleanup-race.md` +- `.github/workflows/deploy-reusable.yml` +- `scripts/deploy/sync-wrangler-config.ts` +- `apps/api/src/durable-objects/task-runner/node-selection.ts` +- `apps/api/src/services/node-selector.ts` +- `packages/shared/src/types/resource.ts` + +## Execution Constraints + +- Do not deploy to staging or any other environment. +- Do not merge. +- Do not weaken or bypass exact VM-agent compatibility for unknown/genuinely + incompatible agents. +- Validate locally, through static analysis, CI, and specialist reviews. +- Finish with the implementation branch pushed and a clearly marked draft PR open.