refactor: centralize provider readiness termination policy

[steipete]

Summary

Vast and RunPod now share ordinary readiness-budget and termination policy through PollReadiness, rather than maintaining separate versions of the same cancellation logic.

The shared helper owns the child deadline, interrupted-read classification, completed-observation precedence, and cause-preserving termination wrapper. Adapters retain endpoint/state predicates, native API-error recognition, retry/backoff decisions, and public diagnostics. RunPod's jittered schedule and Vast's terminal-state handling remain local. Existing PollReady callers are deliberately unchanged pending their individual contract audit.

Verification

• Full shared, Vast, and RunPod race suites and vet pass after integration with both landed fixes.
• The new shared matrix covers pre-cancellation, independent client deadlines, context error/cause variants, read/sleep expiry, completed responses and terminal decisions, and retention/clearing of the last completed transient diagnostic.
• Existing real HTTP deadline tests remain in both providers.
• Complete independent Codex review found no actionable issues through P2.
• Fresh native Vast and RunPod acquisition, actual SSH, and normal-release smokes passed on final source 38db1ee5154a38af9f4f4125f772be05a140f4e4. Binary SHA-256: dcd36be2ae2ef963ae04185aa0954ca12b06987dee2eae1b95c56d722990efd8.
• Each provider's independent inventory and exact-ID read confirmed absence after successful normal cleanup; local claims were absent and private runtimes removed.
• Current-head CI passed.

The separately prepared OVH consumer remains outside this PR and retains its own native-proof requirement. This is an internal consolidation with no intended user-visible change, so no new changelog entry is added.

Captured final-source runtime results

The following is a field-selected copy of the native harness output. Credentials, resource identities, endpoints, local paths, and process IDs are omitted. Native runs exercise acquisition/SSH and normal release; deadline cancellation is covered separately through the production HTTP clients against loopback servers.

{
  "vast": {
    "source": "38db1ee5154a38af9f4f4125f772be05a140f4e4",
    "binarySha256": "dcd36be2ae2ef963ae04185aa0954ca12b06987dee2eae1b95c56d722990efd8",
    "status": "passed",
    "nativeReadiness": {
      "ready": true,
      "state": "ready",
      "actualSSH": true
    },
    "cleanup": {
      "ownInstances": 0,
      "instanceMissingById": true,
      "localClaims": 0
    },
    "events": [
      {
        "phase": "doctor",
        "exit": 0,
        "seconds": 2.23
      },
      {
        "phase": "warmup",
        "exit": 0,
        "seconds": 42.597
      },
      {
        "phase": "ready",
        "exit": 0,
        "seconds": 2.175
      },
      {
        "phase": "destroy_1",
        "exit": 0,
        "seconds": 1.453
      }
    ]
  },
  "runpod": {
    "source": "38db1ee5154a38af9f4f4125f772be05a140f4e4",
    "binarySha256": "dcd36be2ae2ef963ae04185aa0954ca12b06987dee2eae1b95c56d722990efd8",
    "status": "passed",
    "nativeReadiness": {
      "ready": true,
      "state": "ready",
      "actualSSH": true
    },
    "cleanup": {
      "taskPodsRemaining": 0,
      "exactAbsent": true,
      "normalReleaseSucceeded": true,
      "localClaimAbsent": true
    },
    "events": [
      {
        "phase": "doctor",
        "exit": 0,
        "seconds": 2.693
      },
      {
        "phase": "warmup",
        "exit": 0,
        "seconds": 34.469
      },
      {
        "phase": "ready",
        "exit": 0,
        "seconds": 1.379
      },
      {
        "phase": "release",
        "exit": 0,
        "seconds": 1.771
      }
    ]
  },
  "privateRuntimesRemoved": true
}

Production-client deadline test output from the full race run:

{"Action": "pass", "Package": "github.com/openclaw/crabbox/internal/providers/runpod", "Test": "TestRunpodClientRejectsBareHTTPURL", "Elapsed": 0}
{"Action": "pass", "Package": "github.com/openclaw/crabbox/internal/providers/runpod", "Test": "TestRunpodClientAllowsLoopbackHTTPURL", "Elapsed": 0}
{"Action": "pass", "Package": "github.com/openclaw/crabbox/internal/providers/runpod", "Test": "TestRunpodSSHWaitBoundsNativeHTTPClient", "Elapsed": 0.05}
{"Action": "pass", "Package": "github.com/openclaw/crabbox/internal/providers/vast", "Test": "TestReadinessDeadlineCancelsNativeHTTPClient", "Elapsed": 0.08}

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed September 21, 2026, 2:01 PM ET / 18:01 UTC.

ClawSweeper review

What this changes

The PR shares Vast and RunPod readiness deadlines and cancellation handling while retaining each provider’s readiness checks, polling schedule, and public diagnostics.

Merge readiness

⛔ Blocked before merge - 2 items remain

This remains a useful, bounded consolidation absent from current main. No actionable correctness defect was found; the reported native smokes support the approach, but inspectable runtime evidence is still needed before merge.

Priority: P3
Reviewed head: 38db1ee5154a38af9f4f4125f772be05a140f4e4

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The focused extraction and regression matrix are solid; runtime claims still need inspectable supporting evidence.
Proof confidence	🦐 gold shrimp (3/6)	Needs stronger real behavior proof before merge: Needs real behavior proof before merge: the body reports native acquisition, SSH, and cleanup on an earlier commit whose changed production files match this head, but provides no inspectable output showing those results or the shared termination path. Preserve that source attribution and attach redacted terminal screenshots, logs, or linked runtime evidence; production-owner traces using the real HTTP clients can support cancellation coverage. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: Needs real behavior proof before merge: the body reports native acquisition, SSH, and cleanup on an earlier commit whose changed production files match this head, but provides no inspectable output showing those results or the shared termination path. Preserve that source attribution and attach redacted terminal screenshots, logs, or linked runtime evidence; production-owner traces using the real HTTP clients can support cancellation coverage. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	7 items	Policy and repository identity: Read the complete root AGENTS.md and verified the origin repository. No ancestor-scoped policy exists under the changed directories, and no maintainer-notes directory was present. Applied provider-neutral architecture, testing, and contributor changelog guidance. Introduced change and merge scope: The merge-base-to-head patch changes five files. The verified test merge has the pinned main and original head as its two parents and changes those same five files; unrelated main-branch work is preserved. Behavior and solution fit: PollReadiness extracts the existing adapter deadline, interrupted-read classification, completed-result precedence, and display-safe termination wrapper. PollReady has a simpler contract without these adapter callbacks, so retaining its existing consumers avoids changing their contracts.
Findings	None	None.
Security	None	None.

How this fits together

During machine acquisition, Crabbox polls provider APIs until an SSH endpoint becomes ready. The shared readiness helper returns either the ready resource or an error that the adapter uses to stop acquisition and invoke its existing cleanup path.

flowchart TD
 A[Machine acquisition] --> B[Provider readiness adapter]
 B --> C[Shared deadline and polling]
 C --> D[Provider API observation]
 D --> E{Ready or stopped?}
 E -->|Ready| F[SSH preparation]
 E -->|Error or cancellation| G[Acquisition cleanup]

Loading

Before merge

• Add real behavior proof - Needs stronger real behavior proof before merge: Needs real behavior proof before merge: the body reports native acquisition, SSH, and cleanup on an earlier commit whose changed production files match this head, but provides no inspectable output showing those results or the shared termination path. Preserve that source attribution and attach redacted terminal screenshots, logs, or linked runtime evidence; production-owner traces using the real HTTP clients can support cancellation coverage. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Complete next step (P2) - Add inspectable, redacted runtime evidence to the PR body, preserving the source commit for each result. Updating the body should trigger re-review; otherwise ask a maintainer to comment @clawsweeper re-review.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	Production +89/-63 (net +26); tests +166/-0	The modest production growth extracts duplicated policy into a reusable helper with focused termination coverage.

Technical review

Best possible solution:

Keep one shared readiness termination policy with provider-specific predicates and diagnostics, supported by inspectable production-path evidence.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR consolidates existing behavior rather than reporting a new failure; source comparison and focused tests define the behavior to preserve.

Is this the best way to solve the issue?

Yes: extracting the matching adapter policies is a narrow solution, while leaving existing PollReady consumers unchanged avoids imposing a different cancellation contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 6aa35cea5840.

Labels

Label changes:

• add P3: This is a bounded internal consolidation with no intended user-visible behavior change.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Needs real behavior proof before merge: the body reports native acquisition, SSH, and cleanup on an earlier commit whose changed production files match this head, but provides no inspectable output showing those results or the shared termination path. Preserve that source attribution and attach redacted terminal screenshots, logs, or linked runtime evidence; production-owner traces using the real HTTP clients can support cancellation coverage. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

• P3: This is a bounded internal consolidation with no intended user-visible behavior change.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Needs real behavior proof before merge: the body reports native acquisition, SSH, and cleanup on an earlier commit whose changed production files match this head, but provides no inspectable output showing those results or the shared termination path. Preserve that source attribution and attach redacted terminal screenshots, logs, or linked runtime evidence; production-owner traces using the real HTTP clients can support cancellation coverage. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

• Policy and repository identity: Read the complete root AGENTS.md and verified the origin repository. No ancestor-scoped policy exists under the changed directories, and no maintainer-notes directory was present. Applied provider-neutral architecture, testing, and contributor changelog guidance. (AGENTS.md:25, 38db1ee5154a)
• Introduced change and merge scope: The merge-base-to-head patch changes five files. The verified test merge has the pinned main and original head as its two parents and changes those same five files; unrelated main-branch work is preserved. (38db1ee5154a)
• Behavior and solution fit: PollReadiness extracts the existing adapter deadline, interrupted-read classification, completed-result precedence, and display-safe termination wrapper. PollReady has a simpler contract without these adapter callbacks, so retaining its existing consumers avoids changing their contracts. (internal/providers/shared/observer.go:67, 38db1ee5154a)
• Current main and release still lack the consolidation: Inspected the shared observer on pinned main and latest release v0.63.0; both contain PollReady and Poll but lack PollReadiness. GitHub confirms fix(vast): bound SSH readiness observations and waits #2448 and fix(runpod): preserve SSH readiness termination causes #2449 merged the underlying provider fixes, not this shared extraction. (internal/providers/shared/observer.go:48, 6aa35cea5840)
• Focused regression coverage: The new shared matrix covers cancellation before and during reads, budget expiry, completed results, independent client deadlines, terminal observations, and retained retry diagnostics. Existing provider tests exercise the production HTTP clients against blocked loopback requests. Tests were inspected, not executed, under the read-only review constraint. (internal/providers/shared/observer_test.go:84, 38db1ee5154a)
• Native evidence identity and coverage: The supplied body, identified by sourceRevision d951d790c1ae439c9d99190ac4bafac184f432437c7f2472b2f031517db6e65d, reports successful Vast and RunPod acquisition, SSH, and cleanup on 365daed. All three changed production files are identical between that commit and the reviewed head. This is relevant earlier-source evidence, but the body supplies no captured runtime output or linked artifact and explicitly leaves final-source qualification pending. A live read returned the same body and only bot status comments. (38db1ee5154a)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Attach source-attributed native smoke output and relevant production-client deadline or cancellation results, redacting credentials, IP addresses, and private endpoints.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

@clawsweeper re-review

The body now includes captured results on final source 38db1ee: fresh Vast and RunPod acquisition, actual SSH, successful normal cleanup, independent inventory/exact-ID absence, and production HTTP-client deadline test output. Sensitive identities and endpoints are omitted. Current-head CI passed.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Waiting
• Detail: A newer exact-review queue owner superseded this run. This run completed as a no-op.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/35638615404
• Updated: 2026-09-21T18:29:33.458Z