Skip to content

fix(gcp): preserve readiness cancellation causes and deadlines - #2467

Merged
steipete merged 4 commits into
mainfrom
codex/gcp-readiness-causes-n832
Sep 21, 2026
Merged

steipete merged 4 commits into
mainfrom
codex/gcp-readiness-causes-n832

Conversation

@steipete

@steipete steipete commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Move GCP's ordinary public-IP wait onto shared.PollReadiness so interrupted reads retain the original caller cause and canonical context identity, and owned budget expiry retains deadline classification. The old helper dropped the custom cause when the SDK returned ctx.Err() and returned a plain timeout error without deadline identity.

Preserve the two-minute IP-observation budget, five-second interval, nonempty-IP-only readiness predicate, immediate API-error handling, exact timeout message, and normal CLI exit code 1. Do not add trimming, running-state requirements, retries, or a provider-independent exit code 5. Completed ready responses and typed API errors retain precedence at coincident cancellation. Caller diagnostics retain the existing pre-cancel/wait cause text and typed exit code; interrupted reads now behave consistently with those paths.

The response predicate recognizes *googleapi.Error through errors.As: the real REST Compute client returns completed HTTP errors through GAX wrappers that preserve that cause. Transport failures and a later SDK retry-context stop are not classified as completed responses merely because an earlier attempt received a response.

No creation, rollback, claim, SSH, or heartbeat behavior is changed. Provider docs and the maintainer-added Unreleased entry explain the error/classification correction. Three other adapters still use the old helper; this PR does not redirect them wholesale.

Verification

  • Five new baseline cases fail: lost deadline identity for blocked/pending timeout and lost custom/canonical cancellation identity before a read, during a read, and during a wait.
  • Extended existing tests verify typed caller exit-code preservation, final run classification, the original timeout text/code, completed API-response precedence, late successful responses, and the unchanged untrimmed-IP/no-status-gate predicate. Existing independent client-deadline and interval controls remain.
  • Full GCP/shared race suites and vet pass before and after main integration.
  • Complete independent Codex reviews found no actionable issues through P2. A separate source audit verified actual SDK response-error provenance and the integration boundary.

Fresh native lifecycle proof

A newly built, credential-free binary at source bfdf326e5c847a880f4a76e26b90a303da5b2e8c completed a real one-instance GCP lifecycle in the previously approved test project. It used an e2-micro, 30 GB boot disk, 30-minute TTL, a unique task tag, restricted SSH ingress, isolated user/SDK state, and no guest service account. Provider API calls and SSH were real.

Acquire, readiness, remote command execution, explicit and omitted heartbeat policy, native TTL capping, and read-only status checks passed. Warmup took 189.394 seconds including creation and SSH/bootstrap; the two-minute IP-discovery budget is not a limit for that whole operation. Normal CLI stop succeeded. The task's managed firewall was then removed, and independent checks found zero instances, disks, firewalls, local claims, or private keys. Temporary credentials, Cloud SDK caches, runtime directories, and the owned credential window were removed.

Field-selected captured evidence, omitting project/account/resource identities and addresses:

{
  "nativeSource": "bfdf326e5c847a880f4a76e26b90a303da5b2e8c",
  "binarySHA256": "23ff9bc8117f689817d2b735614b5053c47c0e28ced8fd27aeb08eb3028a7831",
  "machineType": "e2-micro",
  "rootGB": 30,
  "ttl": "30m",
  "acquireExit": 0,
  "readyExit": 0,
  "remoteCommandMarkerVerified": true,
  "ready": true,
  "heartbeatExplicitIdle": "1h30m0s",
  "heartbeatOmittedIdle": "1h30m0s",
  "nativeTTLClampVerified": true,
  "statusDidNotMutateNativeLabels": true,
  "normalStopSucceeded": true,
  "cleanup": {
    "instances": 0,
    "firewalls": 0,
    "disks": 0,
    "claims": 0,
    "privateKeys": 0
  }
}

This healthy native run verifies integration; deterministic tests establish cancellation/deadline behavior that a successful lifecycle cannot prove.

Source attribution after integration

Current head 23414b9b850f4319d83db5d1c929df517d9c83b2 integrates main 574d3d2808a29f3d1281202283a8ca150cb0cf80. Relative to the native-tested source, the only changed files are the unrelated Scaleway local SDK-error implementation, its tests/docs, the changelog, and the additional GCP test file. GCP production code, core, shared polling, command/build inputs, manifests, and initialization code are unchanged. The complete current introduced diff is still only the four GCP code/test/docs/changelog files; integration tests and review pass.

The rebuilt integrated binary may differ. Native execution remains attributed to the actual source above, not relabeled as a new-head run. No new account permissions, release actions, production deployment, or unrelated qualification-hold changes are included.

The latest commit adds only the real-HTTPS tests below. No production executable input changed after the integrated source review or the source-attributed native run.

Real HTTPS proof of the changed failure paths

The added TestWaitForServerIPRealHTTPS calls the production core.NewGCPClient, its actual Google Compute REST SDK, GetServer, and the changed waitForServerIP together. It uses synthetic ADC and a local TLS-verified HTTPS fixture; a concrete transport permits only the two expected Google destinations and dials only the fixture. The test process has no live credentials and runs under a loopback-only network sandbox. This is real HTTP/SDK proof, not a native Google outage or a new cloud lifecycle claim.

Caller cancellation occurs only after the HTTPS server observes the request. The owned-budget case waits the real two minutes with an uncanceled three-minute parent. Both verify the server observes request cancellation, the diagnostic, exit code, cause identity, and final run classification. The same test against the original backend fails both cases. Full current GCP/shared race suites and vet pass; complete independent Codex review through P2 is clean.

Captured after-fix output, with only fixed synthetic names:

real HTTPS through NewGCPClient -> GetServer -> waitForServerIP:
ownedBudget=false elapsed=23.286708ms identity=context canceled code=7
 diagnostic="caller stopped GCP readiness" classification=canceled/canceled
ownedBudget=true elapsed=2m0.002364208s identity=context deadline exceeded code=1
 diagnostic="timeout waiting for gcp public ip on readiness-instance" classification=timed-out/timeout
PASS TestWaitForServerIPRealHTTPS (120.04s)

The baseline cancellation retained a URL-wrapped custom error but lost the canonical canceled identity and consistent public diagnostic. The baseline owned timeout preserved text/code but lost deadline identity. Neither baseline result is counted as passing proof. The actual two-minute case runs normally and skips only under Go's explicit short-test mode.

@clawsweeper

clawsweeper Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 21, 2026
@clawsweeper

clawsweeper Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 21, 2026, 7:08 PM ET / 23:08 UTC (Revision 2).

ClawSweeper review

What this changes

The PR preserves cancellation causes and deadline classification during GCP public-IP discovery, with regression coverage and documentation.

Merge readiness

Ready for maintainer review

The fix remains necessary on main. The new production-client HTTPS evidence resolves the previous proof blocker, and no actionable correctness or security findings remain.

Priority: P2
Reviewed head: 23414b9b850f4319d83db5d1c929df517d9c83b2

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused implementation with convincing production-transport proof and no remaining actionable findings.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.
Evidence reviewed 8 items Introduced change and preserved behavior: The pinned delta changes four files. The adapter adopts the existing shared readiness helper while retaining the two-minute budget, five-second interval, nonempty-IP predicate, immediate API errors, and timeout diagnostic.
Current main still needs the correction: Main still calls PollReady, which returns a plain timeout diagnostic for owned expiry and passes interrupted fetch errors through without the shared termination wrapper.
Release behavior: The supplied latest release, v0.63.0, also uses PollReady for GCP public-IP discovery; it does not contain this correction.
Findings None None.
Security None None.

How this fits together

Crabbox’s GCP adapter polls a newly created virtual machine for its public IP before starting SSH readiness checks. Polling failures feed acquisition cleanup and the CLI’s error classification.

flowchart TD
  A[New GCP virtual machine] --> B[Public IP discovery]
  C[Caller cancellation or time budget] --> B
  B --> D[Google Compute REST client]
  D --> E{Observation result}
  E -->|Public IP available| F[SSH readiness]
  E -->|Error or interrupted request| G[Preserved error and run classification]
  G --> H[Acquisition cleanup]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test delta Production +20/-4; tests +207/-0 The small adapter change reuses existing shared policy, with most growth devoted to regression and real-transport coverage.

Technical review

Best possible solution:

Use the shared termination wrapper while keeping GCP’s existing readiness, response-precedence, and diagnostic policies in its adapter.

Do we have a high-confidence way to reproduce the issue?

Yes, source establishes the failure path: main’s PollReady loses deadline identity on owned timeout and does not normalize interrupted-read cancellation. The contributor supplies failing-baseline and passing-after-fix evidence; this review did not execute tests.

Is this the best way to solve the issue?

Yes. Reusing the existing shared readiness policy avoids a second cancellation implementation while preserving GCP-specific behavior and public diagnostics.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 574d3d2808a2.

Labels

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This is a bounded GCP cancellation and timeout-classification correction without evidence of a broader outage.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.
  • proof: sufficient: Contributor real behavior proof is sufficient. The captured after-fix HTTPS run exercises the production GCP client and changed readiness owner, proving caller-cancellation identity and actual two-minute deadline classification with server-observed request cancellation. This resolves the previous proof gap; the separately attributed native lifecycle remains supporting integration evidence.

Evidence

What I checked:

  • Introduced change and preserved behavior: The pinned delta changes four files. The adapter adopts the existing shared readiness helper while retaining the two-minute budget, five-second interval, nonempty-IP predicate, immediate API errors, and timeout diagnostic. (internal/providers/gcp/backend.go:118, 23414b9b850f)
  • Current main still needs the correction: Main still calls PollReady, which returns a plain timeout diagnostic for owned expiry and passes interrupted fetch errors through without the shared termination wrapper. (internal/providers/gcp/backend.go:116, 574d3d2808a2)
  • Release behavior: The supplied latest release, v0.63.0, also uses PollReady for GCP public-IP discovery; it does not contain this correction. (internal/providers/gcp/backend.go:116, 8a8b22d6fd4f)
  • Production-path failure proof: The captured PR body and fix(gcp): preserve readiness cancellation causes and deadlines #2467 (comment) report after-fix execution through NewGCPClient, the Google Compute REST SDK, GetServer, and waitForServerIP. The inspected harness uses real TLS HTTP transport, cancels only after server receipt, verifies server-observed cancellation, and checks cause identity, diagnostics, exit codes, and classification. Captured outcomes are canceled/canceled with code 7 and timed-out/timeout with code 1 after 2m0.002364208s. The same scenarios reportedly fail against the original backend. This directly satisfies the previous review’s requested coverage. (internal/providers/gcp/backend_doctor_test.go:401, 23414b9b850f)
  • Native proof retains its original source identity: The supplied native lifecycle evidence belongs to bfdf326 and demonstrates real acquisition, SSH execution, normal stop, and cleanup. Comparing that source with the reviewed head confirms only Scaleway implementation/tests/docs, the changelog, and the GCP test file changed; GCP production and shared polling code are unchanged. The native run is not attributed to the newer binary. (internal/providers/gcp/backend.go:118, bfdf326e5c84)
  • Existing shared owner and routing history: Merged refactor: centralize provider readiness termination policy #2456 supplies the shared readiness helper and explicitly leaves existing PollReady consumers for individual audits. GitHub commit metadata identifies steipete as its author, and its patch adds the current helper at line 65. Local deeper blame/history reads encountered unavailable objects; the relevant commit patch and merged PR were inspected through GitHub instead. (internal/providers/shared/observer.go:65, cc03ce0115b7)

Likely related people:

  • unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)
  • reviewed 2026-09-21T22:38:00.094Z sha c1d0c4e :: needs real behavior proof before merge. :: none

@steipete

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current head 23414b9 adds the requested real HTTPS/SDK proof through the production GCP client and changed readiness function. Both interrupted-read cancellation and the actual two-minute owned timeout pass, including cause identity, exact diagnostics, exit codes, classification, and server-observed cancellation. The same tests fail against the original backend. Full current GCP/shared race suites, vet, and independent review pass. Captured output and source attribution are in the updated PR body; native lifecycle proof remains attributed to its actual source.

@clawsweeper

clawsweeper Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

@clawsweeper

clawsweeper Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Sep 21, 2026
@steipete
steipete merged commit 649c84c into main Sep 21, 2026
35 checks passed
@steipete
steipete deleted the codex/gcp-readiness-causes-n832 branch September 21, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant