Skip to content

test(gcp): keep canceled HTTPS responses incomplete - #2484

Merged
steipete merged 1 commit into
mainfrom
codex/gcp-readiness-fixture-n854
Sep 22, 2026
Merged

steipete merged 1 commit into
mainfrom
codex/gcp-readiness-fixture-n854

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Summary

Keep the two GCP cancellation-test handlers blocked after observing request cancellation, until their existing cleanup release closes. Returning from a handler without writing a response can produce an implicit empty HTTP 200 while the client is still handling cancellation. The Google REST client may then attempt to decode that empty success response.

Main CI failed the real HTTPS readiness cancellation case with a protobuf syntax error instead of the expected cancellation cause and exit code: https://github.com/openclaw/crabbox/actions/runs/35721672744 . That failure is consistent with this fixture race; its exact transport ordering was not captured, and 100 repetitions of unchanged main passed locally. This is a source-supported fixture correction, not a claimed deterministic local reproduction of the CI failure.

No production code, timeout budgets, error classification, or assertions change. Existing teardown releases the handlers before closing the test servers. The adjacent direct SDK cancellation fixture uses the same pattern and gets the same correction.

Verification

  • Both fixed cancellation tests passed 100 repetitions under the race detector using native local HTTPS and the real Google SDK. Synthetic credentials only; no cloud resources.
  • All readiness tests passed under the race detector, including the actual two-minute owned-budget case:
caller cancellation: elapsed=34.091458ms, identity=context canceled,
  code=7, diagnostic="caller stopped GCP readiness", classification=canceled/canceled
owned budget: elapsed=2m0.001166542s, identity=context deadline exceeded,
  code=1, diagnostic="timeout waiting for gcp public ip on readiness-instance",
  classification=timed-out/timeout
  • Existing completed-response precedence and error-cause assertions remain unchanged and pass.
  • Independent complete-introduced-scope Codex review through P2 found no actionable findings; a separate source-aware audit checked teardown ordering.
  • Test-only correction: no user-visible behavior or changelog change.

@clawsweeper

clawsweeper Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

steipete added a commit that referenced this pull request Sep 22, 2026
Bound Cloudflare Access signing-key fetch and body consumption to 15 seconds, abort stalled transport, and publish only the winning cache result. Preserve optional shared/admin bearer identity fallback, key-rotation refresh allowance, failure caching, and concurrent-load coalescing. Maintain documentation and changelog.

Maintainer disposition: complete introduced-scope Codex review through P2 is clean. Native loopback HTTP with a real clock recorded the partial response closing after 15 seconds before cleanup while bearer authentication retained its fallback identity. All 3,410 Worker tests, both typechecks/builds, lint and formatting passed. Exact-head CI including Go core and Release Check succeeded; no unresolved review threads. No live Cloudflare tenant claim.

An earlier main CI run hit the separate GCP cancellation-fixture race tracked in #2484; this candidate's complete CI passed without retry.

#2483
@clawsweeper clawsweeper Bot added P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 22, 2026
@clawsweeper

clawsweeper Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 22, 2026, 8:02 AM ET / 12:02 UTC.

ClawSweeper review

What this changes

The PR keeps two GCP cancellation-test HTTPS handlers blocked until cleanup, preventing them from completing empty responses during cancellation assertions.

Merge readiness

Ready for maintainer review

The fixture correction remains useful and is absent from fetched main. No actionable defect was found; cleanup safely releases both handlers, and the supplied real-HTTPS evidence supports the change.

Priority: P3
Reviewed head: 42bbe6746fd32e0390e7f753af4f2adf28942625

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A narrow, well-supported fixture correction with relevant real-transport evidence and no blocking findings.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.
Evidence reviewed 7 items Introduced scope: The pinned merge-base-to-head diff adds only one comment and one release-channel wait in each of two test handlers; production code and assertions are unchanged.
Cleanup ordering: The SDK test defers cancellation and release closure before client/server shutdown. The readiness fixture also closes release before server.Close, including assertion-failure exits, so the new waits have an existing teardown path.
Current main still lacks the correction: Comparing fetched main with the reviewed head for both affected files yields exactly the two proposed handler changes.
Findings None None.
Security None None.

How this fits together

Crabbox’s GCP readiness checks use the Google SDK to observe a provisioned machine before SSH setup. These fixtures exercise cancellation through real HTTPS transport and verify the resulting errors and exit classifications.

flowchart LR
  A[Readiness request] --> B[Google SDK client]
  B --> C[Local HTTPS fixture]
  D[Caller cancellation or timeout] --> B
  C --> E[Observe request cancellation]
  E --> F[Wait for cleanup release]
  B --> G[Assert cause and exit classification]
  G --> F
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

None.

Technical review

Best possible solution:

Keep canceled fixture responses incomplete until teardown while preserving production cancellation semantics and completed-response precedence.

Do we have a high-confidence way to reproduce the issue?

No deterministic reproduction of the original CI race is established: the author reports 100 unchanged-main repetitions passed. Source supports the unwanted handler-completion possibility, and supplied after-fix output verifies cancellation behavior.

Is this the best way to solve the issue?

Yes. Waiting on the existing cleanup channel narrowly prevents premature response completion without changing error classification, timing budgets, or assertions.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against be7cf9d50a61.

Labels

Label changes:

  • add P3: This is a focused test-fixture reliability correction with no production behavior change.
  • add proof: sufficient: Contributor real behavior proof is sufficient. Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.

Label justifications:

  • P3: This is a focused test-fixture reliability correction with no production behavior change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.
  • proof: sufficient: Contributor real behavior proof is sufficient. Supplied after-fix output exercises the production GCP readiness owner and real Google SDK over local TLS, showing explicit caller cancellation and actual budget expiry with preserved causes and classifications; source confirms cleanup releases both modified handlers. No stored-data contract changes.

Evidence

What I checked:

  • Introduced scope: The pinned merge-base-to-head diff adds only one comment and one release-channel wait in each of two test handlers; production code and assertions are unchanged. (internal/cli/gcp_test.go:196, 42bbe6746fd3)
  • Cleanup ordering: The SDK test defers cancellation and release closure before client/server shutdown. The readiness fixture also closes release before server.Close, including assertion-failure exits, so the new waits have an existing teardown path. (internal/providers/gcp/backend_doctor_test.go:446, 42bbe6746fd3)
  • Current main still lacks the correction: Comparing fetched main with the reviewed head for both affected files yields exactly the two proposed handler changes. (internal/providers/gcp/backend_doctor_test.go:439, be7cf9d50a61)
  • Captured production-path proof: The supplied PR body, captured under sourceRevision 2f21d409bb36093cd2e76d01bf685d081dc118c17e155f875a37e88a702fdcc7, reports 100 race-detector repetitions using native local HTTPS and the real Google SDK. Copied output records caller cancellation in 34.091458ms with exit code 7 and canceled/canceled classification, and the actual two-minute budget with exit code 1 and timed-out/timeout classification. Source confirms the readiness fixture invokes NewGCPClient, GetServer, and waitForServerIP with synthetic credentials and a real HTTP transport. The body explicitly does not claim deterministic reproduction of the original CI race. (internal/providers/gcp/backend_doctor_test.go:401, 42bbe6746fd3)
  • Related merged work: GitHub verifies fix(gcp): preserve readiness cancellation causes and deadlines #2467 merged as 649c84c. That work preserves production cancellation causes and supplies the readiness HTTPS fixture; it does not include this fixture correction. (649c84c9438d)
  • Area history and routing: Main-branch history identifies Peter Steinberger on the readiness-cause change and earlier readiness/transport work. The GitHub commit endpoint maps the readiness-cause commit author to steipete. Local blame encountered an unavailable historical blob; routing is supported by the available commit history and GitHub patch instead. (internal/providers/gcp/backend_doctor_test.go:401, 649c84c9438d)

Likely related people:

  • unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete merged commit 823588b into main Sep 22, 2026
34 checks passed
@steipete
steipete deleted the codex/gcp-readiness-fixture-n854 branch September 22, 2026 12:22
@steipete

Copy link
Copy Markdown
Contributor Author

Landed as 823588b through the normal protected squash path. The resulting tree matches the expected integration with current main. Both canceled HTTPS handlers now wait for their existing cleanup release instead of completing an implicit empty response; production code and assertions are unchanged.

Both cancellation tests passed ten further race-detector repetitions on the actual merged checkout. Main CI, including the real readiness-budget test in Go core, succeeded at https://github.com/openclaw/crabbox/actions/runs/35726820631. The earlier CI error remains a source-supported fixture-race diagnosis, not a claim that its exact transport ordering was reproduced locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant