Coordinator flakiness family: transient 5xx / dropped writes on lease-create, run-history-finish, and census registration

[excelsier]

Symptom 1: lease-create 500 (two occurrences)

crabbox run (managed Azure lease creation) occasionally fails before any
box is usably created, with the coordinator's POST /v1/leases returning
HTTP 500.

• Occurrence 1: ref 5llfol70h8qko73rc38qsti0. Response attributed to a
  Durable Object storage reset. No box was created, no lease was left
  dangling, and our independent leak-alarm (which cross-checks Azure
  directly, not just the coordinator's own bookkeeping) reported clean
  for that window — so whatever happened inside the DO, it failed before
  committing any state that could leak.
• Occurrence 2: box cbx_bb7fb1dce94b, witnessed directly in one of our own
  gate runs: coordinator lease create returned uncertain result for cbx_bb7fb1dce94b; repeating token-bound POST: coordinator POST /v1/leases: http 500: error code: 1101, followed by abandoning uncertain coordinator create cbx_bb7fb1dce94b; recording durable cancellation. Same endpoint and HTTP status as occurrence 1. This one's
  client-side log surfaces Cloudflare error code 1101 rather than the
  "Durable Object storage reset" wording occurrence 1's response carried, so
  we can't independently confirm both share the identical proximate cause —
  but both are transient 500s on the same write path with no lasting side
  effect: this occurrence's client treated the create as uncertain and
  recorded a durable cancellation rather than leaving a dangling lease.

Neither occurrence left the underlying system in a bad state — in both
cases the failure was caught before anything a caller would need to clean
up got committed. Occurrence 2's "repeating token-bound POST" before giving
up shows there is already some retry happening client-side ahead of the
durable-cancellation fallback; we don't have visibility into whether the
coordinator itself also retries this failure shape before returning the
500, or what the client-side retry ceiling is before it gives up (see "What
we're asking" below — if that ceiling is low, raising it might be enough on
its own).

Symptom 2: run-history "finish" returning HTTP 500 (one occurrence, cross-verified)

On box cbx_aced84636bf1 (2026-08-27, ~14:30Z), a routine gate run's client
accepted the coordinator's release response, but the separate run-history
"finish" write failed:

warning: coordinator accepted release for cbx_aced84636bf1; remote
cleanup remains pending and local claim/SSH artifacts were preserved
run history finish failed … http 500

The run's own timing record shows leaseStopped: true — the lease itself
was released; only the run-history bookkeeping write failed. We
independently verified the box's Azure resources tore down cleanly within
~15 minutes of this (zero residue), so nothing leaked. Net effect: "release
succeeded, history write didn't" — the same shape as Symptom 1 (a
transient 500 on a coordinator write), just on the finish path instead of
lease-create, and benign in outcome only because the underlying operation
had already completed before the write that failed.

Symptom 3: coordinator census gaps — provider-side resources with no coordinator record

During routine reconciliation work cross-checking the coordinator's own
inventory against live Azure-side resource state, two resources turned up
with no coordinator record at all: run_3b5a7e618a22 and
cbx_2b400af9b810. Both were real, live provider-side artifacts the
coordinator's bookkeeping had zero visibility into — not stale, absent, as
if the registering write never landed or was lost outright. Observed the
same day as symptoms 1 and 2. We have not verified whether this shares a
root cause with the DO-storage-reset pattern above landing on a different
write path, or is a distinct defect — flagging it as observed, not asserting
a shared cause we haven't proven.

What we're asking

All three symptoms share a shape: a coordinator write (lease-create,
run-history-finish, or whatever write should have produced a census-visible
registration) transiently fails or silently drops state, without the
underlying operation actually being unsafe — in every case we could verify,
the real-world resource state ended up correct or was independently
recoverable. Durable Object storage resets are a known, transient,
Cloudflare-platform-level event — not a permanent failure, and not something
a caller can fix by changing what it sends. If these write paths don't
already retry internally on this failure signature before surfacing a 5xx
(or silently dropping the write, for Symptom 3) to the client, closing that
gap would remove a class of "gate reported something alarming for no reason
a human can act on" incidents for anyone relying on the managed coordinator
path. If retry already exists and we're seeing its exhausted-budget tail,
knowing the ceiling would help us decide whether to add our own outer retry
— today we recover via an external ~5-minute cooldown + relaunch (Symptom 1)
or by independently re-deriving true state (Symptoms 2-3), both of which
cost real wall-clock on every occurrence.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 29, 2026, 11:39 PM ET / August 30, 2026, 03:39 UTC.

Summary
Keep open: current main still leaves a same-generation, unacknowledged managed create unscheduled for recovery, while the report documents resulting untracked Azure resources and billing exposure. The merged Azure fragment cleanup improves cleanup once an exact lease record exists, but does not resolve the missing-record/reset path.

Reproducibility: yes. at source level: cancel an in-flight Azure create or simulate a same-generation Worker reset before the provider response, then allow the provider resource to appear. Current recovery explicitly declines that unsettled, same-generation record, but a full hosted Durable Object fault run was not executed in this read-only review.

Maintainer decision needed

Question	Recommendation
Which exact ownership evidence may survive a same-generation coordinator reset or cancellation and later authorize Azure cleanup?	Persist a durable recovery claim: Record the exact lease, provider scope, and immutable cleanup context before provider dispatch, then recover only resources that revalidate against that claim.

Why: The report needs automated recovery, but selecting a tag-only or age-only deletion path would violate the project’s explicit ownership boundary.

Next step
A maintainer must choose the durable ownership and cleanup-authority boundary before a repair can safely be scoped.

Review details

Best possible solution:

Persist or derive an exact, non-credential recovery claim for an uncertain Azure create, then reconcile it after cancellation/reset through the existing ownership-verified cleanup path and add fault-injection coverage for the delayed provider completion.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: cancel an in-flight Azure create or simulate a same-generation Worker reset before the provider response, then allow the provider resource to appear. Current recovery explicitly declines that unsettled, same-generation record, but a full hosted Durable Object fault run was not executed in this read-only review.

Is this the best way to solve the issue?

Unclear: durable uncertain-create recovery is the right outcome, but the exact persistent ownership evidence and deletion authority need maintainer approval before implementation.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• Automatically deleting Azure resources from provider tags alone would weaken the repository’s exact-ownership safety boundary.
• The safe trigger for recovery after a same-generation reset or cancellation is unresolved: current code intentionally avoids an age-only cutoff because provider creates have no shared upper bound.
• PR fix: clean interrupted Azure network fragments #1514 does not prove that a full VM created after a coordinator reset can be discovered and cleaned when no matching coordinator record remains.

Codex review notes: model internal, reasoning high; reviewed against bf50dbca63b8.

Label changes

Label justifications:

• P1: The report describes repeated coordinator failures that can leave running metered Azure VMs without automatic cleanup ownership.
• impact:data-loss: Coordinator lease and inventory state can be absent for real provider-side resources.
• impact:other: The untracked provider resources can continue accruing cloud charges until an external sweep intervenes.

Evidence reviewed

What I checked:

• Same-generation uncertain creates are not recovered: Recovery returns without scheduling work when a provisioning request has not settled and the coordinator generation is unchanged. A Durable Object reset/cancelled response can leave exactly that durable state, so the reported path remains uncovered. (worker/src/fleet.ts:23510, bf50dbca63b8)
• Cancellation preserves unresolved creation state: The cancellation endpoint deliberately retains the original provisioning timestamp and marks the lease as potentially provisioned, but does not mark the provider request settled; this feeds the unscheduled recovery condition above. (worker/src/fleet.ts:3337, bf50dbca63b8)
• Azure sweep cannot delete a resource without a matching lease: The Azure orphan sweep may report tag-only candidates, but destructive release requires an exact ownership lease from coordinator storage. That protects against unsafe deletion but leaves a missing coordinator record unresolved. (worker/src/fleet.ts:16430, bf50dbca63b8)
• Run-history finish already has bounded retry: The CLI retries retryable finish failures up to three times with the identical payload and then instructs receipt-based recovery, so the reported finish symptom is partly mitigated but still has an exhausted-retry tail. (internal/cli/run_recorder.go:221, bf50dbca63b8)
• Merged Azure cleanup is narrower than this report: Merged PR fix: clean interrupted Azure network fragments #1514 added cleanup coverage for an exact public-IP prefix before NIC creation, but its merged commit does not establish recovery for a resource whose coordinator lease record is absent. (worker/test/azure.test.ts:1272, 521d884485d8)
• Lifecycle safety is an explicit project requirement: VISION.md requires claims to retain enough provider/resource context to guarantee cleanup without persisting credentials, while provider-specific lifecycle rules remain in adapters. (VISION.md:5, bf50dbca63b8)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Yaroslav Krempovych: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[excelsier]

Follow-up with a fourth symptom, observed ~1h after filing — it turns the report's Symptom 1 from "annoying but clean" into "leaves billing residue":

Aborted provisioning leaves cloud-side resources undeleted. Occurrence 2's failed run (POST /v1/leases → 500, error 1101 → abandoning uncertain coordinator create cbx_bb7fb1dce94b; recording durable cancellation) had already created the lease's public IP in Azure before the coordinator call failed. The durable-cancellation path cleaned up nothing cloud-side: the pip sat alone (no NIC/VM/disk ever followed, no lease record anywhere) for ~66 minutes until our external sweep flagged it and we deleted it manually. So the abort path appears to record the cancellation client-side without unwinding resources it had already provisioned — on a metered resource.

Suggested addition to the ask: on abort-after-partial-provision, either delete the already-created resources as part of the durable cancellation, or emit their ids in the cancellation record so external tooling knows exactly what to reap.

[excelsier]

Third lease-create occurrence tonight, and it's the worst variant yet — the coordinator 500'd but Azure provisioning succeeded anyway, leaving a running unrecorded VM:

• POST /v1/leases → http 500: {"error":"Durable Object's isolate exceeded its memory limit and was reset."} — new signature (memory pressure, vs the earlier storage-reset and error-1101 texts), hit twice (create + the client's one retry), then the client recorded its durable cancellation and moved on.
• Azure-side, the full resource set completed anyway: pip → NIC → VM (running) → osdisk, all within ~35s of the failed call. crabbox usage/list had zero record of it. Nothing would ever SSH to it; it would have billed indefinitely (Standard_D8as_v5) — we found it via our external sweep ~5 minutes in and deleted the set manually.

So the uncertain-create path can now leave anything from a lone pip (previous comment) up to a running, unrecorded, full-price VM. That moves the cleanup ask from nice-to-have to important: on abort-after-uncertain-create, reap what was provisioned (or emit the resource ids in the cancellation record so external tooling can). Frequency datapoint: three lease-create 500s across ~3 hours on the same evening, three different error texts, suggesting DO resource pressure rather than a single stuck object.

[excelsier]

Frequency + severity update: a 4th lease-create 500 tonight (~4.5h window), and the 2nd instance of the worst variant — POST /v1/leases → 500 (error code 1101), one client retry, durable cancellation recorded… while Azure-side the full provisioning completed anyway: pip → NIC → running VM → osdisk, all within ~35s, zero coordinator record. Same operator, same evening, different lease each time (cbx_6417bbb1ed79 this occurrence). We caught this one ~4 minutes after birth via the census-on-abort rule we adopted after the first instance; without it, another full-price VM billing indefinitely. Pattern across all four: the 500s cluster under concurrent lease-create load from multiple gates. Reiterating the ask with sharper priority: reap-on-cancel (or emit provisioned-resource ids in the cancellation record) now looks essential rather than nice-to-have.