Two distinct intermittent failures showed up in the k8s-compat vitest project while validating PR #534, whose diff touches only scripts/release/* and one runbook doc. Neither reproduces in isolation.
Signature 1 — Docker smoke ownership (local, full-suite concurrency)
FAIL |k8s-compat| assert-docker-smoke.test.ts > Docker smoke ownership
> removes a renamed owned app but preserves its same-name replacement
AssertionError: expected undefined to be defined
❯ assert-docker-smoke.test.ts:1832:27
- Failed once under
DAWN_REQUIRE_DOCKER=1 pnpm ci:validate (1 of 5,841 tests).
- The same file passes 94/94 run on its own, and the single test passes on its own.
- An immediately preceding full
ci:validate on the same k8s-compat code passed, as did a subsequent one (exit 0, 459 test files).
The assertion is that a same-name replacement container survives while the renamed original is removed, so a plausible cause is container-name/state contention when many Docker-touching lanes run concurrently.
Signature 2 — chart RBAC parity timeout (GitHub Actions)
FAIL |k8s-compat| chart-rbac.test.ts > dawn-orchestrator Role parity
> matches the provider permission declaration exactly
Error: Test timed out in 5000ms.
Seen in CI / validate on the PR head. A 5s default timeout on a parity test suggests it is doing more work (or more I/O) than the budget assumes on a loaded runner, rather than a logic fault.
Why this is worth tracking
Both are unrelated to the PR that surfaced them, and both are silent-until-red: they cost a full re-run to diagnose and erode trust in a green gate. Signature 1 is also a new signature — distinct from the previously seen sandbox-docker fork-bomb flake.
Suggested next steps
- Signature 1: check whether the ownership fixtures derive container names from a per-worker unique token, or whether two concurrent files can collide on the same name.
- Signature 2: measure the parity test's actual runtime on a cold runner and either raise its timeout deliberately or move the expensive setup out of the test body.
Additional local validation observation — 2026-09-10 UTC
During #605 retained-PR validation, #489 head 34e166b6c3503a4f30bd3d8670b6932665ea91f6 completed the source suite (5,782 passed, 218 skipped), release-controller, packaging, framework and smoke checks, but the runtime harness timed out in test/runtime/run-runtime-contract.test.ts:235, records restart-induced cancellation from real b4 dev as a non-execution failure, at approximately 181.5 seconds against its 180-second limit. Runtime summary: 16 passed, 1 skipped, 1 failed. This is a distinct observation; it is not evidence that either original Kubernetes/Docker signature reproduced or was fixed.
An unchanged-head isolated rerun passed in 15.78 seconds (31.32 seconds including registry setup). Its persisted artifacts show the raw request returned HTTP 503 with cancellation during shutdown, the CLI returned server_transport_error for HTTP 503, and all three restarts reached ready. The transcript also reports a force-killed stuck dev child during teardown. The full-run timeout bypassed scenario preservation, and cleanup removed its temporary artifacts, so the original blocked await is unknown. No definitive CPU/load or lifecycle-stall cause is established; no timeout or assertion was changed.
The same reviewed head passed GitHub's runtime verification and all required technical checks before merge: CI run. Local evidence is retained at /tmp/pr489-final-ci-validate.log, /tmp/pr489-restart-diagnostic.log, and /tmp/pr489-restart-diagnostic/artifacts/testing/restart-cancellation/graph-basic/. Next reproduction should preserve artifacts independently of the timed-out test's cleanup and inspect child-process teardown before changing timeouts.
The dependent #506 also passed its final main-based runtime verification and every technical check without changing this test or its timeout: CI run, reviewed head 16d8a2692cbee267be251589af25414b911e7644. This is additional passing evidence, not a root-cause fix for the local timeout.
Confirmed Kubernetes SOCKS fixture clock race — 2026-09-10 UTC
The documentation-only #622 at 7883ee9cfd8592c5a0d70e90ee1f472d3728aa08 failed the existing uses the public generated VersionApi operation test in source validation job 103069058198. Node v24.17.0 threw RangeError: headersTimeout must be <= requestTimeout with a received header timeout of 5000, at packages/sandbox/test/kube-socks-proxy.test.ts:679, before listening or making a Kubernetes request.
This cause is confirmed independently of runner load: the unchanged fixture computes the two timeout values from separate Date.now() calls. Advancing the clock by one millisecond produces 5000/4999 and reproduces the exact native HTTP construction error; holding it fixed produces valid 5000/5000 options. The same expressions existed before the documentation PR. This is a distinct fixture race, not evidence for the other Docker, RBAC, or runtime-harness hypotheses above.
The isolated repair in #623 captures one remaining-budget value for both options. Deterministic regressions failed for positive budgets before the fix and passed afterward; all 12 existing/new SOCKS tests, sandbox build/typecheck/lint, and whitespace checks passed locally. The five-second case deadline, cleanup limits, network assertions, and production code remain unchanged. Independent review passed, and #623 merged as 592c065d77d809949741822fa7880d4eb03dd12c after all technical checks, including required validate and the Docker/Kubernetes end-to-end checks, passed on reviewed head 377d598d895e64ce77460dc3f3ff6085c7fab6bf. CI evidence. This issue remains open for its other unresolved signatures.
Two distinct intermittent failures showed up in the
k8s-compatvitest project while validating PR #534, whose diff touches onlyscripts/release/*and one runbook doc. Neither reproduces in isolation.Signature 1 — Docker smoke ownership (local, full-suite concurrency)
DAWN_REQUIRE_DOCKER=1 pnpm ci:validate(1 of 5,841 tests).ci:validateon the same k8s-compat code passed, as did a subsequent one (exit 0, 459 test files).The assertion is that a same-name replacement container survives while the renamed original is removed, so a plausible cause is container-name/state contention when many Docker-touching lanes run concurrently.
Signature 2 — chart RBAC parity timeout (GitHub Actions)
Seen in
CI / validateon the PR head. A 5s default timeout on a parity test suggests it is doing more work (or more I/O) than the budget assumes on a loaded runner, rather than a logic fault.Why this is worth tracking
Both are unrelated to the PR that surfaced them, and both are silent-until-red: they cost a full re-run to diagnose and erode trust in a green gate. Signature 1 is also a new signature — distinct from the previously seen
sandbox-dockerfork-bomb flake.Suggested next steps
Additional local validation observation — 2026-09-10 UTC
During #605 retained-PR validation, #489 head
34e166b6c3503a4f30bd3d8670b6932665ea91f6completed the source suite (5,782 passed, 218 skipped), release-controller, packaging, framework and smoke checks, but the runtime harness timed out intest/runtime/run-runtime-contract.test.ts:235,records restart-induced cancellation from real b4 dev as a non-execution failure, at approximately 181.5 seconds against its 180-second limit. Runtime summary: 16 passed, 1 skipped, 1 failed. This is a distinct observation; it is not evidence that either original Kubernetes/Docker signature reproduced or was fixed.An unchanged-head isolated rerun passed in 15.78 seconds (31.32 seconds including registry setup). Its persisted artifacts show the raw request returned HTTP 503 with cancellation during shutdown, the CLI returned
server_transport_errorfor HTTP 503, and all three restarts reached ready. The transcript also reports a force-killed stuck dev child during teardown. The full-run timeout bypassed scenario preservation, and cleanup removed its temporary artifacts, so the original blocked await is unknown. No definitive CPU/load or lifecycle-stall cause is established; no timeout or assertion was changed.The same reviewed head passed GitHub's runtime verification and all required technical checks before merge: CI run. Local evidence is retained at
/tmp/pr489-final-ci-validate.log,/tmp/pr489-restart-diagnostic.log, and/tmp/pr489-restart-diagnostic/artifacts/testing/restart-cancellation/graph-basic/. Next reproduction should preserve artifacts independently of the timed-out test's cleanup and inspect child-process teardown before changing timeouts.The dependent #506 also passed its final main-based runtime verification and every technical check without changing this test or its timeout: CI run, reviewed head
16d8a2692cbee267be251589af25414b911e7644. This is additional passing evidence, not a root-cause fix for the local timeout.Confirmed Kubernetes SOCKS fixture clock race — 2026-09-10 UTC
The documentation-only #622 at
7883ee9cfd8592c5a0d70e90ee1f472d3728aa08failed the existinguses the public generated VersionApi operationtest in source validation job 103069058198. Node v24.17.0 threwRangeError: headersTimeout must be <= requestTimeoutwith a received header timeout of 5000, atpackages/sandbox/test/kube-socks-proxy.test.ts:679, before listening or making a Kubernetes request.This cause is confirmed independently of runner load: the unchanged fixture computes the two timeout values from separate
Date.now()calls. Advancing the clock by one millisecond produces 5000/4999 and reproduces the exact native HTTP construction error; holding it fixed produces valid 5000/5000 options. The same expressions existed before the documentation PR. This is a distinct fixture race, not evidence for the other Docker, RBAC, or runtime-harness hypotheses above.The isolated repair in #623 captures one remaining-budget value for both options. Deterministic regressions failed for positive budgets before the fix and passed afterward; all 12 existing/new SOCKS tests, sandbox build/typecheck/lint, and whitespace checks passed locally. The five-second case deadline, cleanup limits, network assertions, and production code remain unchanged. Independent review passed, and #623 merged as
592c065d77d809949741822fa7880d4eb03dd12cafter all technical checks, including required validate and the Docker/Kubernetes end-to-end checks, passed on reviewed head377d598d895e64ce77460dc3f3ff6085c7fab6bf. CI evidence. This issue remains open for its other unresolved signatures.