Skip to content

flaky: two intermittent k8s-compat failures (docker smoke ownership, chart RBAC parity timeout) #535

Description

@blove

Two distinct intermittent failures showed up in the k8s-compat vitest project while validating PR #534, whose diff touches only scripts/release/* and one runbook doc. Neither reproduces in isolation.

Signature 1 — Docker smoke ownership (local, full-suite concurrency)

FAIL |k8s-compat| assert-docker-smoke.test.ts > Docker smoke ownership
  > removes a renamed owned app but preserves its same-name replacement
AssertionError: expected undefined to be defined
 ❯ assert-docker-smoke.test.ts:1832:27
  • Failed once under DAWN_REQUIRE_DOCKER=1 pnpm ci:validate (1 of 5,841 tests).
  • The same file passes 94/94 run on its own, and the single test passes on its own.
  • An immediately preceding full ci:validate on the same k8s-compat code passed, as did a subsequent one (exit 0, 459 test files).

The assertion is that a same-name replacement container survives while the renamed original is removed, so a plausible cause is container-name/state contention when many Docker-touching lanes run concurrently.

Signature 2 — chart RBAC parity timeout (GitHub Actions)

FAIL |k8s-compat| chart-rbac.test.ts > dawn-orchestrator Role parity
  > matches the provider permission declaration exactly
Error: Test timed out in 5000ms.

Seen in CI / validate on the PR head. A 5s default timeout on a parity test suggests it is doing more work (or more I/O) than the budget assumes on a loaded runner, rather than a logic fault.

Why this is worth tracking

Both are unrelated to the PR that surfaced them, and both are silent-until-red: they cost a full re-run to diagnose and erode trust in a green gate. Signature 1 is also a new signature — distinct from the previously seen sandbox-docker fork-bomb flake.

Suggested next steps

  • Signature 1: check whether the ownership fixtures derive container names from a per-worker unique token, or whether two concurrent files can collide on the same name.
  • Signature 2: measure the parity test's actual runtime on a cold runner and either raise its timeout deliberately or move the expensive setup out of the test body.

Additional local validation observation — 2026-09-10 UTC

During #605 retained-PR validation, #489 head 34e166b6c3503a4f30bd3d8670b6932665ea91f6 completed the source suite (5,782 passed, 218 skipped), release-controller, packaging, framework and smoke checks, but the runtime harness timed out in test/runtime/run-runtime-contract.test.ts:235, records restart-induced cancellation from real b4 dev as a non-execution failure, at approximately 181.5 seconds against its 180-second limit. Runtime summary: 16 passed, 1 skipped, 1 failed. This is a distinct observation; it is not evidence that either original Kubernetes/Docker signature reproduced or was fixed.

An unchanged-head isolated rerun passed in 15.78 seconds (31.32 seconds including registry setup). Its persisted artifacts show the raw request returned HTTP 503 with cancellation during shutdown, the CLI returned server_transport_error for HTTP 503, and all three restarts reached ready. The transcript also reports a force-killed stuck dev child during teardown. The full-run timeout bypassed scenario preservation, and cleanup removed its temporary artifacts, so the original blocked await is unknown. No definitive CPU/load or lifecycle-stall cause is established; no timeout or assertion was changed.

The same reviewed head passed GitHub's runtime verification and all required technical checks before merge: CI run. Local evidence is retained at /tmp/pr489-final-ci-validate.log, /tmp/pr489-restart-diagnostic.log, and /tmp/pr489-restart-diagnostic/artifacts/testing/restart-cancellation/graph-basic/. Next reproduction should preserve artifacts independently of the timed-out test's cleanup and inspect child-process teardown before changing timeouts.

The dependent #506 also passed its final main-based runtime verification and every technical check without changing this test or its timeout: CI run, reviewed head 16d8a2692cbee267be251589af25414b911e7644. This is additional passing evidence, not a root-cause fix for the local timeout.

Confirmed Kubernetes SOCKS fixture clock race — 2026-09-10 UTC

The documentation-only #622 at 7883ee9cfd8592c5a0d70e90ee1f472d3728aa08 failed the existing uses the public generated VersionApi operation test in source validation job 103069058198. Node v24.17.0 threw RangeError: headersTimeout must be <= requestTimeout with a received header timeout of 5000, at packages/sandbox/test/kube-socks-proxy.test.ts:679, before listening or making a Kubernetes request.

This cause is confirmed independently of runner load: the unchanged fixture computes the two timeout values from separate Date.now() calls. Advancing the clock by one millisecond produces 5000/4999 and reproduces the exact native HTTP construction error; holding it fixed produces valid 5000/5000 options. The same expressions existed before the documentation PR. This is a distinct fixture race, not evidence for the other Docker, RBAC, or runtime-harness hypotheses above.

The isolated repair in #623 captures one remaining-budget value for both options. Deterministic regressions failed for positive budgets before the fix and passed afterward; all 12 existing/new SOCKS tests, sandbox build/typecheck/lint, and whitespace checks passed locally. The five-second case deadline, cleanup limits, network assertions, and production code remain unchanged. Independent review passed, and #623 merged as 592c065d77d809949741822fa7880d4eb03dd12c after all technical checks, including required validate and the Docker/Kubernetes end-to-end checks, passed on reviewed head 377d598d895e64ce77460dc3f3ff6085c7fab6bf. CI evidence. This issue remains open for its other unresolved signatures.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions