Skip to content

vitest cross-file interference: tests pass alone, fail in combination (reproducible on main) #499

Description

@EtanHey

Found while working #483. Pre-existing on main — proven by running the combination against
49a0b2b with the PR's own changes stashed.

What

Some test files pass in isolation but fail when run together. On 49a0b2b, with no working-tree
changes:

npx vitest run tests/send-to-v2-background-verify.test.ts \
               tests/agent-engine.test.ts \
               tests/pointer-discipline.test.ts
  -> Test Files  1 failed | 2 passed (3)
     Tests      28 failed | 398 passed (426)

Each of those files alone:

npx vitest run tests/agent-engine.test.ts                  -> 363 passed
npx vitest run tests/send-to-v2-background-verify.test.ts  ->  29 passed

The failures are not confined to one subsystem — AgentEngine > waitFor, spawnAgent docking,
boot session capture, role placement reconciliation, plus pointer-discipline and
release-receipts. That spread points at shared process state (module-level singletons, env, or a
shared state dir) rather than a defect in any one test.

Why it matters

The full suite is usually green (132 files, 3106 passed), so this surfaces intermittently rather
than deterministically — which makes it worse, not better:

  • It produced a spurious pre-push rejection on the fix(t2): stop delivery receipts asserting outcomes nobody observed #483 branch that could not be reproduced on
    re-run. A gate that fails at random trains people to re-run instead of read, which is exactly how
    a real regression gets waved through.
  • It means "the suite is green" is not currently a stable claim about a given commit.

Suggested

Identify the shared state. Likely candidates given the failure spread: module-level caches reachable
via vi.resetModules() boundaries, process.env mutated without restore, or a fixed state dir
(tmpdir()-derived constants shared across files rather than mkdtempSync per file — several files
use a fixed TEST_DIR name, e.g. cmux-send-to-v2-verify-test).

tests/test-state-isolation.test.ts exists, so the discipline is already intended; this is a gap in
it rather than a new idea.

— cmuxlayerClaude-ee15a0bc (worker) · claude-code/claude-opus-5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions