|
| 1 | +# Oracle-negation spike: are our unit tests load-bearing? |
| 2 | + |
| 3 | +**Question** (2026-08-07): with a large unit suite and a mock-free |
| 4 | +provider-integration lane, which unit tests are decorative and safe to remove? |
| 5 | +Coverage intersection cannot answer this — two tests on the same covered path |
| 6 | +can check entirely different behaviour (the checked-coverage argument: |
| 7 | +coverage says a line *ran*, not that any assertion *depended* on it). So this |
| 8 | +spike measured oracle liveness directly. |
| 9 | + |
| 10 | +## Method |
| 11 | + |
| 12 | +Invert every assertion in every `*.test.ts` file and re-run the suite. A test |
| 13 | +that **passes with all its assertions inverted** has no live oracle: its |
| 14 | +assertions either never execute or cannot distinguish anything. A test that |
| 15 | +fails is load-bearing on at least one executed assertion. |
| 16 | + |
| 17 | +- `expect`: a proxy routes every terminal matcher through one extra `.not`. |
| 18 | + Chai's `.not` is a flag-*set*, not a toggle, so stacking `.not` on a |
| 19 | + user-negated chain is a no-op — the proxy instead tracks user `.not` as |
| 20 | + parity and inverts those chains by *removing* the negation (calling the raw |
| 21 | + matcher). `resolves`/`rejects` wrap recursively; statics pass through. |
| 22 | +- `node:assert/strict`: a wrapper module inverts each method (holding throws, |
| 23 | + failing swallows; promise-returning methods invert asynchronously). |
| 24 | + `assert.fail` keeps its always-throw semantics — a reached |
| 25 | + `fail('unreachable')` is a live oracle. |
| 26 | +- A codemod rewrites the two (uniform) import shapes in test files only; |
| 27 | + shared helpers/worlds keep natural assertions. Setup files and |
| 28 | + `test/contention-retry-fixtures/` are excluded. |
| 29 | +- Self-check per the vacuity doctrine in `testing.md`: seven planted tests |
| 30 | + (dead-branch expect/assert, never-invoked callback, plain matcher, user |
| 31 | + `.not`, async `rejects`, executed assert) all produced the expected verdict |
| 32 | + before the sweep ran. |
| 33 | + |
| 34 | +Harness lives in session scratch (`.tmp/negation/`): `neg.ts` (proxy), |
| 35 | +`negated-assert.ts` (wrapper), `codemod.mjs` (import rewriter). Re-creating it |
| 36 | +from this description is ~150 lines. |
| 37 | + |
| 38 | +## Result: zero vacuous tests |
| 39 | + |
| 40 | +Sweep across all five Vitest projects, 677 files codemodded, 5,687 |
| 41 | +verdict-bearing tests: **5,537 failed under negation (live), 150 passed** — |
| 42 | +and every survivor decomposed into a harness artifact or a real-but-indirect |
| 43 | +oracle, verified by reading each cluster: |
| 44 | + |
| 45 | +| Survivors | Class | Verdict | |
| 46 | +| --- | --- | --- | |
| 47 | +| 112 | `assert.rejects`/`assert.throws` with a validator callback: the negated validator throws *inside* the un-negated outer wrapper, which reports "rejected as required" either way | Live — the pattern itself proves an executed error-path oracle | |
| 48 | +| 31 | Oracle lives in a shared un-negated helper (`assertRpcError`, `assertInvalidArgsMessage`, `gesture-plan-test-utils.ts`, interaction-contract helpers) | Live — sampled four distinct clusters and confirmed each | |
| 49 | +| 6 | A guard assertion *inside an in-file fake* (e.g. `readSessionPort`'s `assert.notEqual(index, -1)` in `snapshot-helper-session.test.ts`) inverted and broke the fake, flipping the production path so the real assertions went false-and-swallowed | Live — artifact of negating the whole file | |
| 50 | +| 1 | `watchos-sentinel.test.ts`: the only assertion sat in a `catch` that never fires (tvOS interactor creation succeeds) | Conditionally live; strengthened to an unconditional success pin in this spike — the strengthened oracle flips from survivor to failure under negation | |
| 51 | + |
| 52 | +So the suite's oracles are in excellent shape, and **no deletions are |
| 53 | +justified by vacuity evidence**. This matches the assertion-density audit run |
| 54 | +alongside: 3.4 assertions/test overall, only ~2.4% of assertions are pure |
| 55 | +mock-introspection (`toHaveBeenCalledWith`-style with no behavioural check). |
| 56 | + |
| 57 | +## Companion measurement: coverage uniqueness of mock-coupled tests |
| 58 | + |
| 59 | +150 test files mock first-party modules (306 of 316 `vi.mock` calls target our |
| 60 | +own code). Excluding them from a full coverage run: |
| 61 | + |
| 62 | +| Run | Lines | Branches | |
| 63 | +| --- | --- | --- | |
| 64 | +| Full suite | 89.73% | 78.94% | |
| 65 | +| Minus the 150 mock-coupled files | 77.85% | 67.25% | |
| 66 | +| Mock-free integration lanes only | 48.41% | 37.86% | |
| 67 | + |
| 68 | +The mock-coupled files uniquely hold 4,847 lines across 283 production files — |
| 69 | +but only **47 files drop below 20% coverage** without them. Those 47 are not a |
| 70 | +test-discipline problem; they map onto production modules that bypass the |
| 71 | +provider seams (`src/platforms/android/devices.ts` calls `runCmd('adb', …)` |
| 72 | +around `AndroidAdbProvider`; `runner-session.ts` calls `runCmdBackground('xcodebuild', …)` |
| 73 | +because `AppleToolProvider` has no background member; `agent-browser-provider.ts`, |
| 74 | +`daemon-client-lifecycle.ts`, `perf-xctrace.ts` likewise). Close a seam and its |
| 75 | +mock-only tests become provider-scenario-reachable — and the module becomes |
| 76 | +eligible for the mutation registry, whose membership rule excludes |
| 77 | +subprocess-spawning code by construction. |
| 78 | + |
| 79 | +## What would license pruning (and what to do instead) |
| 80 | + |
| 81 | +Neither coverage overlap (unpredictable fault-detection loss in the suite- |
| 82 | +minimization literature) nor oracle liveness (this spike: everything is live) |
| 83 | +identifies removable tests here. The remaining mechanical instrument is |
| 84 | +**mutation score deltas**: a test whose removal does not lower its module's |
| 85 | +mutation score is redundant *with evidence*. That is `pnpm mutation:affected` |
| 86 | +scope-widened, and it becomes affordable per-PR by mutating only changed lines. |
| 87 | + |
| 88 | +Follow-ups this spike motivates, in value order: |
| 89 | + |
| 90 | +1. **Diff-scoped mutation gate** — mechanize the `testing.md` red-run rule: |
| 91 | + mutate the lines a PR changes; require the PR's tests to kill them. |
| 92 | +2. **Seam closures** for the 47 sole-owner modules above; each closure moves |
| 93 | + files out of the serialized `subprocess-stub` project and shrinks the |
| 94 | + contention-retry waiver list. |
| 95 | +3. **Transcript provenance** — provider-scenario worlds are hand-authored |
| 96 | + beliefs about `simctl`/`adb` output. Add a capture mode on the live-device |
| 97 | + lanes and a nightly drift diff, the same shape as the replay-compat corpus |
| 98 | + and its provenance check. |
| 99 | +4. **Wiring-assertion strengthening** — the files where mock-introspection |
| 100 | + assertions concentrate (`snapshot-handler.test.ts`, `interaction.test.ts`, |
| 101 | + `session-open-url-prewarm.test.ts`, `react-native.test.ts`) are candidates |
| 102 | + for asserting response payloads instead of dispatch call shapes. |
| 103 | + |
| 104 | +## Re-run checklist |
| 105 | + |
| 106 | +1. Rebuild the three harness files (see Method) under `.tmp/negation/`. |
| 107 | +2. Codemod, then `pnpm exec vitest run --reporter=json --outputFile=…`. |
| 108 | +3. Classify survivors *before* believing them: peel `assert.rejects` |
| 109 | + validators, in-file-fake breakage, and helper-oracle files first — in this |
| 110 | + run, 149 of 150 survivors were exactly those three classes. |
| 111 | +4. `git restore src packages test scripts` — the codemod globs stop at src/packages/test today, but restore wider than you codemodded; the negated imports must never reach a commit. |
0 commit comments