ci: duration-aware test sharding

Replaces vitest's own --shard (splits test files by COUNT only, no
duration awareness) with an explicit, historically-duration-balanced file
list per shard for the full-suite case. Confirmed this session, via real
per-shard timing pulled from multiple CI runs: shard 4 consistently runs
~20-30% longer than shard 5 (the fastest), every run sampled -- a
structural artifact of hash-sort-and-slice sharding, not noise. Since
validate-tests-merge waits on all 6 shards, the whole job's wall-clock is
set by the slowest one.

scripts/fetch-test-timing.mjs pulls real per-test-file historical duration
from Codecov's Test Analytics API (test-results/), which already pools
JUnit uploads across every push-to-main run -- no need to build a
self-tracked history from scratch. Requires a Codecov personal API access
token (CODECOV_API_TOKEN, distinct from the existing upload-only
CODECOV_TOKEN secret), fetched on a schedule
(.github/workflows/test-timing-refresh.yml, every 6 hours) rather than
per-PR, since Codecov doesn't publish a rate limit for this endpoint and
this repo's PR volume made "query fresh on every PR" a real risk.

scripts/compute-test-shards.mjs bin-packs the full test/**/*.test.ts set
into 6 balanced shards via greedy LPT (sort files descending by duration,
assign each to the currently-lightest shard). New/untracked files (no
timing data yet) fall back to the median of known durations rather than
0 -- treating an unknown file as free would let newly-added heavy test
files land unbalanced. Verified against this repo's real ~1000 test
files and against synthetic realistic-shaped durations (a few heavy
outliers, mostly small files): balances to within ~0.1s across all 6
shards.

Ships with a hard invariant, checked before any output is written: the
union of every shard's file list must equal EXACTLY the discovered file
set, no file missing, no file in more than one shard. A violation refuses
to write output at all rather than risk it -- a bug here would otherwise
mean a test file silently never runs in CI, the exact failure class this
session's earlier Codecov-enforcement work exists to prevent. Verified the
check actually catches a missing/duplicated/phantom file, not just that it
exists.

A real bug was caught by this verification, not shipped: the initial
greedy "pick the lightest shard" reduce always resolved ties in favor of
shard 1 (equal totals never satisfy `<`), which is the common case, not a
theoretical one -- it's exactly what happens on this feature's own first
run, before any timing data has ever been fetched, when every file falls
back to the same weight. Without the fix, 100% of ~1000 files landed in
shard 1 while shards 2-6 stayed empty, verified by running the script
against this repo's real file set before wiring it into ci.yml. Fixed by
rotating the tie-break starting shard after every assignment.

Deliberately scoped to the full-suite case only. A PR using scoped test
selection (--changed=origin/main, for miner/mcp/discoveryIndex-only
diffs) keeps vitest's own native --shard unchanged: that file set isn't
known until vitest resolves --changed itself, so a precomputed assignment
can't cover it without duplicating vitest's own dependency-graph
resolution here.

Verified end-to-end locally (not just each piece in isolation): the exact
combined shell logic from ci.yml's "Test with coverage" step -- bin-pack,
extract this shard's files via the same mapfile/node one-liner, run
vitest against them, write JUnit -- run for real against a slice of this
repo's actual test files. Full local test/unit suite (979 files, 18,487
tests) passes after all changes, including the new
test/unit/compute-test-shards.test.ts (round-robin fallback distribution,
exact-set-match with no duplicates, weighted-balance quality, and the
invariant's two failure-mode error messages).

Author: JSONbored

.github/workflows/ci.yml

@@ -739,6 +739,21 @@ jobs:
           key: turbo-tests-${{ hashFiles('package-lock.json') }}-${{ github.run_id }}
       - name: Prepare test reports dir
         run: mkdir -p reports/junit
+      # Restore-only -- this job never saves it, test-timing-refresh.yml is the sole writer, on its own
+      # schedule (pulling from Codecov's Test Analytics API, which already pools per-test durations
+      # across every push-to-main run). key never actually matches (no save ever uses this literal
+      # string); it exists only so the step always falls through to the restore-keys prefix match,
+      # picking whatever the most recent refresh wrote. A cache MISS here is always safe: see
+      # scripts/compute-test-shards.mjs's fallback -- it splits evenly across shards when no timing
+      # data is available for a file (or none at all), the same balance vitest's own --shard already
+      # gives today, so this can only make shard balance better than today's baseline, never worse.
+      - name: Restore test timing cache
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: test-timing.json
+          key: test-timing-unused
+          restore-keys: |
+            test-timing-
       - name: Test with coverage (shard ${{ matrix.shard }}/6)
         id: coverage
         env:
@@ -795,8 +810,22 @@ jobs:
             # ~2,900-file suite is 0% covered, polluting Codecov's project trend on every scoped PR. Real
             # coverage for files actually exercised is unaffected either way.
             SCOPE_ARGS+=(--changed=origin/main --coverage.all=false)
+            npm run test:coverage -- --maxWorkers=4 --shard=${{ matrix.shard }}/6 --reporter=default --reporter=blob --reporter=junit --outputFile.blob=blob-report/report-${{ matrix.shard }}.blob --outputFile.junit=reports/junit/vitest.xml "${EXCLUDE_ARGS[@]}" "${SCOPE_ARGS[@]}"
+          else
+            # Duration-aware sharding (#ci-duration-aware-sharding), full-suite case only: vitest's own
+            # --shard splits by file COUNT alone, no duration awareness -- confirmed via real per-shard CI
+            # timing data to produce a consistent ~20-30% gap between the slowest and fastest of the 6
+            # shards, every run sampled. Deliberately not applied to the scoped-selection branch above:
+            # that file set isn't known until vitest resolves --changed itself, so a precomputed
+            # assignment can't cover it without duplicating vitest's own dependency-graph resolution here.
+            # compute-test-shards.mjs enforces its own hard invariant (the union of all 6 shards' files
+            # must exactly equal the discovered file set, no file missing or duplicated) and refuses to
+            # write output at all if that's ever violated, so a bug here fails this step loudly rather
+            # than silently dropping a test file from CI.
+            node scripts/compute-test-shards.mjs --shards=6 --timing=test-timing.json --output=shard-assignment.json
+            mapfile -t SHARD_FILES < <(node -e "console.log(JSON.parse(require('fs').readFileSync('shard-assignment.json','utf8'))['${{ matrix.shard }}'].join('\n'))")
+            npm run test:coverage -- --maxWorkers=4 "${SHARD_FILES[@]}" --reporter=default --reporter=blob --reporter=junit --outputFile.blob=blob-report/report-${{ matrix.shard }}.blob --outputFile.junit=reports/junit/vitest.xml "${EXCLUDE_ARGS[@]}"
           fi
-          npm run test:coverage -- --maxWorkers=4 --shard=${{ matrix.shard }}/6 --reporter=default --reporter=blob --reporter=junit --outputFile.blob=blob-report/report-${{ matrix.shard }}.blob --outputFile.junit=reports/junit/vitest.xml "${EXCLUDE_ARGS[@]}" "${SCOPE_ARGS[@]}"
       - name: Test failure guidance
         if: ${{ failure() && steps.coverage.conclusion == 'failure' }}
         run: |

.github/workflows/test-timing-refresh.yml

@@ -0,0 +1,47 @@
+name: Test timing refresh
+
+# Pulls per-test-file historical duration data from Codecov's Test Analytics API (see
+# scripts/fetch-test-timing.mjs) and caches it for validate-tests' duration-aware shard bin-packer
+# (scripts/compute-test-shards.mjs, ci.yml's "Test with coverage" step) to consume. Runs on a schedule
+# rather than per-PR: Codecov doesn't publish a numeric rate limit for this read endpoint, and this
+# repo's PR volume (hundreds/day) makes "query fresh on every PR" a real risk of hitting one, for data
+# that doesn't meaningfully change run-to-run anyway.
+
+on:
+  schedule:
+    - cron: "0 */6 * * *" # every 6 hours
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: test-timing-refresh
+  cancel-in-progress: true
+
+jobs:
+  refresh:
+    name: refresh
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Setup Node
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
+        with:
+          node-version-file: .nvmrc
+      - name: Fetch test timing from Codecov
+        env:
+          CODECOV_API_TOKEN: ${{ secrets.CODECOV_API_TOKEN }}
+        run: node scripts/fetch-test-timing.mjs --output=test-timing.json
+      # Run_id-suffixed key (always a fresh entry, never a re-save of an existing one) + prefix
+      # restore-keys fallback on the consuming side (ci.yml's "Restore test timing cache" step) -- same
+      # accumulating-cache pattern already used for .turbo/cache and .tsbuildinfo elsewhere in this repo,
+      # since this data changes every run rather than being wholesale-replaced per some fixed input hash.
+      - name: Save test timing cache
+        if: ${{ !cancelled() }}
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: test-timing.json
+          key: test-timing-${{ github.run_id }}

scripts/compute-test-shards.mjs

@@ -0,0 +1,146 @@
+#!/usr/bin/env node
+// Bin-packs the full test/**/*.test.ts file set into N balanced shards by historical duration (greedy
+// LPT -- Longest Processing Time first: sort files descending by duration, repeatedly assign the next
+// file to whichever shard currently has the smallest total), replacing vitest's own --shard (file
+// COUNT only, no duration awareness -- confirmed this session via real per-shard CI timing data to
+// produce a consistent ~20-30% gap between the slowest and fastest of 6 shards, every sampled run).
+//
+// Deliberately scoped to the full-suite case only -- see ci.yml's "Test with coverage" step for how
+// this output is (and is NOT) consumed. A PR using scoped test selection (--changed=origin/main)
+// keeps vitest's own native --shard: that file set isn't known until vitest resolves --changed
+// itself, so a precomputed assignment can't apply to it without duplicating vitest's own dependency-
+// graph resolution here.
+//
+// HARD INVARIANT, checked before any output is written: the union of every shard's file list must
+// equal EXACTLY the discovered test file set, with no file in more than one shard. A violation means
+// this script would make CI silently never run some test file at all -- exactly the failure class that
+// this whole session's Codecov-enforcement work exists to prevent, just introduced by the tool meant
+// to speed that same pipeline up. This refuses to write ANY output if the invariant doesn't hold,
+// rather than risk it: a hard failure here (a broken CI step everyone sees) is a wildly better outcome
+// than a silent one (missing coverage nobody notices until something ships broken).
+
+import { readdirSync, readFileSync, writeFileSync, existsSync } from "node:fs";
+import { join } from "node:path";
+
+const TEST_ROOT = "test";
+const EXCLUDED_DIR = join(TEST_ROOT, "workers"); // mirrors vitest.config.ts's exclude: ["test/workers/**/*.test.ts"]
+
+const shardsArg = Number(process.argv.find((a) => a.startsWith("--shards="))?.split("=")[1] ?? 6);
+const timingArg = process.argv.find((a) => a.startsWith("--timing="))?.split("=")[1];
+const outputArg = process.argv.find((a) => a.startsWith("--output="))?.split("=")[1];
+if (!outputArg) throw new Error("--output=<path> is required");
+
+function discoverTestFiles(dir) {
+  const results = [];
+  for (const entry of readdirSync(dir, { withFileTypes: true })) {
+    const full = join(dir, entry.name);
+    if (entry.isDirectory()) {
+      if (full === EXCLUDED_DIR) continue;
+      results.push(...discoverTestFiles(full));
+    } else if (entry.name.endsWith(".test.ts")) {
+      results.push(full);
+    }
+  }
+  return results;
+}
+
+function loadTimingData(path) {
+  if (!path || !existsSync(path)) return {};
+  const parsed = JSON.parse(readFileSync(path, "utf8"));
+  return parsed.averageSecondsByFile ?? {};
+}
+
+function median(values) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const mid = Math.floor(sorted.length / 2);
+  return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
+}
+
+function packShards(files, durationByFile, shardCount) {
+  // New/untracked files (no historical row -- a file added since the last timing refresh, or the
+  // refresh workflow hasn't run yet at all) get the median of known files' durations rather than 0:
+  // treating an unknown file as free would let a burst of newly-added heavy test files land in the
+  // same shard unbalanced, silently reintroducing the exact imbalance this script exists to remove.
+  // If NO file has any known duration at all (the real state before the first timing refresh has ever
+  // run), median() of an empty array is 0 -- every file would tie at the same weight, which matters
+  // below, not just as an edge case: see the tie-break comment.
+  const knownDurations = Object.values(durationByFile);
+  const fallback = median(knownDurations);
+
+  const weighted = files
+    .map((file) => ({ file, duration: durationByFile[file] ?? fallback }))
+    .sort((a, b) => b.duration - a.duration);
+
+  const shards = Array.from({ length: shardCount }, (_unused, index) => ({ index, files: [], total: 0 }));
+  // Picking the lightest shard via a plain reduce always resolves ties in favor of the FIRST shard
+  // (shard.total < min.total is never true between two equal totals, so the running minimum never
+  // moves off its starting candidate) -- harmless when durations vary, but catastrophic whenever many
+  // files tie at the same weight: every one of them would collapse onto shard 1 while the rest stay
+  // empty. This is the real, common case, not a theoretical one -- it's exactly what happens on this
+  // script's very first run, before any timing data has ever been fetched (every file falls back to
+  // the same value, verified by running this script with no --timing argument against this repo's
+  // real ~1000 test files: without the rotation below, 100% of them landed in shard 1). Rotating the
+  // tie-break starting point after every assignment makes N equal-weight files distribute round-robin
+  // across all shards instead, regardless of whether the tied weight happens to be zero or not.
+  let tiebreakStart = 0;
+  for (const { file, duration } of weighted) {
+    let lightest = shards[tiebreakStart];
+    for (let offset = 1; offset < shardCount; offset += 1) {
+      const candidate = shards[(tiebreakStart + offset) % shardCount];
+      if (candidate.total < lightest.total) lightest = candidate;
+    }
+    lightest.files.push(file);
+    lightest.total += duration;
+    tiebreakStart = (lightest.index + 1) % shardCount;
+  }
+  return shards;
+}
+
+function assertInvariant(files, shards) {
+  const original = new Set(files);
+  const seen = new Set();
+  const duplicates = [];
+  for (const shard of shards) {
+    for (const file of shard.files) {
+      if (seen.has(file)) duplicates.push(file);
+      seen.add(file);
+    }
+  }
+  const missing = files.filter((file) => !seen.has(file));
+  const extra = [...seen].filter((file) => !original.has(file));
+
+  if (missing.length > 0 || extra.length > 0 || duplicates.length > 0) {
+    const details = [
+      missing.length > 0 ? `missing from every shard: ${JSON.stringify(missing)}` : null,
+      duplicates.length > 0 ? `assigned to more than one shard: ${JSON.stringify(duplicates)}` : null,
+      extra.length > 0 ? `assigned but not in the discovered file set: ${JSON.stringify(extra)}` : null,
+    ]
+      .filter(Boolean)
+      .join("; ");
+    throw new Error(`compute-test-shards: shard-assignment invariant violated -- ${details}`);
+  }
+}
+
+if (!existsSync(TEST_ROOT)) {
+  throw new Error(`compute-test-shards: ${TEST_ROOT}/ does not exist -- run this from the repo root`);
+}
+const files = discoverTestFiles(TEST_ROOT).sort(); // sorted for deterministic ordering before weighting
+if (files.length === 0) throw new Error(`compute-test-shards: discovered zero test files under ${TEST_ROOT}/ -- refusing to write an empty assignment`);
+
+const durationByFile = loadTimingData(timingArg);
+const shards = packShards(files, durationByFile, shardsArg);
+assertInvariant(files, shards);
+
+const assignment = {};
+shards.forEach((shard, index) => {
+  assignment[String(index + 1)] = shard.files;
+});
+
+writeFileSync(outputArg, JSON.stringify(assignment));
+
+const knownCount = files.filter((f) => f in durationByFile).length;
+console.log(`Assigned ${files.length} files to ${shardsArg} shards (${knownCount} with known timing, ${files.length - knownCount} using the fallback estimate).`);
+shards.forEach((shard, index) => {
+  console.log(`  shard ${index + 1}: ${shard.files.length} files, ~${shard.total.toFixed(1)}s estimated`);
+});

scripts/fetch-test-timing.mjs

@@ -0,0 +1,101 @@
+#!/usr/bin/env node
+// Fetches per-test-file historical duration data from Codecov's Test Analytics API and aggregates it
+// into a per-file average, for the test-shard bin-packer (scripts/compute-test-shards.mjs) to consume.
+// Codecov already ingests a JUnit report per shard on every push to main (see ci.yml's coverage-upload
+// steps, report_type: test_results) and pools it across runs -- this reads that pooled history back
+// out instead of this repo tracking its own duration history from scratch.
+//
+// Filtered to branch=main deliberately: a PR's own JUnit upload is override_branch'd to that PR's own
+// branch name (see ci.yml's upload steps), not "main" -- so branch=main naturally selects only
+// push-triggered, full-unscoped-suite runs, which is exactly the population the shard bin-packer needs
+// (duration-aware sharding only applies to the full-suite case; see compute-test-shards.mjs).
+//
+// Requires a Codecov personal API access token (Codecov Settings -> Access -> Generate Token), NOT the
+// existing CODECOV_TOKEN secret -- that one is an upload-only token and doesn't authenticate this read
+// API. Codecov's docs don't publish a numeric rate limit for this endpoint, so this is deliberately run
+// on a schedule (test-timing-refresh.yml), not per-PR.
+
+import { writeFileSync } from "node:fs";
+
+const MAX_PAGES = Number(process.argv.find((a) => a.startsWith("--max-pages="))?.split("=")[1] ?? 20);
+const outputArg = process.argv.find((a) => a.startsWith("--output="));
+const OUTPUT_PATH = outputArg ? outputArg.split("=")[1] : null;
+
+const repo = process.env.GITHUB_REPOSITORY;
+if (!repo) throw new Error("GITHUB_REPOSITORY is required (e.g. JSONbored/loopover)");
+const [owner, repoName] = repo.split("/");
+const token = process.env.CODECOV_API_TOKEN;
+if (!token) throw new Error("CODECOV_API_TOKEN is required");
+
+const RETRYABLE_STATUS = new Set([429, 500, 502, 503, 504]);
+
+async function fetchWithRetry(url, maxAttempts = 4) {
+  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
+    const response = await fetch(url, {
+      headers: { Authorization: `bearer ${token}`, Accept: "application/json" },
+    });
+    if (response.ok) return response.json();
+    if (!RETRYABLE_STATUS.has(response.status) || attempt === maxAttempts) {
+      throw new Error(`Codecov API error ${response.status} on ${url}: ${await response.text()}`);
+    }
+    const delayMs = 2 ** attempt * 1000;
+    console.warn(`Codecov API returned ${response.status} (attempt ${attempt}/${maxAttempts}), retrying in ${delayMs}ms`);
+    await new Promise((resolve) => setTimeout(resolve, delayMs));
+  }
+  throw new Error("unreachable");
+}
+
+async function fetchAllTestRuns() {
+  const rows = [];
+  let url = `https://api.codecov.io/api/v2/gh/${owner}/repos/${repoName}/test-results/?branch=main&page_size=100`;
+  let pages = 0;
+  while (url && pages < MAX_PAGES) {
+    const body = await fetchWithRetry(url);
+    rows.push(...body.results);
+    url = body.next;
+    pages += 1;
+  }
+  return { rows, truncated: url !== null };
+}
+
+function aggregateByFile(rows) {
+  // Per-file duration must be averaged ACROSS RUNS, not just summed across every row: a file with many
+  // test cases would otherwise dwarf a file with few, and a file that appears in many historical rows
+  // (many runs) would inflate further with each additional run pooled in -- neither reflects "how long
+  // does this file actually take in a single run." So first sum each file's rows *within* a single
+  // commit (that commit's real per-run file duration), then average those per-commit totals across all
+  // commits the file appears in.
+  const perCommitTotals = new Map(); // filename -> Map(commit_sha -> totalSeconds)
+  for (const row of rows) {
+    if (!row.filename || row.duration_seconds == null) continue;
+    if (!perCommitTotals.has(row.filename)) perCommitTotals.set(row.filename, new Map());
+    const commits = perCommitTotals.get(row.filename);
+    commits.set(row.commit_sha, (commits.get(row.commit_sha) ?? 0) + row.duration_seconds);
+  }
+
+  const averages = {};
+  for (const [filename, commits] of perCommitTotals) {
+    const totals = [...commits.values()];
+    averages[filename] = totals.reduce((sum, value) => sum + value, 0) / totals.length;
+  }
+  return averages;
+}
+
+const { rows, truncated } = await fetchAllTestRuns();
+const averageSecondsByFile = aggregateByFile(rows);
+
+const report = {
+  fetchedAt: new Date().toISOString(),
+  sourceRowCount: rows.length,
+  fileCount: Object.keys(averageSecondsByFile).length,
+  truncated, // true if MAX_PAGES was hit before the API ran out of pages -- more history existed than was pulled
+  averageSecondsByFile,
+};
+
+const json = JSON.stringify(report, null, 2);
+if (OUTPUT_PATH) {
+  writeFileSync(OUTPUT_PATH, json);
+  console.log(`Wrote ${report.fileCount} files' timing data (from ${report.sourceRowCount} rows) to ${OUTPUT_PATH}`);
+} else {
+  process.stdout.write(`${json}\n`);
+}