Skip to content

Cut CI setup and tarball smoke time: skip npm registry round trips - #3046

Merged
SawyerHood merged 1 commit into
mainfrom
bb/diagnose-and-speed-up-ci-thr_pjyixc9d3f
Sep 4, 2026
Merged

Cut CI setup and tarball smoke time: skip npm registry round trips#3046
SawyerHood merged 1 commit into
mainfrom
bb/diagnose-and-speed-up-ci-thr_pjyixc9d3f

Conversation

@SawyerHood

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

CI wall time ranged from 9 to 30 minutes for the same workload, and the variance was almost entirely time spent waiting on the npm registry. Every job bootstraps pnpm through pnpm/action-setup, which runs npm ci and then pnpm self-update against the registry: 30 s on a good day, and 7 minutes on the runs where the registry stalled (the stall hits every job of a run at once, so a whole run slips). Across the last 30 runs the Setup workspace step had a median of 140–160 s per job and a p90 of 300–450 s, against 25 s when the registry answered promptly.

The tarball smoke is the critical path of nearly every run (median 382 s on Linux, 345 s on macOS, max 940 s). Profiling it locally with --timing showed the npx --yes --package <tarball> call blocking for 172 s on POST /-/npm/v1/security/advisories/bulk: the smoke's npm install passes --no-audit, but neither of its npx invocations did, so each ran an audit against the registry's slow advisories endpoint.

What changed

  • .github/actions/setup-workspace: pnpm is now downloaded as the pinned release binary from GitHub Releases (install-pnpm.sh, checked against the committed pnpm-9.15.0.sha256), replacing pnpm/action-setup. Bumping PNPM_VERSION needs a new checksum file; the script fails with instructions when one is missing.
  • packages/bb-app/scripts/smoke-tarball.mjs: both npx calls pass --no-audit --no-fund, matching the existing npm install. The smoke also prints per-stage timings (bb-app tarball smoke: <stage> <seconds>) so the next regression is attributable from the CI log.

Other workflows that use pnpm/action-setup directly (deploys, publish, desktop builds) are unchanged; they can adopt the same script later.

How you verified

  • install-pnpm.sh run locally with fake GITHUB_PATH/GITHUB_ENV: installs pnpm 9.15.0 and pnpx; exits 1 with a clear message for a version without a checksum file and for a checksum mismatch.
  • pnpm exec turbo run smoke:tarball --filter=bb-app locally: 246 s before (211 s in the npx step), 40 s after (npx step 4.6 s).
  • CI timings on this PR compared against the 30-run baseline above: see the PR comment once the run finishes.

AGENT GENERATED

Every CI job bootstrapped pnpm through pnpm/action-setup, which runs
`npm ci` and `pnpm self-update` against the npm registry: 30 s at best,
and 7 minutes on runs where the registry stalled, on every job of the run
at once. Setup workspace ran at a median of 140-160 s per job across the
last 30 runs, against 25 s for a job that got a fast registry.

The tarball smoke, the critical path of nearly every run at a median of
382 s on Linux, spent most of that in `npx --package <tarball>`: unlike
the smoke's `npm install`, the npx calls never passed `--no-audit`, so
each one blocked on the registry's bulk advisories endpoint. Locally that
single call took 211 s of a 246 s smoke.

- The setup action downloads the pinned pnpm release binary from GitHub
  Releases, verified against a committed sha256 list, instead of using
  pnpm/action-setup.
- The smoke passes `--no-audit --no-fund` to both npx invocations.
- The smoke prints per-stage timings so the next regression is
  attributable from the CI log instead of needing a local reproduction.

Local smoke: 246 s before, 40 s after.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@SawyerHood

Copy link
Copy Markdown
Collaborator Author

CI timings for this PR's run (33830272710) against the last 30 main/PR runs before it (median / p90 from the jobs API):

Step Before (p50 / p90) This run
Setup workspace, Linux jobs 140–160 s / 300–450 s 20–28 s
Setup workspace, macOS smoke 125 s / 272 s 57 s
Smoke bb-app tarball, Linux 382 s / 516 s 103 s (smoke script itself: 36 s)
Smoke bb-app tarball, macOS 345 s / 519 s 108 s (smoke script itself: 43 s)
Whole run 689–1829 s across the sampled runs 238 s

Per-stage smoke timings now appear in the log; on Linux the npx install that used to block on the audit endpoint took 4.4 s.

AGENT GENERATED

@SawyerHood
SawyerHood merged commit 98db53e into main Sep 4, 2026
16 checks passed
@SawyerHood
SawyerHood deleted the bb/diagnose-and-speed-up-ci-thr_pjyixc9d3f branch September 4, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant