Skip to content

chore(benchmark): compare branches against the noise of the same code - #888

Draft
nigrosimone wants to merge 3 commits into
fastify:mainfrom
nigrosimone:bench-noise
Draft

nigrosimone wants to merge 3 commits into
fastify:mainfrom
nigrosimone:bench-noise

Conversation

@nigrosimone

@nigrosimone nigrosimone commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

npm run bench:cmp benchmarks one branch and then the other, and marks the rows that moved more than 5%. On a laptop the same code moves more than that between two runs (here a fixed CPU loop varied up to 2x), so which branch wins is a lottery.

Now each benchmark runs in four workers, two on the other revision (A) and two on the working tree (B), that take turns in batches of about 10ms. A/A and B/B are the same code against itself and give the noise band of A/B: a row is faster or slower only outside the band, and by at least what is left of A/B. This is #886 against main on Node 26:

short array of numbers...................................... A/A ±2.8%  B/B ±1.9%  A/B +4.5% ±3.1%  faster by at least 1.4%
short array of integers..................................... A/A ±2.4%  B/B ±1.5%  A/B +19.0% ±2.5%  faster by at least 16.1%
short array of short strings................................ A/A ±2.1%  B/B ±2.2%  A/B +235.7% ±2.7%  faster by at least 226.8%
short array of long strings................................. A/A ±3.4%  B/B ±1.7%  A/B +237.3% ±3.4%  faster by at least 226.1%
short array of objects with properties of different types... A/A ±3.0%  B/B ±2.2%  A/B +29.6% ±3.3%  faster by at least 25.5%
simple object............................................... A/A ±1.9%  B/B ±1.3%  A/B +0.7% ±2.1%  noise

The five arrays #886 rewrites are faster, and 23 of the 24 rows where it writes the same code read noise. On Node 24, where it writes the same code everywhere, all 30 rows read noise. bench:cmp:ci, cli-select and simple-git are removed, npm run bench:cmp -- --against <ref> covers them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant