Skip to content

Make it easier to get a summary of the bench results locally - #177

Merged
NullVoxPopuli merged 9 commits into
mainfrom
nvp/bench-summary
Mar 15, 2026
Merged

Make it easier to get a summary of the bench results locally#177
NullVoxPopuli merged 9 commits into
mainfrom
nvp/bench-summary

Conversation

@NullVoxPopuli

@NullVoxPopuli NullVoxPopuli commented Mar 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

e.g.:

pnpm bench:sumary

gets you:

━━━ Summary ━━━

   Benchmark    Control (avg)   Experiment (avg)        Δ
   ──────────   ─────────────   ────────────────   ──────
🟢 gts small          1.58 ms            1.34 ms   -14.9%
🟡 gts medium         5.00 ms            5.10 ms    +2.0%
🔴 gts large         11.62 ms           12.60 ms    +8.4%
🟢 gjs small          1.01 ms          949.44 µs    -5.6%
⚪ gjs medium         3.72 ms            3.70 ms    -0.5%
🟡 gjs large          8.85 ms            9.13 ms    +3.3%
🟡 hbs small        115.55 µs          111.28 µs    -3.7%
🟡 hbs medium         1.71 ms            1.67 ms    -2.5%
⚪ hbs large          8.90 ms            8.86 ms    -0.5%

🟢 faster · 🔴 slower · 🟡 within 5% · ⚪ within 1%

I'm still trying to figure out how to reduce variance here -- not sure what's going on

it appears to be somewhat due to the power-save govenor -- setting the CPU profile to perf gets nicer results:

━━━ Summary ━━━

   Benchmark    Control (p50)   Experiment (p50)        Δ
   ──────────   ─────────────   ────────────────   ──────
🟢 gts small        707.33 µs          664.83 µs    -6.0%
🟢 gts medium         4.09 ms            3.28 ms   -19.7%
🟡 gts large          9.40 ms            9.78 ms    +4.0%
🟡 gjs small        626.22 µs          632.95 µs    +1.1%
🟡 gjs medium         2.47 ms            2.42 ms    -2.0%
🟡 gjs large          6.65 ms            6.82 ms    +2.5%
🟢 hbs small         55.66 µs           52.22 µs    -6.2%
🟡 hbs medium       917.71 µs          942.92 µs    +2.7%
🟢 hbs large          6.89 ms            6.07 ms   -11.8%

🟢 faster · 🔴 slower · 🟡 within 5% · ⚪ within 1%

but... perhaps more confusing in the other direction

@github-actions

github-actions Bot commented Mar 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🏎️ Benchmark Comparison

Benchmark Control (avg) Experiment (avg) Δ
🟡 gts small 2.10 ms 2.07 ms -1.4%
🟡 gts medium 6.94 ms 6.79 ms -2.1%
🔴 gts large 17.28 ms 18.60 ms +7.6%
🟡 gjs small 1.48 ms 1.47 ms -1.2%
⚪ gjs medium 5.49 ms 5.48 ms -0.2%
⚪ gjs large 13.64 ms 13.65 ms +0.1%
🟡 hbs small 199.66 µs 204.90 µs +2.6%
🟡 hbs medium 2.60 ms 2.56 ms -1.3%
🟡 hbs large 13.57 ms 13.92 ms +2.6%

🟢 faster · 🔴 slower · 🟡 within 5% · ⚪ within 1%

Full mitata output 
clk: ~2.04 GHz
cpu: AMD EPYC 7763 64-Core Processor
runtime: node 24.14.0 (x64-linux)

benchmark                   avg (min … max) p75 / p99    (min … top 1%)
------------------------------------------- -------------------------------
gts small (control)            2.10 ms/iter   2.51 ms █▄                   
                       (1.09 ms … 10.57 ms)   5.60 ms ██▃▂                 
                    ( 20.18 kb …   2.81 mb) 633.52 kb ████▇▅▅▃▃▂▄▃▃▄▄▃▁▁▂▂▁

gts small (experiment)         2.07 ms/iter   2.32 ms ▆█                   
                       (1.08 ms … 20.22 ms)   5.45 ms ██▆▂                 
                    ( 60.92 kb …   2.01 mb) 615.31 kb ████▆█▆▃▄▃▃▃▄▂▃▃▂▂▂▁▁

                             ┌                                            ┐
                             ╷┌────────┬───┐                              ╷
         gts small (control) ├┤        │   ├──────────────────────────────┤
                             ╵└────────┴───┘                              ╵
                             ╷┌────────┬─┐                               ╷
      gts small (experiment) ├┤        │ ├───────────────────────────────┤
                             ╵└────────┴─┘                               ╵
                             └                                            ┘
                             1.08 ms            3.34 ms             5.60 ms

summary
  gts small (experiment)
   1.01x faster than gts small (control)

------------------------------------------- -------------------------------
gts medium (control)           6.94 ms/iter   7.93 ms  █▅                  
                       (3.74 ms … 28.78 ms)  17.22 ms ▇██▅▇▇  ▂            
                    (744.45 kb …  12.53 mb)   2.65 mb ███████▇█▄▇▄▃▁▃▁▁▃▁▁▃

gts medium (experiment)        6.79 ms/iter   7.68 ms  █▄                  
                       (3.65 ms … 25.98 ms)  25.59 ms ▅██▆▄                
                    ( 55.17 kb …  12.79 mb)   2.54 mb █████▄▆▂▃▁▁▂▁▁▁▁▁▁▁▁▂

                             ┌                                            ┐
                             ╷ ┌────┬─┐                  ╷
        gts medium (control) ├─┤    │ ├──────────────────┤
                             ╵ └────┴─┘                  ╵
                             ╷ ┌───┬─┐                                    ╷
     gts medium (experiment) ├─┤   │ ├────────────────────────────────────┤
                             ╵ └───┴─┘                                    ╵
                             └                                            ┘
                             3.65 ms           14.62 ms            25.59 ms

summary
  gts medium (experiment)
   1.02x faster than gts medium (control)

------------------------------------------- -------------------------------
gts large (control)           17.28 ms/iter  20.16 ms   █ ▃▃  ▃            
                       (9.02 ms … 39.62 ms)  37.22 ms ▂▇█▇██ ▇█▂           
                    (  4.82 mb …  17.22 mb)   6.91 mb ██████▆███▆▁▁▁▁▁▁▁▆▁▆

gts large (experiment)        18.60 ms/iter  22.02 ms  █▂ █ █              
                       (8.70 ms … 40.19 ms)  39.86 ms ▅██ █▅█▅▅▅▅          
                    (  4.61 mb …  18.28 mb)   7.10 mb ███▇███████▇▇▁▁▁▁▇▁▁▇

                             ┌                                            ┐
                             ╷    ┌──────┬────┐                       ╷
         gts large (control) ├────┤      │    ├───────────────────────┤
                             ╵    └──────┴────┘                       ╵
                             ╷    ┌────────┬────┐                         ╷
      gts large (experiment) ├────┤        │    ├─────────────────────────┤
                             ╵    └────────┴────┘                         ╵
                             └                                            ┘
                             8.70 ms           24.28 ms            39.86 ms

summary
  gts large (control)
   1.08x faster than gts large (experiment)

------------------------------------------- -------------------------------
gjs small (control)            1.48 ms/iter   1.32 ms █                    
                     (875.17 µs … 17.95 ms)   5.63 ms █▆                   
                    (128.45 kb …   2.31 mb) 551.81 kb ██▄▃▃▂▂▁▁▁▂▁▂▂▂▁▁▁▁▁▁

gjs small (experiment)         1.47 ms/iter   1.29 ms █▄                   
                     (848.59 µs … 18.52 ms)   4.98 ms ██                   
                    ( 47.84 kb …   7.48 mb) 564.37 kb ██▅▃▂▂▃▂▂▁▁▂▂▂▂▂▁▂▁▁▁

                             ┌                                            ┐
                             ╷┌────┬                                      ╷
         gjs small (control) ├┤    │──────────────────────────────────────┤
                             ╵└────┴                                      ╵
                             ╷┌────┬                                ╷
      gjs small (experiment) ├┤    │────────────────────────────────┤
                             ╵└────┴                                ╵
                             └                                            ┘
                             848.59 µs           3.24 ms            5.63 ms

summary
  gjs small (experiment)
   1.01x faster than gjs small (control)

------------------------------------------- -------------------------------
gjs medium (control)           5.49 ms/iter   5.97 ms █▃                   
                       (3.02 ms … 28.59 ms)  22.56 ms ██                   
                    (652.76 kb …   7.69 mb)   2.25 mb ████▅▅▃▁▃▁▂▁▂▁▁▁▁▁▁▁▂

gjs medium (experiment)        5.48 ms/iter   6.22 ms █                    
                       (3.11 ms … 28.40 ms)  22.15 ms █▃▃                  
                    (615.88 kb …   4.22 mb)   2.18 mb ████▆▄▄▂▂▂▁▁▁▁▁▁▁▁▁▁▂

                             ┌                                            ┐
                             ╷┌────┬┐                                     ╷
        gjs medium (control) ├┤    │├─────────────────────────────────────┤
                             ╵└────┴┘                                     ╵
                             ╷┌────┬┐                                    ╷
     gjs medium (experiment) ├┤    │├────────────────────────────────────┤
                             ╵└────┴┘                                    ╵
                             └                                            ┘
                             3.02 ms           12.79 ms            22.56 ms

summary
  gjs medium (experiment)
   1x faster than gjs medium (control)

------------------------------------------- -------------------------------
gjs large (control)           13.64 ms/iter  13.94 ms  █▄                  
                       (7.04 ms … 39.77 ms)  39.58 ms  ██ ▆                
                    (  4.09 mb …   8.34 mb)   5.86 mb ▇██▇█▅▃▃▅▃▁▁▁▁▁▁▅▁▁▁▃

gjs large (experiment)        13.65 ms/iter  14.80 ms  ▃█                  
                       (6.93 ms … 38.94 ms)  34.51 ms  ██▂ ▅               
                    (  3.76 mb …   7.73 mb)   5.86 mb ████████▆▁▁▁▁▃▁▁▁▃▁▁▆

                             ┌                                            ┐
                             ╷ ┌──────┬┐                                  ╷
         gjs large (control) ├─┤      │├──────────────────────────────────┤
                             ╵ └──────┴┘                                  ╵
                             ╷  ┌─────┬─┐                          ╷
      gjs large (experiment) ├──┤     │ ├──────────────────────────┤
                             ╵  └─────┴─┘                          ╵
                             └                                            ┘
                             6.93 ms           23.25 ms            39.58 ms

summary
  gjs large (control)
   1x faster than gjs large (experiment)

------------------------------------------- -------------------------------
hbs small (control)          199.66 µs/iter 134.48 µs █                    
                      (97.39 µs … 23.08 ms)   2.47 ms █                    
                    (  9.08 kb …   3.21 mb) 132.18 kb █▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

hbs small (experiment)       204.90 µs/iter 135.41 µs █                    
                      (96.11 µs … 20.00 ms)   2.60 ms █                    
                    ( 20.23 kb …   2.28 mb) 130.67 kb █▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

                             ┌                                            ┐
                             ┌─┬                                        ╷
         hbs small (control) │ │────────────────────────────────────────┤
                             └─┴                                        ╵
                             ┌─┬                                          ╷
      hbs small (experiment) │ │──────────────────────────────────────────┤
                             └─┴                                          ╵
                             └                                            ┘
                             96.11 µs            1.35 ms            2.60 ms

summary
  hbs small (control)
   1.03x faster than hbs small (experiment)

------------------------------------------- -------------------------------
hbs medium (control)           2.60 ms/iter   2.16 ms █                    
                       (1.29 ms … 22.66 ms)  19.51 ms █                    
                    (536.00 kb …   4.13 mb)   1.88 mb █▆▃▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

hbs medium (experiment)        2.56 ms/iter   2.04 ms █                    
                       (1.30 ms … 20.97 ms)  19.42 ms █                    
                    (463.74 kb …   4.17 mb)   1.85 mb █▆▃▄▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

                             ┌                                            ┐
                             ┌──┬                                         ╷
        hbs medium (control) │  │─────────────────────────────────────────┤
                             └──┴                                         ╵
                             ┌──┬                                         ╷
     hbs medium (experiment) │  │─────────────────────────────────────────┤
                             └──┴                                         ╵
                             └                                            ┘
                             1.29 ms           10.40 ms            19.51 ms

summary
  hbs medium (experiment)
   1.01x faster than hbs medium (control)

------------------------------------------- -------------------------------
hbs large (control)           13.57 ms/iter  17.30 ms █                    
                       (7.98 ms … 31.46 ms)  28.95 ms █                    
                    (  5.93 mb …   9.59 mb)   8.46 mb ██▆█▆▃▄▁▁▄▄▁▁▁▁▃▃▄▁▃▄

hbs large (experiment)        13.92 ms/iter  17.13 ms █                    
                       (8.49 ms … 30.25 ms)  28.67 ms █                    
                    (  6.53 mb …   9.46 mb)   8.49 mb █▅▄▇▅▄▂▂▂▄▂▁▁▄▁▁▅▁▂▂▂

                             ┌                                            ┐
                             ╷┌──────────┬───────┐                        ╷
         hbs large (control) ├┤          │       ├────────────────────────┤
                             ╵└──────────┴───────┘                        ╵
                              ╷┌──────────┬──────┐                       ╷
      hbs large (experiment)  ├┤          │      ├───────────────────────┤
                              ╵└──────────┴──────┘                       ╵
                             └                                            ┘
                             7.98 ms           18.47 ms            28.95 ms

summary
  hbs large (control)
   1.03x faster than hbs large (experiment)

NullVoxPopuli
NullVoxPopuli commented Mar 15, 2026
View reviewed changes
Comment thread tests/parser.bench.mjs
@NullVoxPopuli

NullVoxPopuli commented Mar 15, 2026

Copy link
Copy Markdown
Member Author

@copilot do we expose-gc when we run the benchmark?

Copilot AI mentioned this pull request Mar 15, 2026
Closed

Copilot AI commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

@NullVoxPopuli I've opened a new pull request, #178, to work on those changes. Once the pull request is ready, I'll request review from you.

@NullVoxPopuli

NullVoxPopuli commented Mar 15, 2026

Copy link
Copy Markdown
Member Author

@copilot fix lints

Copilot AI commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

@NullVoxPopuli I've opened a new pull request, #179, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot AI mentioned this pull request Mar 15, 2026
Merged
Copilot AI and others added 2 commits March 15, 2026 22:05
@NullVoxPopuli
Fix trailing whitespace lint error in parser.bench.mjs
b32c059 
Co-authored-by: NullVoxPopuli <199018+NullVoxPopuli@users.noreply.github.com>
@NullVoxPopuli
Merge pull request #179 from ember-tooling/copilot/sub-pr-177
0d774d6 
Fix trailing whitespace lint error in parser.bench.mjs
@NullVoxPopuli
NullVoxPopuli merged commit f5cfc76 into main Mar 15, 2026
34 checks passed
@NullVoxPopuli
NullVoxPopuli deleted the nvp/bench-summary branch March 15, 2026 22:18
@github-actions github-actions Bot mentioned this pull request Mar 15, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

internal

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NullVoxPopuli