Skip to content

Conversation

@Pavel-Durov
Copy link
Contributor

@Pavel-Durov Pavel-Durov commented Dec 16, 2025

This allows users to specify the confidence level for the summary statistics.
Example of haste diff output:

$ haste diff -c 99 1 0
confidence level: 99%

 Benchmark             Datum1 (ms)   Datum0 (ms)  Ratio  Summary           
 permute/lua/1000      8483 ±  597   7882 ±  292   0.93  indistinguishable 
 bounce/lua/1500       9458 ±  381   9054 ±  192   0.96  indistinguishable 
 list/lua/1500         8758 ±  552   8512 ±  381   0.97  indistinguishable 
 richards/lua/100     42362 ±  484  41613 ± 1391   0.98  indistinguishable 
 deltablue/lua/12000  13889 ±  555  13665 ±  302   0.98  indistinguishable 
 nbody/lua/250000      4649 ±  401   4604 ±  100   0.99  indistinguishable 
 json/lua/100         16726 ±   85  16623 ±  165   0.99  indistinguishable 
 cd/lua/250           39655 ±  757  39490 ±  175   1.00  indistinguishable 
 mandelbrot/lua/500    1308 ±    3   1307 ±    4   1.00  indistinguishable 
 sieve/lua/3000        4160 ±   52   4174 ±   34   1.00  indistinguishable 
 storage/lua/1000     23806 ±  100  23894 ±  124   1.00  indistinguishable 
 queens/lua/1000       5114 ±  290   5160 ±  169   1.01  indistinguishable 
 towers/lua/600        9377 ±  348   9517 ±  810   1.01  indistinguishable 
 havlak/lua/1500      78254 ± 1153  82589 ±12269   1.06  indistinguishable 

$ haste diff -c 95 1 0
confidence level: 95%

 Benchmark            Datum1 (ms)  Datum0 (ms)  Ratio  Summary           
 permute/lua/1000      8483 ± 454   7882 ± 222   0.93  indistinguishable 
 bounce/lua/1500       9458 ± 290   9054 ± 146   0.96  indistinguishable 
 list/lua/1500         8758 ± 420   8512 ± 290   0.97  indistinguishable 
 richards/lua/100     42362 ± 368  41613 ±1058   0.98  indistinguishable 
 deltablue/lua/12000  13889 ± 422  13665 ± 230   0.98  indistinguishable 
 nbody/lua/250000      4649 ± 305   4604 ±  76   0.99  indistinguishable 
 json/lua/100         16726 ±  65  16623 ± 126   0.99  indistinguishable 
 cd/lua/250           39655 ± 576  39490 ± 133   1.00  indistinguishable 
 mandelbrot/lua/500    1308 ±   2   1307 ±   3   1.00  indistinguishable 
 sieve/lua/3000        4160 ±  39   4174 ±  26   1.00  indistinguishable 
 storage/lua/1000     23806 ±  76  23894 ±  94   1.00  indistinguishable 
 queens/lua/1000       5114 ± 221   5160 ± 129   1.01  indistinguishable 
 towers/lua/600        9377 ± 265   9517 ± 616   1.01  indistinguishable 
 havlak/lua/1500      78254 ± 877  82589 ±9335   1.06  indistinguishable 
pd@bencher16 ~/yk-benchmarks main ~ 9ada704646f9c85fea307722806cfd30ce084831

$ haste diff -c 90 1 0
confidence level: 90%

 Benchmark            Datum1 (ms)  Datum0 (ms)  Ratio  Summary           
 permute/lua/1000      8483 ± 381   7882 ± 186   0.93  7.08% faster      
 bounce/lua/1500       9458 ± 243   9054 ± 122   0.96  4.27% faster      
 list/lua/1500         8758 ± 353   8512 ± 243   0.97  indistinguishable 
 richards/lua/100     42362 ± 309  41613 ± 888   0.98  indistinguishable 
 deltablue/lua/12000  13889 ± 354  13665 ± 193   0.98  indistinguishable 
 nbody/lua/250000      4649 ± 256   4604 ±  64   0.99  indistinguishable 
 json/lua/100         16726 ±  54  16623 ± 106   0.99  indistinguishable 
 cd/lua/250           39655 ± 483  39490 ± 112   1.00  indistinguishable 
 mandelbrot/lua/500    1308 ±   2   1307 ±   3   1.00  indistinguishable 
 sieve/lua/3000        4160 ±  33   4174 ±  22   1.00  indistinguishable 
 storage/lua/1000     23806 ±  64  23894 ±  79   1.00  indistinguishable 
 queens/lua/1000       5114 ± 185   5160 ± 108   1.01  indistinguishable 
 towers/lua/600        9377 ± 222   9517 ± 517   1.01  indistinguishable 
 havlak/lua/1500      78254 ± 736  82589 ±7835   1.06  indistinguishable 

This allows users to specify the confidence level for  the summary statistics.
@vext01
Copy link
Contributor

vext01 commented Dec 16, 2025

LGTM

@vext01 vext01 added this pull request to the merge queue Dec 16, 2025
Merged via the queue into ykjit:main with commit 2df6ebe Dec 16, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants