Discussion: making the result table more compact

[pallosp]

My workflow includes running benchmarks in the VS Code terminal regularly.

Currently I can't give the tasks longer names than 20 characters, because they would make the table overflow and break its layout. Therefore I'm suggesting a couple of changes to make the metric columns narrower.

(index)

• Rename the column to #. Saves 5-6 characters.

Latency average

• Limit its precision. The digits after the 5th one are just noise. While this won't save more than 1-2 characters, it will make the numbers easier to read.
• Optional: rename average to avg.

Latency median

• Rename median to 50%.
• Drop the ± part from the numbers. I don't think they give any actionable insight.
• Drop the trailing zeros. Round to integer when the median has too many digits.
• Return numbers, not strings.

These changes together may save 3-7 characters.

Throughput average

• Optional: rename average to avg.

Throughput median

• Rename median to 50%. Saves 3 characters.
• Return numbers, not strings.

Before

Inserting 50k elements into various data structures
┌─────────┬──────────────────────┬──────────────────────┬───────────────────────┬────────────────────────────┬───────────────────────────┬─────────┐
│ (index) │ Task name            │ Latency average (ns) │ Latency median (ns)   │ Throughput average (ops/s) │ Throughput median (ops/s) │ Samples │
├─────────┼──────────────────────┼──────────────────────┼───────────────────────┼────────────────────────────┼───────────────────────────┼─────────┤
│ 0       │ 'float keyed Map'    │ '3218139.39 ± 3.66%' │ '2756229.00 ± 146.00' │ '331 ± 2.19%'              │ '363'                     │ 312     │
│ 1       │ 'int keyed Map'      │ '2211478.20 ± 1.77%' │ '2076166.00'          │ '462 ± 1.06%'              │ '482'                     │ 453     │
│ 2       │ '10 int keyed Maps'  │ '1304029.77 ± 1.12%' │ '1218542.00'          │ '782 ± 0.88%'              │ '821'                     │ 767     │
│ 3       │ 'sparse Array, 0.5'  │ '147581.90 ± 1.37%'  │ '124250.00'           │ '7543 ± 0.44%'             │ '8048'                    │ 6776    │
│ 4       │ 'sparse Array, 0.1'  │ '655357.51 ± 0.44%'  │ '654250.00'           │ '1537 ± 0.43%'             │ '1528'                    │ 1526    │
│ 5       │ 'sparse Array, 0.05' │ '1293428.82 ± 0.42%' │ '1306833.50 ± 0.50'   │ '776 ± 0.51%'              │ '765'                     │ 774     │
└─────────┴──────────────────────┴──────────────────────┴───────────────────────┴────────────────────────────┴───────────────────────────┴─────────┘

After

Inserting 50k elements into various data structures
┌───┬──────────────────────┬──────────────────────┬──────────────────┬────────────────────────────┬────────────────────────┬─────────┐
│ # │ Task name            │ Latency average (ns) │ Latency 50% (ns) │ Throughput average (ops/s) │ Throughput 50% (ops/s) │ Samples │
├───┼──────────────────────┼──────────────────────┼──────────────────┼────────────────────────────┼────────────────────────┼─────────┤
│ 0 │ 'float keyed Map'    │ '3218139 ± 3.66%'    │ 2756229          │ '331 ± 2.19%'              │ 363                    │ 312     │
│ 1 │ 'int keyed Map'      │ '2211478 ± 1.77%'    │ 2076166          │ '462 ± 1.06%'              │ 482                    │ 453     │
│ 2 │ '10 int keyed Maps'  │ '1304029 ± 1.12%'    │ 1218542          │ '782 ± 0.88%'              │ 821                    │ 767     │
│ 3 │ 'sparse Array, 0.5'  │ '147581 ± 1.37%'     │ 124250           │ '7543 ± 0.44%'             │ 8048                   │ 6776    │
│ 4 │ 'sparse Array, 0.1'  │ '655357 ± 0.44%'     │ 654250           │ '1537 ± 0.43%'             │ 1528                   │ 1526    │
│ 5 │ 'sparse Array, 0.05' │ '1293428 ± 0.42%'    │ 1306833          │ '776 ± 0.51%'              │ 765                    │ 774     │
└───┴──────────────────────┴──────────────────────┴──────────────────┴────────────────────────────┴────────────────────────┴─────────┘

If you agree, I'm happy to send a pull request.

[jerome-benoit]

You can use a custom convert arrow function (https://tinylibs.github.io/tinybench/classes/Bench.html#table) to format the results the way you want.
What is currently missing is example usage of it.

[pallosp]

Thanks, that's a great workaround.

I think there is still value in improving the default rendering, as it reduces both the time to write benchmarks and the time to interpret the results.

[jerome-benoit]

The main issue is that same words are repeated because console.table() is not supporting columns grouping.

You can make a PR to refine the default, but:

• median is more explicit than 50% alone. To make it explicit, the term percentile should be added. med is a fine shortcut
• the standard deviation is a meaningful information but should only be displayed if upper than a threshold (8%)

[pallosp]

It turns out that console.table() is pretty dumb. It doesn't even allow to remove to rename the index column.

Are you suggesting to rename average to avg and median to med?

The standard deviations are indeed useful. I didn't want to remove them. What I found less useful is the distance between the two middle samples when there are even number of them.

[jerome-benoit]

Are you suggesting to rename average to avg and median to med?

It was your suggestions ;-) I just think that 50% is less self-explanatory than med.

The standard deviations are indeed useful. I didn't want to remove them. What I found less useful is the distance between the two middle samples when there are even number of them.

It's in fact the confidence interval around the mean that is displayed.

[jdmarshall]

The plus or minus needs to be there but one of the first things I did making my own table was drop the quotes around the numeric values. The fact that they’re inconsistent is just weird. Why is throughput in quotes and samples not in quotes?

[jerome-benoit]

string vs. number handling in console.table()

[jdmarshall]

How about 'mean' instead of 'average'?

And are the cool kids using p50 these days instead of 50%?

I half-snoozed through confidence intervals in physics, but I believe that if you're showing ±2.4% then showing two decimal places as if they are significant figures is wrong.

[jdmarshall]

console.table is also responsible for that '(index)' nonsense as well. Hmmm.

But it looks like console.table exists solely in the examples, so if someone tweaks the column names coming from tinybench, any additional cleanup could be represented in the examples by importing one of the console.table replacements.

Based on my forks, it looks like I used https://github.com/ayonious/console-table-printer/ and then had trouble with it trying to put ansi color codes into non-interactive terminals.

@jerome-benoit Do you have a cli table generator you like?

(also none of the ones I can find support nested column names so that might have to be out)

[jerome-benoit]

I half-snoozed through confidence intervals in physics, but I believe that if you're showing ±2.4% then showing two decimal places as if they are significant figures is wrong.

Reading the code will tell you. Unless you formally prove Student distribution based confidence interval computation is wrong for mean and MAD for median, they are as accurate as possible given the original Student assumptions.

[jdmarshall]

No the code isn’t going to tell me how physicists track significant figures. Even if you copied a physics book into the code it’s a tertiary reference.

[jdmarshall]

Drop the ± part from the numbers. I don't think they give any actionable insight.

I disagree with this. All profilers lie, some more than most. The numbers you don’t think are necessary are used by the people who come after you declare this code is as fast as it can get and find another 10%.

A broad interval and similar numbers to a previous run helps indicate that something interfered with this run and you should try it again. And that your change might not have accomplished anything.

[jerome-benoit]

No the code isn’t going to tell me how physicists track significant figures. Even if you copied a physics book into the code it’s a tertiary reference.

It's going to tell you how the confidence interval is actually computed, what is the method used and if it's correctly implemented, as a primary reference. Unless you have prove:

• the method used had flaws,
• the implementation of the method is buggy,
• the display of the method outputs is buggy,

You are just making bold claims backed by ... nothing.

[jdmarshall]

You’re showing hundredths of a nanoseconds on a VM that is not even accurate to the nanosecond. Two decimal points on a median value is a bold claim. I agree with @pallosp that the decimal points are not helpful. In fact they cannot possibly be correct.

And if multiple runs report values that don’t overlap in their error bars - which happens all the time - then what are the numbers telling a user? It’s like the cosmological crisis. I can’t reproduce my own results let alone someone else’s. Not even on the same hardware.