Skip to content

Conversation

@Robbepop
Copy link
Member

@Robbepop Robbepop commented Oct 11, 2022

I got suspicious after I saw huge regressions in the ballpark of 30-80% on my local development platform so I just wanted to make sure that it is not my local environment by running those same benchmarks on the CI runners.

edit: Those regressions are very real ...

@paritytech-cicd-pr
Copy link

paritytech-cicd-pr commented Oct 11, 2022

BENCHMARKS

NATIVEWASMTIME
BENCHMARKMASTERPRDIFFMASTERPRDIFFWASMTIME OVERHEAD
execute/
count_until
888.50µs 1.24ms 🔴 40.06% 2.38ms 3.10ms 🔴 30.07% 🔴 149%
execute/
factorial_iterative
400.45ns 542.71ns 🔴 35.46% 993.53ns 1.30µs 🔴 30.67% 🔴 139%
execute/
factorial_recursive
761.07ns 840.76ns 🔴 10.47% 1.56µs 1.71µs 🔴 9.65% 🔴 103%
execute/
fib_iterative
1.94ms 2.55ms 🔴 36.62% 5.12ms 6.72ms 🔴 31.43% 🔴 163%
execute/
fib_recursive
6.88ms 8.46ms 🔴 23.79% 14.30ms 15.56ms 🔴 9.01% 🟡 84%
execute/
global_bump
1.43ms 1.83ms 🔴 27.29% 3.63ms 4.53ms 🔴 24.79% 🔴 148%
execute/
host_calls
31.80µs 32.44µs ⚪ 2.03% 45.05µs 48.33µs 🔴 7.37% 🟢 49%
execute/
memory_fill
1.54ms 2.32ms 🔴 50.88% 4.33ms 5.81ms 🔴 34.03% 🔴 151%
execute/
memory_sum
1.52ms 2.34ms 🔴 53.50% 4.37ms 5.84ms 🔴 33.68% 🔴 150%
execute/
memory_vec_add
3.03ms 4.81ms 🔴 60.16% 8.97ms 12.11ms 🔴 35.19% 🔴 152%
execute/
recursive_is_even
1.30ms 1.35ms 🔴 3.91% 2.60ms 2.69ms 🔴 3.15% 🟡 99%
execute/
recursive_ok
173.10µs 181.42µs 🔴 5.00% 350.49µs 380.08µs 🔴 8.82% 🔴 109%
execute/
recursive_scan
213.44µs 235.58µs 🔴 10.47% 468.96µs 505.36µs 🔴 7.65% 🔴 115%
execute/
recursive_trap
17.29µs 17.99µs 🔴 4.15% 35.50µs 39.14µs 🔴 10.18% 🔴 118%
execute/
regex_redux
665.58µs 871.56µs 🔴 30.94% 1.60ms 2.08ms 🔴 29.51% 🔴 138%
execute/
rev_complement
596.99µs 876.68µs 🔴 46.98% 1.55ms 2.11ms 🔴 35.85% 🔴 140%
execute/
tiny_keccak
473.22µs 833.78µs 🔴 77.02% 1.42ms 1.80ms 🔴 26.52% 🔴 116%
execute/
trunc_f2i
1.07ms 1.31ms 🔴 22.19% 2.73ms 3.49ms 🔴 27.80% 🔴 167%
instantiate/
wasm_kernel
70.04µs 71.45µs ⚪ 2.09% 128.03µs 92.10µs 🟢 -27.74% 🟢 29%
translate/
wasm_kernel
4.88ms 4.72ms 🟢 -3.24% 9.47ms 9.88ms 🔴 3.72% 🔴 109%

Link to pipeline

@Robbepop Robbepop changed the title Compare benchmarks between rustc stable and beta channels Compare benchmarks between rustc stable and nightly channels Oct 12, 2022
@Robbepop Robbepop marked this pull request as draft October 14, 2022 15:27
@Robbepop
Copy link
Member Author

Closing this since the demonstration for rust-lang/rust#102952 has been shown.

@Robbepop Robbepop closed this Oct 23, 2022
@Robbepop Robbepop deleted the rf-benchcmp-stable-beta branch March 11, 2024 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants