2025 benchmark update

[Kobzol]

In 2025, it will be time to refresh and update our primary benchmarks again, based on our benchmark update policy.

The update policy, and a guide of how to actually update the benchmarks is described here. In short:

1. Download the new benchmark

$ cargo run --bin collector download -c Primary -a <library/binary> crate <name> <version>

2. Add Cargo.lock in the downloaded crate to git, even if it is ignored!
3. Commit
4. Copy perf-config.json and .patch files from the old version
5. Add [workspace] to Cargo.toml if it's not already there
6. Check that

$ cargo build -p collector; cargo run --bin collector bench_local `rustup +stable which rustc` --include <benchmark-name>

works

• If it fails because "patch does not apply", re-create the old patch files based on the updated benchmark source code. For each patch file, perform the same change as was in the original patch file, and then run git diff --patch --relative from the directory of the benchmark and store the output into the patch filename.

7. Modify collector/compile-benchmarks/README.md
8. Modify collector/compile-benchmarks/REUSE.toml
9. Commit
10. Run bench_local for both the old and the new benchmark version, then run the website locally (cargo run --bin site), click on "Display raw data" and "Show non-relevant results" in the "Filters" section and post a screenshot of the results. Before running the benchmark, make sure to clear any stale benchmark data (rm results.db*).

Progress

• bitmaps (@Kobzol)
  • Updated to 3.2.1 (Update bitmaps to 3.2.1 #2040, 14. 4.)
  • Removed old version (Remove bitmaps-3.1.0 benchmark #2075)
• cargo (@Kobzol)
  • Updated to 0.87.1 (Update cargo to 0.87.1 #2071, 16. 4.)
  • Removed old version (Remove cargo 0.60.0 benchmark #2079)
• clap (@Kobzol)
  • Replaced with clap_derive 4.5.32 (Add clap_derive 4.5.32 benchmark #2073, 28. 4.)
  • Removed old version (Remove clap-3.1.6 benchmark #2102)
• cranelift-codegen (@Kobzol)
  • Updated to 0.119.0 (Update cranelift codegen to 0.119.0 #2077, 23. 4.)
  • Removed old version (Remove cranelift-codegen-0.82.1 benchmark #2089)
• diesel (@Kobzol)
  • Updated to 2.2.10 (Update diesel to 2.2.10 #2082, 25. 4.)
  • Removed old version (Remove diesel-1.4.8 benchmark #2090)
• exa (@Kobzol)
  • Replaced with eza 0.21.2 (Add eza 0.21.2 benchmark #2084, 29. 4.)
  • Removed old version (Remove exa-0.10.1 benchmark #2103)
• html5ever (@Kobzol)
  • Updated to 0.31.0 (Update html5ever to 0.31.0 #2069, 15. 4.)
  • Removed old version (Remove html5ever-0.26.0 benchmark #2076)
• hyper (@Kobzol)
  • Updated to 1.6.0 (Update hyper to 1.6.0 #2091, 30. 4.)
  • Removed old version (Remove hyper-0.14.18 #2111)
• image (@Kobzol)
  • Updated to 0.25.6 (Update image to 0.25.6 #2095, 1. 5.)
  • Removed old version (Remove image-0.24.1 #2112)
• libc (@Kobzol)
  • Updated to 0.2.172 (Benchmark update 2025 libc 0.2.172 #2104, 5. 5.)
  • Removed old version
• nalgebra (@Kobzol)
  • Skipped, 0.33.2 (latest version) is too similar to 0.33.0 (in-tree version), as the benchmark was added recently-ish
• regex (@Kobzol)
  • Replaced with regex-automata-0.4.8 (Add regex-automata-0.4.8 benchmark #2109, 7. 5.)
  • Removed old version
• ripgrep (@nnethercote)
  • Updated to 14.1.1 (Add ripgrep-14.1.1 benchmark #2105, 6. 5.)
  • Removed old version
• ripgrep-tiny (@nnethercote)
  • Updated to 14.1.1 (Add ripgrep-14.1.1-tiny benchmark #2106, 7. 5.)
  • Removed old version
• serde (@nnethercote)
  • Updated to 1.0.219 (Add serde-1.0.219 benchmark #2101, 5.5.)
  • Removed old version (Remove serde-1.0.136 benchmark #2115)
• serde_derive (@nnethercote)
  • Updated to 1.0.219 (Add serde_derive-1.0.219 benchmark #2098, 2. 5.)
  • Removed old version (Remove serde_derive-1.0.136 benchmark #2114)
• stm32f4 (@nnethercote)
  • Updated to 0.15.1 (Add stm32f4-0.15.1 benchmark #2099, 2. 5.)
  • Removed old version (Remove stm32f4-0.14.0 benchmark #2113)
• syn (@nnethercote)
  • Updated to 2.0.101 (Add syn-2.0.101 benchmark #2094, 1. 5.)
  • Removed old version (Remove syn-1.0.89 benchmark #2110)
• typenum (@nnethercote)
  • Updated to 0.18.0 (Add typenum-1.18.0 benchmark #2088, 30. 4.)
  • Removed old version (Remove typenum-1.17.0 benchmark #2108)
• unicode-normalization (@nnethercote)
  • Updated to 0.1.24 (Add unicode-normalization-0.1.24 #2087, 30. 4.)
  • Removed old version (Remove unicode-normalization-0.1.19 benchmark #2107)
• webrender (@nnethercote)
  • No new version to be added
  • Removed old version (Remove webrender-2022 #2085)

[nnethercote]

As per the discussion on Zulip, we're going to wait a little bit and get some ducks lined up in a row before starting on this.

[Kobzol]

We have started the benchmark update process this week.

[Kobzol]

Not sure if we need to update ripgrep-tiny, as it's kind of a special cased benchmark for minimum binary size.

[Kobzol]

All primary benchmarks have been ported over 🎉 Now we just need to wait a ~week to also remove all the remaining old versions.