Faster CPython Benchmark Infrastructure

🔒 ▶️ START A BENCHMARK RUN

Results

Here are some recent and important revisions. 👉 Complete list of results.

Most recent pystats on main (22a4421)

linux x86_64 (linux)

date	fork/ref	hash/flags	vs. 3.12.6:	vs. 3.13.0rc2:	vs. base:
2025-01-17	python/d4544cb232dee5f836a6	d4544cb (NOGIL)	1.071x ↓ 📄📈	1.103x ↓ 📄📈	1.145x ↓ 📄📈🧠
2025-01-17	python/d4544cb232dee5f836a6	d4544cb	1.091x ↑ 📄📈	1.048x ↑ 📄📈
2025-01-16	python/b44ff6d0df9ec2d60be6	b44ff6d (NOGIL)	1.062x ↓ 📄📈	1.096x ↓ 📄📈	1.132x ↓ 📄📈🧠
2025-01-16	python/b44ff6d0df9ec2d60be6	b44ff6d	1.081x ↑ 📄📈	1.038x ↑ 📄📈
2025-01-15	python/d05140f9f77d7dfc753d	d05140f (NOGIL)	1.073x ↓ 📄📈	1.103x ↓ 📄📈	1.129x ↓ 📄📈🧠
2025-01-15	python/d05140f9f77d7dfc753d	d05140f	1.074x ↑ 📄📈	1.034x ↑ 📄📈
2025-01-15	python/6e4f64109b0eb6c9f1b5	6e4f641	1.067x ↑ 📄📈	1.023x ↑ 📄📈
2025-01-15	python/6e4f64109b0eb6c9f1b5	6e4f641 (NOGIL)	1.097x ↓ 📄📈	1.128x ↓ 📄📈	1.148x ↓ 📄📈🧠
2025-01-15	python/b52de22ac345ad8583bc	b52de22	1.051x ↑ 📄📈	1.010x ↑ 📄📈
2025-01-15	python/b52de22ac345ad8583bc	b52de22 (NOGIL)	1.102x ↓ 📄📈	1.134x ↓ 📄📈	1.142x ↓ 📄📈🧠

linux x86_64 (vultr)

date	fork/ref	hash/flags	vs. 3.12.6:	vs. 3.13.0rc2:	vs. base:
2025-01-17	python/d4544cb232dee5f836a6	d4544cb (NOGIL)	1.060x ↓ 📄📈	1.090x ↓ 📄📈	1.135x ↓ 📄📈🧠
2025-01-17	python/d4544cb232dee5f836a6	d4544cb	1.087x ↑ 📄📈	1.048x ↑ 📄📈
2025-01-16	python/211f41316b7f205d18eb	211f413	1.088x ↑ 📄📈	1.050x ↑ 📄📈	1.003x ↓ 📄📈🧠
2025-01-16	python/211f41316b7f205d18eb	211f413 (NOGIL)	1.046x ↓ 📄📈	1.076x ↓ 📄📈
2025-01-16	python/b44ff6d0df9ec2d60be6	b44ff6d (NOGIL)	1.047x ↓ 📄📈	1.077x ↓ 📄📈	1.126x ↓ 📄📈🧠
2025-01-16	python/b44ff6d0df9ec2d60be6	b44ff6d	1.090x ↑ 📄📈	1.052x ↑ 📄📈
2025-01-16	python/f48702dade921beed3e2	f48702d	1.091x ↑ 📄📈	1.053x ↑ 📄📈	1.007x ↓ 📄📈🧠
2025-01-16	python/3893a92d956363fa2443	3893a92	1.099x ↑ 📄📈	1.060x ↑ 📄📈
2025-01-16	mpage/aa_test_2	f1b75b3	1.084x ↑ 📄📈	1.046x ↑ 📄📈	1.013x ↓ 📄📈🧠
2025-01-16	mpage/aa_test_1	de82556	1.092x ↑ 📄📈	1.054x ↑ 📄📈	1.000x ↑ 📄📈🧠
2025-01-16	mpage/aa_test_0	54bb905	1.080x ↑ 📄📈	1.042x ↑ 📄📈	1.007x ↓ 📄📈🧠
2025-01-16	mpage/gh_115999_specialize	65a6ad3	1.088x ↑ 📄📈	1.050x ↑ 📄📈	1.001x ↓ 📄📈🧠
2025-01-16	mpage/gh_115999_specialize	65a6ad3 (NOGIL)	1.059x ↓ 📄📈	1.090x ↓ 📄📈	1.013x ↓ 📄📈🧠
2025-01-15	python/d05140f9f77d7dfc753d	d05140f (NOGIL)	1.053x ↓ 📄📈	1.084x ↓ 📄📈	1.127x ↓ 📄📈🧠
2025-01-15	python/d05140f9f77d7dfc753d	d05140f	1.083x ↑ 📄📈	1.045x ↑ 📄📈
2025-01-15	mpage/gh_115999_specialize	73aa5a2	1.107x ↑ 📄📈	1.068x ↑ 📄📈	1.021x ↑ 📄📈🧠
2025-01-15	mpage/gh_115999_specialize	73aa5a2 (NOGIL)	1.058x ↓ 📄📈	1.088x ↓ 📄📈	1.004x ↓ 📄📈🧠
2025-01-15	mpage/gh_115999_specialize	3cedaed (NOGIL)	1.053x ↓ 📄📈	1.084x ↓ 📄📈	1.000x ↑ 📄📈🧠
2025-01-15	mpage/gh_115999_specialize	3cedaed	1.102x ↑ 📄📈	1.063x ↑ 📄📈	1.016x ↑ 📄📈🧠
2025-01-15	python/6e4f64109b0eb6c9f1b5	6e4f641	1.110x ↑ 📄📈	1.071x ↑ 📄📈
2025-01-15	python/6e4f64109b0eb6c9f1b5	6e4f641 (NOGIL)	1.059x ↓ 📄📈	1.089x ↓ 📄📈	1.153x ↓ 📄📈🧠
2025-01-15	python/b52de22ac345ad8583bc	b52de22	1.107x ↑ 📄📈	1.068x ↑ 📄📈
2025-01-15	python/b52de22ac345ad8583bc	b52de22 (NOGIL)	1.066x ↓ 📄📈	1.096x ↓ 📄📈	1.156x ↓ 📄📈🧠
2025-01-14	python/1c13c56a34fc4c4d8969	1c13c56 (NOGIL)	1.167x ↓ 📄📈	1.194x ↓ 📄📈
2025-01-13	python/3efe28a40b136164f0d3	3efe28a (NOGIL)	1.167x ↓ 📄📈	1.195x ↓ 📄📈
2025-01-11	python/553cdc6d6856c1b4539a	553cdc6 (NOGIL)	1.160x ↓ 📄📈	1.187x ↓ 📄📈	1.002x ↓ 📄📈🧠
2024-12-20	python/128cc47fbd44e3e09c50	128cc47 (NOGIL)	1.191x ↓ 📄📈	1.218x ↓ 📄📈	1.003x ↓ 📄📈🧠
2024-12-20	python/78ffba4221dcb2e39fd5	78ffba4 (NOGIL)	1.190x ↓ 📄📈	1.217x ↓ 📄📈

* indicates that the exact same versions of pyperformance was not used.

Improvement of the geometric mean of key merged benchmarks, computed with pyperf compare. The results have a resolution of 0.01 (1%).

Documentation

Running benchmarks from the GitHub web UI

Visit the 🔒 benchmark action and click the "Run Workflow" button.

The available parameters are:

• fork: The fork of CPython to benchmark. If benchmarking a pull request, this would normally be your GitHub username.
• ref: The branch, tag or commit SHA to benchmark. If a SHA, it must be the full SHA, since finding it by a prefix is not supported.
• machine: The machine to run on. One of linux-amd64 (default), windows-amd64, darwin-arm64 or all.
• benchmark_base: If checked, the base of the selected branch will also be benchmarked. The base is determined by running git merge-base upstream/main $ref.
• pystats: If checked, collect the pystats from running the benchmarks.

To watch the progress of the benchmark, select it from the 🔒 benchmark action page. It may be canceled from there as well. To show only your benchmark workflows, select your GitHub ID from the "Actor" dropdown.

When the benchmarking is complete, the results are published to this repository and will appear in the complete table. Each set of benchmarks will have:

• The raw .json results from pyperformance.
• Comparisons against important reference releases, as well as the merge base of the branch if benchmark_base was selected. These include
  • A markdown table produced by pyperf compare_to.
  • A set of "violin" plots showing the distribution of results for each benchmark.

The most convenient way to get results locally is to clone this repo and git pull from it.

Running benchmarks from the GitHub CLI

To automate benchmarking runs, it may be more convenient to use the GitHub CLI. Once you have gh installed and configured, you can run benchmarks by cloning this repository and then from inside it:

gh workflow run benchmark.yml -f fork=me -f ref=my_branch

Any of the parameters described above are available at the commandline using the -f key=value syntax.

Collecting Linux perf profiling data

To collect Linux perf sampling profile data for a benchmarking run, run the _benchmark action and check the perf checkbox. Follow this by a run of the _generate action to regenerate the plots.

License

This repo is licensed under the BSD 3-Clause License, as found in the LICENSE file.