Introduce Ractor benchmarks and harness #375

luke-gruber · 2025-08-08T18:29:40Z

This PR introduces a set of benchmarks for assessing the performance of Ractors. The main parts of this work are

1. Introduce a set of Ractor specific performance benchmarks

A new harness harness-ractor.rb has been introduced. This harness will run the benchmark code inside a number of Ractors concurrently and collect timing information. By default this is 0 (ractors disabled), 1, 2, 4 and 8.

A number of Ractor specific benchmarks designed to test specific scenarios of multi-ractor mode have been introduced into the benchmarks-ractor directory. They've also been included in benchmarks.yml with the name prefix ractor/.

In order to run these benchmarks with the new harness a new category ractor-only has been introduced. The ractor only benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor-only

2. Adapt existing benchmarks to be Ractor compatible

Existing benchmarks in the benchmarks directory have been updated (where it's possible to do so in a simple manner) to be multi-Ractor safe (mostly making sure any common state is made shareable using Ractor.make_shareable). Benchmarks that have been updated in this way have been marked inside benchmarks.yml with ractor: true.

These tests can be run with either the normal benchmark harness or the Ractor harness, they should be expected to work with both.

A new category ractor has been introduced that will run all of these tests with the Ractor harness alongside the Ractor only tests mentioned above.

All Ractor benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor

Any benchmark can be run with the Ractor harness using the run_once.sh script in conjunction with the HARNESS environment variable

HARNESS=harness-ractor ./run_once.sh benchmarks/activerecord/benchmark.rb

Or a category can be run using the --harness argument to the main benchmark runner

./run_benchmarks.rb --harness=harness-ractor --category=headline

But be aware that running like this will ignore the ractor key in benchmarks.yml and will run all the chosen benchmarks with the ractor harness, regardless of compatibility. Crashes are likely.

3. Harness chain.

For convenience and as an aid to profiling with ractors, this PR also implements the ability to chain harnesses together. For example, running the ractor-only benchmarks through the Vernier profiler can be done as follows:

HARNESS_CHAIN="vernier,ractor" ./run_benchmarks.rb --harness=harness-chain --category=ractor-only

Co-Authored-By: Luke Gruber <[email protected]> Co-Authored-By: Étienne Barrié <[email protected]>

Committed seperately so the noisy diff can be easily excluded from reviews if needed.

These can be run as follows: ruby -Iharness-ractor/harness.rb benchmarks-ractor/json_parse_float/benchmark.rb Co-Authored-By: Luke Gruber <[email protected]>

Co-Authored-By: Luke Gruber <[email protected]>

If your ruby's bundler is different from the saved result in `Gemfile.lock`, bundler autoswitches by starting a new process. When it does this it sometimes doesn't pick up the old LOAD_PATH and results in a LoadError. Switch to use `Bundler.setup`, which is a simpler process and uses the current bundler.

2 implementations, one using Process.fork, and one using Ractors - for comparison

This is a copy of the Optcarrot benchmark from benchmarks/optcarrot, but modified to be compatible with multiple Ractors

Instead -Iharness-name can be used

This commit introduces a harness that can be used to chain two (or more) harnesses together. For example. To run a Ractor test with Vernier you can wrap the ractor harness with the vernier one as follows HARNESS_CHAIN="vernier,ractor" ruby -Iharness-chain benchmarks-ractor/optcarrot/benchmark.rb

This commit modifies the benchmark runner to add 2 new categories: --category=ractor-only Runs only those benchmarks in benchmarks-ractor using the Ractor harness. --category=ractor In addition to the ractor-only benchmarks, this will run benchmarks specified in benchmarks.yml with `ractor: true` using the Ractor harness. In addition to this we've modified the burn in and run once scripts to use the new harnesses and categories too Co-Authored-By: Luke Gruber <[email protected]>

Also make sure that the benchmarks list test is up to date

tenderlove · 2025-10-15T21:29:07Z

This looks good to me, but I feel I'm missing something. Are the bechmarks just copied fully in to benchmarks-ractor and then made to be Ractor compatible? Also it seems like there were a bunch of images copied, should they have been?

luke-gruber · 2025-10-15T21:35:35Z

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel. I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

tenderlove · 2025-10-15T21:43:33Z

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel.

Got it, makes sense.

I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

If they're not buying us anything, I think we should remove them. They look like possible benchmark results? I'd hate for people to be confused by them.

eregon · 2025-10-15T22:17:43Z

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

That seems like a nice way to make OptCarrot Ractor-compatible without copying many files.
That specific file could be modified to not run the benchmark (e.g. if $0 != __FILE__) and required by benchmarks/optcarrot/benchmark.rb if using the Ractor harness (like for other benchmarks which were not copied). Or e.g. just copying the .freeze calls in that file.

I think it's best to avoid large copies in the repo, and benchmarks-ractor/optcarrot seems the only offender in this PR.

k0kubun · 2025-10-16T01:39:06Z

Looks good to me.

Make 30k_methods benchmark Ractor compatible

What was the change? The diff is so large that GitHub doesn't show the diff.

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

If this is possible, I guess we could let benchmarks-ractor/optcarrot/benchmark.rb load optcarrot from ../../benchmarks/optcarrot, use shims like optcarrot-bench-parallel-on-ractor, and run the benchmark.

eightbitraptor · 2025-10-16T15:37:28Z

Thanks @k0kubun @eregon I hadn't seen the Optcarrot ractor script yet. I'll work on integrating that instead 👍

eightbitraptor · 2025-10-18T21:09:34Z

Ok. Branch has been updated with revised Optcarrot Ractor benchmark. Thanks all.

eregon · 2025-10-19T08:13:30Z

harness-ractor/harness.rb

+require_relative '../harness/harness-common'
+
+Warning[:experimental] = false
+ENV["YJIT_BENCH_RACTOR_HARNESS"] = "1"


Since this is a new env variable, how about following the new name of this repo? (it will be harder to change it later, but it's easy now)

Suggested change

ENV["YJIT_BENCH_RACTOR_HARNESS"] = "1"

ENV["RUBY_BENCH_RACTOR_HARNESS"] = "1"

And the same for YJIT_BENCH_RACTORS below.

eregon · 2025-10-19T08:56:52Z

I gave it a try and made benchmarks/optcarrot/benchmark.rb Ractor-compatible in #422, which means no copy at all of OptCarrot files. I checked locally and it runs fine on both ruby-master and 3.4.7.
IOW, optcarrot is treated like other benchmarks (e.g. rubykon) which simply use if ENV["YJIT_BENCH_RACTOR_HARNESS"] to make them Ractor-compatible.
If you like it, I think the best would be copy these commits in this PR (e.g. git merge merge --ff-only no-optcarrot-copy), and potentially squash the removal of benchmarks-ractor/optcarrot with the commits adding it to make the history simpler (unless you want to keep that just in case, fine by me).

The only "copied" benchmark is then knucleotide, and that seems fine as-is because it's pretty small and has significant differences with the fork-based version in benchmarks/knucleotide/benchmark.rb.

eregon · 2025-10-19T09:03:43Z

harness-ractor/harness.rb

+def use_ractor_gemfile(filename)
+  filename = File.expand_path("Gemfile_#{filename}.rb", "benchmarks/ractor/gemfiles")
+  raise "Gemfile #{filename} doesn't exist" unless  File.exist?(filename)
+  use_inline_gemfile do
+    gem "fiddle" # for maxrss
+    instance_eval File.read(filename), filename, 1
+  end
+end


This seems unused, so probably can be removed

eregon · 2025-10-19T09:10:22Z

One more thought is the additions of Ractor.make_shareable even when not using the ractor harness means those benchmarks will fail to run on TruffleRuby (which doesn't define Ractor).
I think a simple solution there is to just define Ractor.make_shareable as noop or freeze in harness/harness-common.rb.
To make it slightly cleaner we could define a top-level make_shareable method so we don't have to define the Ractor constant on Rubies which don't define it.
We can also deal with that after this PR.

eregon · 2025-10-19T09:11:25Z

I went through the diff, LGTM in general.

luke-gruber marked this pull request as draft August 8, 2025 18:31

luke-gruber force-pushed the ractor-harness-luke branch 2 times, most recently from 9a48fe5 to 607cd5b Compare August 12, 2025 15:39

etiennebarrie force-pushed the ractor-harness-luke branch from 910d5f9 to c1c1654 Compare August 13, 2025 14:31

luke-gruber force-pushed the ractor-harness-luke branch from 40880ac to fe13558 Compare October 10, 2025 16:21

eightbitraptor and others added 12 commits October 15, 2025 19:36

Make most benchmarks Ractor compatible

cdededd

Co-Authored-By: Luke Gruber <[email protected]> Co-Authored-By: Étienne Barrié <[email protected]>

Make 30k_methods benchmark Ractor compatible

b429d1a

Committed seperately so the noisy diff can be easily excluded from reviews if needed.

Add Ractor microbenchmarks and harness

39c0c9d

These can be run as follows: ruby -Iharness-ractor/harness.rb benchmarks-ractor/json_parse_float/benchmark.rb Co-Authored-By: Luke Gruber <[email protected]>

Update harness API to match harness-ractor

3c21aea

Co-Authored-By: Luke Gruber <[email protected]>

Add knucleotide benchmark from the benchmark game

f5a281b

2 implementations, one using Process.fork, and one using Ractors - for comparison

Add Optcarrot Ractor benchmark

f0a1ec9

This is a copy of the Optcarrot benchmark from benchmarks/optcarrot, but modified to be compatible with multiple Ractors

Remove the need to specify the full harness path

b31336e

Instead -Iharness-name can be used

Run Ractor tests in CI

1eca4f1

Also make sure that the benchmarks list test is up to date

Update documentation for new Ractor benchmarks

78f97ef

eightbitraptor force-pushed the ractor-harness-luke branch from 7841bf9 to 78f97ef Compare October 15, 2025 18:43

eightbitraptor changed the title ~~[WIP] Ractor harness~~ Introduce Ractor benchmarks and harness Oct 15, 2025

eightbitraptor marked this pull request as ready for review October 15, 2025 19:18

eightbitraptor requested a review from k0kubun October 15, 2025 19:18

eightbitraptor self-assigned this Oct 15, 2025

Add name_filters to --category=ractor benchmarks

a1dbe39

eightbitraptor requested review from rwstauner and tenderlove October 15, 2025 21:23

Remove useless files in benchmarks-ractor/optcarrot

ccdb04c

eregon reviewed Oct 19, 2025

View reviewed changes

k0kubun approved these changes Oct 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce Ractor benchmarks and harness #375

Introduce Ractor benchmarks and harness #375

luke-gruber commented Aug 8, 2025 •

edited

Loading

Uh oh!

tenderlove commented Oct 15, 2025

Uh oh!

luke-gruber commented Oct 15, 2025 •

edited

Loading

Uh oh!

tenderlove commented Oct 15, 2025

Uh oh!

eregon commented Oct 15, 2025 •

edited

Loading

Uh oh!

k0kubun commented Oct 16, 2025

Uh oh!

eightbitraptor commented Oct 16, 2025

Uh oh!

eightbitraptor commented Oct 18, 2025

Uh oh!

eregon Oct 19, 2025

Uh oh!

eregon commented Oct 19, 2025 •

edited

Loading

Uh oh!

eregon Oct 19, 2025

Uh oh!

eregon commented Oct 19, 2025

Uh oh!

eregon commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	ENV["YJIT_BENCH_RACTOR_HARNESS"] = "1"
	ENV["RUBY_BENCH_RACTOR_HARNESS"] = "1"

Introduce Ractor benchmarks and harness #375

Are you sure you want to change the base?

Introduce Ractor benchmarks and harness #375

Conversation

luke-gruber commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Introduce a set of Ractor specific performance benchmarks

2. Adapt existing benchmarks to be Ractor compatible

3. Harness chain.

Uh oh!

tenderlove commented Oct 15, 2025

Uh oh!

luke-gruber commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tenderlove commented Oct 15, 2025

Uh oh!

eregon commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k0kubun commented Oct 16, 2025

Uh oh!

eightbitraptor commented Oct 16, 2025

Uh oh!

eightbitraptor commented Oct 18, 2025

Uh oh!

eregon Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

eregon commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eregon Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

eregon commented Oct 19, 2025

Uh oh!

eregon commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

luke-gruber commented Aug 8, 2025 •

edited

Loading

luke-gruber commented Oct 15, 2025 •

edited

Loading

eregon commented Oct 15, 2025 •

edited

Loading

eregon commented Oct 19, 2025 •

edited

Loading