Benches

[vnermolaev]

Add a Maiko actor-broker-actor throughput benchmark.
Partially addresses #101.

To run Maiko throughput bench analysis

just bench-a2a

Remark.

I also added similar benchmarks comparing the triple maiko-actix-ractor. Maiko performs significantly worse. These benches are currently local but I can push them too, if required.

Absolute throughput
╭─────────────┬───────────────┬───────────┬───────────┬───────────┬───────────╮
│   Framework │   N \ Payload │       32B │      256B │      1KiB │      4KiB │
├─────────────┼───────────────┼───────────┼───────────┼───────────┼───────────┤
│       maiko │         1,000 │   810,106 │   820,823 │   765,626 │   707,549 │
│       maiko │        10,000 │   824,003 │   817,412 │   792,755 │   733,935 │
│       maiko │       100,000 │   824,626 │   809,803 │   801,395 │   695,946 │
│       actix │         1,000 │ 4,516,609 │ 4,683,393 │ 4,075,874 │ 3,358,159 │
│       actix │        10,000 │ 4,746,198 │ 4,701,140 │ 4,244,897 │ 3,256,094 │
│       actix │       100,000 │ 4,662,745 │ 4,699,904 │ 4,257,104 │ 3,466,150 │
│      ractor │         1,000 │ 3,236,237 │ 3,282,361 │ 2,806,116 │ 2,069,979 │
│      ractor │        10,000 │ 3,130,334 │ 3,088,898 │ 2,708,044 │ 2,003,838 │
│      ractor │       100,000 │ 3,047,928 │ 3,027,213 │ 2,684,306 │ 1,826,763 │
╰─────────────┴───────────────┴───────────┴───────────┴───────────┴───────────╯

Unit: messages/sec

Relative throughput vs maiko
╭─────────────┬───────────────┬───────┬────────┬────────┬────────╮
│   Framework │   N \ Payload │   32B │   256B │   1KiB │   4KiB │
├─────────────┼───────────────┼───────┼────────┼────────┼────────┤
│       actix │         1,000 │ 5.58x │  5.71x │  5.32x │  4.75x │
│       actix │        10,000 │ 5.76x │  5.75x │  5.35x │  4.44x │
│       actix │       100,000 │ 5.65x │  5.80x │  5.31x │  4.98x │
│      ractor │         1,000 │ 3.99x │  4.00x │  3.67x │  2.93x │
│      ractor │        10,000 │ 3.80x │  3.78x │  3.42x │  2.73x │
│      ractor │       100,000 │ 3.70x │  3.74x │  3.35x │  2.62x │
╰─────────────┴───────────────┴───────┴────────┴────────┴────────╯
Unit: multiplier (higher is better)

Why

(IMO) Main reasons:

• Maiko bench measures Producer -> Broker -> Consumer with topic routing and envelope handling, not direct actor-to-actor.
• Maiko uses blocking/backpressure semantics and await on each send.
• Actix and Ractor benches are direct producer-to-consumer enqueue loops with non-blocking send APIs.

So we are comparing different semantics:

• Maiko: brokered + backpressure-aware
• Actix: direct mailbox + backpressure (I created a sort of backpressure but requires checking)
• Ractor: direct mailbox + no backpressure (internally unbounded mpsc channel)

This is an oranges-to-apples situation. Please share your thoughts @ddrcode

[ddrcode]

This is really a good piece of work, thank you. I agree that comparing Maiko's performance with Ractor/Actix is - as you said - comparing apples to oranges: different philosophy, no direct messaging etc. Maiko can't be competitive here due to two-step delivery via broker.. But it's good to have that comparison anyway. In practical scenarios I hope the multi-broker solution (0.4.0) will make some noticeable improvements (especially for OverflowStrategy::Block), but still - in its own domain. We won't reach 1m events/s. And it's not a goal. Maiko is mainly optimized for ergonomics, then detached architecture; not for pure performance.

Regarding your tests - interesting idea with playing with various payload sizes. But in reality we mostly test here allocations on ctx.send in step function. Post step, each event is wrapped by Arc, so cloning is size-agnostic.

Please resolve the conflicts after the previous PR merge and it will be good to go.

[vnermolaev]

Regarding your tests - interesting idea with playing with various payload sizes. But in reality we mostly test here allocations on ctx.send in step function. Post step, each event is wrapped by Arc, so cloning is size-agnostic.

I agree with you, however, it still seems useful to me to showcase some performance degradation given the message size, which is apparently the case for actix and ractor, too. Thus I propose to keep that. Let me know if you disagree, it's trivial to drop that.

I will aslo add the local performance ractor/actix benchmarks to make these numbers reproducable.

[vnermolaev]

Please resolve the conflicts after the previous PR merge and it will be good to go.

I re-based, added all benches and a comparison note.
I believe it makes sense to have actix and ractor benches here, because when someone sees another actor framework, the next natural question is "how does it compare to the others performance-wise?".

[ddrcode]

I completely agree - let's keep them. And adding the the ractor/actix tests, makes sense. Thanks

[ddrcode]

All good, thank you for another contribution @vnermolaev