About benchmark: is ugrep's speed beats ripgrep in most benchmarks?

[douglarek]

For an up-to-date performance comparison of the latest ugrep, please see the ugrep performance benchmarks. Ugrep is faster than GNU grep, Silver Searcher, ack, sift. Ugrep's speed beats ripgrep in most benchmarks.

I saw that the ugrep project readme claims to be faster than rg in most cases. I also saw that rg has an official test suite (https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/benchsuite). Taking advantage of some free time, I ran this suite's tests on Linux source code and compiled binaries, and found that ugrep was slower than rg in most cases. I'm not an expert in this area, and I'd like to know your opinion on these test results.

Here are my test steps:

1. git clone https://github.com/BurntSushi/ripgrep.git --depth 1
2. cd ripgrep/benchsuite && git clone https://github.com/gregkh/linux --depth 1 -b linux-6.17.y linux
3. ./benchsuite --download linux
4. ./benchsuite "linux.*"

Result:

linux_literal_default (pattern: PM_RESUME)
------------------------------------------
rg*        0.106 +/- 0.002 (lines: 38)*
ag         0.555 +/- 0.002 (lines: 38)
git grep   0.314 +/- 0.004 (lines: 38)
ugrep      0.194 +/- 0.003 (lines: 38)
grep       1.246 +/- 0.006 (lines: 38)

linux_literal (pattern: PM_RESUME)
----------------------------------
rg*         0.106 +/- 0.004 (lines: 38)*
rg (mmap)   0.564 +/- 0.008 (lines: 38)
ag (mmap)   0.567 +/- 0.004 (lines: 38)
git grep    0.314 +/- 0.003 (lines: 38)
ugrep       0.396 +/- 0.006 (lines: 38)

linux_literal_casei (pattern: PM_RESUME)
----------------------------------------
rg*         0.115 +/- 0.003 (lines: 612)*
rg (mmap)   0.556 +/- 0.007 (lines: 612)
ag (mmap)   0.563 +/- 0.004 (lines: 612)
git grep    0.310 +/- 0.000 (lines: 612)
ugrep       0.420 +/- 0.002 (lines: 612)

linux_re_literal_suffix (pattern: [A-Z]+_RESUME)
------------------------------------------------
rg*        0.108 +/- 0.001 (lines: 2403)*
ag         0.645 +/- 0.002 (lines: 2403)
git grep   1.380 +/- 0.006 (lines: 2403)
ugrep      0.399 +/- 0.003 (lines: 2403)

linux_word (pattern: PM_RESUME)
-------------------------------
rg*        0.109 +/- 0.003 (lines: 9)*
ag         0.558 +/- 0.007 (lines: 9)
git grep   0.313 +/- 0.000 (lines: 9)
ugrep      0.402 +/- 0.005 (lines: 9)

linux_unicode_greek (pattern: \p{Greek})
----------------------------------------
rg*     0.272 +/- 0.008 (lines: 134)*
ugrep   0.443 +/- 0.006 (lines: 134)

linux_unicode_greek_casei (pattern: \p{Greek})
----------------------------------------------
rg*     0.277 +/- 0.011 (lines: 312)*
ugrep   0.444 +/- 0.008 (lines: 134)

linux_unicode_word (pattern: \wAh)
----------------------------------
rg                 0.112 +/- 0.004 (lines: 286)
rg (ASCII)*        0.108 +/- 0.003 (lines: 272)*
ag (ASCII)         0.593 +/- 0.004 (lines: 272)
git grep           7.950 +/- 0.016 (lines: 286)
git grep (ASCII)   2.743 +/- 0.001 (lines: 272)
ugrep              0.389 +/- 0.004 (lines: 286)
ugrep (ASCII)      0.371 +/- 0.004 (lines: 272)

linux_no_literal (pattern: \w{5}\s+\w{5}\s+\w{5}\s+\w{5}\s+\w{5})
-----------------------------------------------------------------
rg                 0.277 +/- 0.005 (lines: 832)
rg (ASCII)*        0.272 +/- 0.013 (lines: 831)*
ag (ASCII)         0.986 +/- 0.007 (lines: 1295)
git grep           15.988 +/- 0.080 (lines: 832)
git grep (ASCII)   3.874 +/- 0.020 (lines: 831)
ugrep              7.594 +/- 0.695 (lines: 832)
ugrep (ASCII)      2.057 +/- 0.004 (lines: 831)

linux_alternates (pattern: ERR_SYS|PME_TURN_OFF|LINK_REQ_RST|CFG_BME_EVT)
-------------------------------------------------------------------------
rg*        0.115 +/- 0.003 (lines: 174)*
ag         0.619 +/- 0.006 (lines: 174)
git grep   0.489 +/- 0.004 (lines: 174)
ugrep      0.421 +/- 0.001 (lines: 174)

linux_alternates_casei (pattern: ERR_SYS|PME_TURN_OFF|LINK_REQ_RST|CFG_BME_EVT)
-------------------------------------------------------------------------------
rg*        0.135 +/- 0.004 (lines: 290)*
ag         0.731 +/- 0.004 (lines: 290)
git grep   1.098 +/- 0.005 (lines: 290)
ugrep      0.419 +/- 0.004 (lines: 290) 

Version:

ugrep: 7.5.0
rg: 15.1.0
ag: 2.2.0

My computer hardware:

OS: Gentoo Linux x86_64
Kernel: Linux 6.17.5-gentoo-dist
CPU: AMD Ryzen 5 5600X (12) @ 4.65 GHz
Memory: 6.08 GiB / 31.26 GiB (19%)

[GwynethLlewelyn]

Of course, every benchmark works under a certain set of assumptions... usually, those that show that 'our' version runs faster 🤣

It should be interesting to see what results you get with the 'official' benchmarks for ugrep. Do they match with Genivia's own numbers? Note that the claim that ugrep is always faster than the others does not hold true on the ugrep's own benchmarks: it performs slightly better in some cases, and slightly worse in others, but the difference between ugrep and rg is minimal — not three times slower on average, as your own benchmarking seems to show.

Granted, ugrep is allegedly more closely aligned with what the original grep's parameters actually do, i.e., out of the box, the same parameters are allegedly providing the same results. rg (deliberately) does a lot of short-circuiting and extrapolation to real-world scenarios where 'strict grep compatibility' might be less important than 'raw speed'. That's why, for the benchmarks, Genivia opted to add a few extra parameters, to make ugrep less compliant and more performant, and thus match rg's speed as closely as possible.

One thing is obvious: each grep-clone programmer has their own notion of what is most important to them, and a certain degree of awareness on how to optimise just that (possibly at the cost of excluding other important things). One thing I notice in both benchmarks is that none of them are at least concerned about how much memory they take! Granted, on current-generation machines, this is not a concern these days; but one might reasonably ask which is the faster tool when running on, say, an embedded device with little free memory? Which one is more optimised to run under a virtual machine with SSDs, as opposed to running on bare metal with rotating disks in a RAID-5 array? Benchmarks only go so far as to show how fast a tool performs under a specific 'lab' scenario.

For example, I like rg's benchmarks done on the (compiled) Linux kernel, since searching across tens of thousands of files and millions (?) of lines of code is a pretty realistic scenario. Similarly, to stress strict Unicode compatibility, it was a nice idea to do complex searches on Russian subtitles — but how well does ag perform using Devanagari or the Tibetan uchen abugida script?

At the end of the day, it's always a question of use-cases...