Benchmark improvements

[Anilm3]

• Remove random object generation
• Change tests so that the same input is evaluated on all iterations
• Allow the ability to specify common and test-specific fixtures
• Add test-specific fixtures (eval.valid)

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (15a165f) 83.28% compared to head (4a04f38) 83.28%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #247   +/-   ##
=======================================
  Coverage   83.28%   83.28%           
=======================================
  Files         106      106           
  Lines        4177     4177           
  Branches     1915     1915           
=======================================
  Hits         3479     3479           
  Misses        279      279           
  Partials      419      419           

Flag	Coverage Δ
waf_test_none	83.18% <ø> (ø)
waf_test_sse2	83.25% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-18 14:36:32

Comparing candidate commit 0fae74c in PR branch anilm3/benchmark-continuation with baseline commit 15a165f in branch master.

Some scenarios are present only in baseline or only in candidate runs. If you didn't create or remove some scenarios in your branch, this maybe a sign of crashed benchmarks 💥💥💥
Check Gitlab CI job log to find if any benchmark has crashed.

Scenarios present only in baseline:

• bool_equals_matcher.random.1x1.128
• exact_match_matcher.random.1x256.128
• float_equals_matcher.random.1x1.1024
• lowercase_transformer.random.20x1.128
• ip_match_matcher.random.5x1.128
• remove_nulls_transformer.random.1x16.128
• remove_nulls_transformer.random.1x1.1024
• remove_nulls_transformer.random.1x128.128
• remove_nulls_transformer.random.1x64.128
• string_equals_matcher.random.1x1.128
• signed_equals_matcher.random.10x1.128
• float_equals_matcher.random.1x1.4096
• regex_match_matcher.case_insensitive_flag.random.1x128.128
• string_equals_matcher.random.20x1.128
• regex_match_matcher.case_insensitive_option.random.1x1.4096
• bool_equals_matcher.random.1x128.128
• exact_match_matcher.random.1x1.1024
• regex_match_matcher.random.1x1.512
• regex_match_matcher.case_insensitive_flag.random.1x1.512
• unsigned_equals_matcher.random.1x256.128
• ruleset_1.10.0.random.5x1.128
• phrase_match_matcher.random.1x1.1024
• phrase_match_matcher.random.1x256.128
• is_xss_matcher.random.1x128.128
• bool_equals_matcher.random.1x256.128
• bool_equals_matcher.random.10x1.128
• is_sqli_matcher.random.1x64.128
• ip_match_matcher.random.20x1.128
• regex_match_matcher.lowercase_transformer.random.1x1.1024
• phrase_match_matcher.random.1x1.4096
• regex_match_matcher.random.5x1.128
• remove_nulls_transformer.random.1x1.128
• exact_match_matcher.random.1x16.128
• unsigned_equals_matcher.random.1x1.4096
• is_xss_matcher.random.1x1.128
• remove_nulls_transformer.random.1x1.4096
• lowercase_transformer.random.1x1.1024
• unsigned_equals_matcher.random.1x128.128
• float_equals_matcher.random.1x128.128
• remove_nulls_transformer.random.10x1.128
• is_sqli_matcher.random.1x1.512
• regex_match_matcher.case_insensitive_flag.random.1x64.128
• regex_match_matcher.random.1x1.1024
• ip_match_matcher.random.1x1.128
• ip_match_matcher.random.1x128.128
• phrase_match_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_flag.random.10x1.128
• regex_match_matcher.case_insensitive_option.random.5x1.128
• bool_equals_matcher.random.1x1.1024
• exact_match_matcher.random.1x1.512
• float_equals_matcher.random.1x16.128
• regex_match_matcher.random.1x16.128
• bool_equals_matcher.random.20x1.128
• exact_match_matcher.random.10x1.128
• exact_match_matcher.random.5x1.128
• ip_match_matcher.random.1x16.128
• string_equals_matcher.random.1x128.128
• ip_match_matcher.random.10x1.128
• phrase_match_matcher.random.1x1.128
• ruleset_1.10.0.random.1x1.512
• regex_match_matcher.case_insensitive_flag.random.1x1.4096
• remove_nulls_transformer.random.1x1.512
• regex_match_matcher.case_insensitive_flag.random.1x1.1024
• regex_match_matcher.case_insensitive_flag.random.20x1.128
• float_equals_matcher.random.1x64.128
• ip_match_matcher.random.1x64.128
• string_equals_matcher.random.10x1.128
• signed_equals_matcher.random.1x1.4096
• unsigned_equals_matcher.random.1x1.512
• unsigned_equals_matcher.random.1x16.128
• bool_equals_matcher.random.5x1.128
• string_equals_matcher.random.1x1.4096
• unsigned_equals_matcher.random.10x1.128
• unsigned_equals_matcher.random.20x1.128
• ruleset_1.10.0.random.1x128.128
• is_xss_matcher.random.20x1.128
• ruleset_1.10.0.random.1x16.128
• signed_equals_matcher.random.20x1.128
• regex_match_matcher.lowercase_transformer.random.1x16.128
• float_equals_matcher.random.5x1.128
• signed_equals_matcher.random.1x64.128
• is_sqli_matcher.random.1x128.128
• regex_match_matcher.lowercase_transformer.random.20x1.128
• phrase_match_matcher.random.5x1.128
• lowercase_transformer.random.10x1.128
• regex_match_matcher.random.1x128.128
• regex_match_matcher.random.10x1.128
• is_sqli_matcher.random.20x1.128
• is_sqli_matcher.random.5x1.128
• lowercase_transformer.random.1x1.128
• regex_match_matcher.case_insensitive_option.random.20x1.128
• float_equals_matcher.random.1x1.512
• signed_equals_matcher.random.1x1.128
• regex_match_matcher.case_insensitive_option.random.1x1.1024
• string_equals_matcher.random.1x16.128
• is_xss_matcher.random.1x64.128
• ruleset_1.10.0.random.1x1.128
• string_equals_matcher.random.1x1.512
• signed_equals_matcher.random.5x1.128
• exact_match_matcher.random.1x1.128
• regex_match_matcher.lowercase_transformer.random.1x256.128
• regex_match_matcher.lowercase_transformer.random.1x1.4096
• is_xss_matcher.random.10x1.128
• is_xss_matcher.random.1x16.128
• ip_match_matcher.random.1x1.4096
• lowercase_transformer.random.5x1.128
• regex_match_matcher.lowercase_transformer.random.1x128.128
• unsigned_equals_matcher.random.1x64.128
• bool_equals_matcher.random.1x1.4096
• regex_match_matcher.random.1x1.4096
• regex_match_matcher.random.1x256.128
• lowercase_transformer.random.1x16.128
• lowercase_transformer.random.1x256.128
• regex_match_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_option.random.1x64.128
• regex_match_matcher.case_insensitive_option.random.1x1.512
• float_equals_matcher.random.10x1.128
• ip_match_matcher.random.1x256.128
• is_xss_matcher.random.1x1.4096
• is_sqli_matcher.random.1x1.1024
• float_equals_matcher.random.1x1.128
• is_sqli_matcher.random.10x1.128
• is_xss_matcher.random.5x1.128
• regex_match_matcher.case_insensitive_option.random.1x16.128
• unsigned_equals_matcher.random.1x1.1024
• bool_equals_matcher.random.1x1.512
• is_xss_matcher.random.1x1.1024
• unsigned_equals_matcher.random.1x1.128
• ruleset_1.10.0.random.1x256.128
• remove_nulls_transformer.random.20x1.128
• regex_match_matcher.case_insensitive_option.random.1x128.128
• regex_match_matcher.case_insensitive_flag.random.1x1.128
• exact_match_matcher.random.20x1.128
• signed_equals_matcher.random.1x128.128
• string_equals_matcher.random.1x64.128
• is_sqli_matcher.random.1x1.4096
• regex_match_matcher.lowercase_transformer.random.5x1.128
• unsigned_equals_matcher.random.5x1.128
• lowercase_transformer.random.1x1.4096
• remove_nulls_transformer.random.5x1.128
• regex_match_matcher.random.1x1.128
• is_sqli_matcher.random.1x256.128
• lowercase_transformer.random.1x1.512
• regex_match_matcher.random.20x1.128
• regex_match_matcher.case_insensitive_flag.random.1x256.128
• is_sqli_matcher.random.1x16.128
• is_sqli_matcher.random.1x1.128
• ip_match_matcher.random.1x1.512
• exact_match_matcher.random.1x1.4096
• ruleset_1.10.0.random.1x1.4096
• ruleset_1.10.0.random.10x1.128
• string_equals_matcher.random.1x256.128
• string_equals_matcher.random.5x1.128
• bool_equals_matcher.random.1x64.128
• is_xss_matcher.random.1x1.512
• regex_match_matcher.lowercase_transformer.random.1x1.512
• remove_nulls_transformer.random.1x256.128
• signed_equals_matcher.random.1x16.128
• phrase_match_matcher.random.20x1.128
• regex_match_matcher.lowercase_transformer.random.10x1.128
• signed_equals_matcher.random.1x256.128
• float_equals_matcher.random.20x1.128
• regex_match_matcher.case_insensitive_option.random.10x1.128
• phrase_match_matcher.random.1x128.128
• regex_match_matcher.case_insensitive_flag.random.5x1.128
• signed_equals_matcher.random.1x1.1024
• ruleset_1.10.0.random.1x1.1024
• regex_match_matcher.lowercase_transformer.random.1x1.128
• ip_match_matcher.random.1x1.1024
• float_equals_matcher.random.1x256.128
• bool_equals_matcher.random.1x16.128
• exact_match_matcher.random.1x128.128
• signed_equals_matcher.random.1x1.512
• is_xss_matcher.random.1x256.128
• ruleset_1.10.0.random.1x64.128
• lowercase_transformer.random.1x64.128
• ruleset_1.10.0.random.20x1.128
• lowercase_transformer.random.1x128.128
• phrase_match_matcher.random.10x1.128
• exact_match_matcher.random.1x64.128
• regex_match_matcher.lowercase_transformer.random.1x64.128
• phrase_match_matcher.random.1x16.128
• regex_match_matcher.case_insensitive_flag.random.1x16.128
• string_equals_matcher.random.1x1.1024
• phrase_match_matcher.random.1x1.512
• regex_match_matcher.case_insensitive_option.random.1x1.128
• regex_match_matcher.case_insensitive_option.random.1x256.128

Found 1 performance improvements and 15 performance regressions! Performance is the same for 0 metrics, 1 unstable metrics.

scenario:bool_equals_matcher.valid

• 🟥 execution_time [+38.913ns; +124.477ns] or [+2.798%; +8.951%]

scenario:exact_match_matcher.valid

• 🟥 execution_time [+273.339ns; +328.379ns] or [+16.579%; +19.918%]

scenario:float_equals_matcher.valid

• 🟥 execution_time [+144.587ns; +218.629ns] or [+8.847%; +13.378%]

scenario:ip_match_matcher.valid

• 🟥 execution_time [+190.395ns; +237.305ns] or [+9.058%; +11.289%]

scenario:is_sqli_matcher.valid

• 🟩 execution_time [-387.151ns; -113.803ns] or [-10.990%; -3.230%]

scenario:is_xss_matcher.valid

• 🟥 execution_time [+1.759µs; +1.862µs] or [+93.399%; +98.873%]

scenario:lowercase_transformer.valid

• 🟥 execution_time [+310.582ns; +346.984ns] or [+20.839%; +23.281%]

scenario:phrase_match_matcher.valid

• 🟥 execution_time [+360.515ns; +398.339ns] or [+20.705%; +22.877%]

scenario:regex_match_matcher.case_insensitive_option.valid

• 🟥 execution_time [+319.895ns; +492.119ns] or [+17.712%; +27.248%]

scenario:regex_match_matcher.lowercase_transformer.valid

• 🟥 execution_time [+221.944ns; +405.980ns] or [+11.287%; +20.647%]

scenario:regex_match_matcher.valid

• 🟥 execution_time [+187.240ns; +359.978ns] or [+9.977%; +19.181%]

scenario:remove_nulls_transformer.valid

• 🟥 execution_time [+249.870ns; +289.122ns] or [+15.939%; +18.442%]

scenario:ruleset_1.10.0.valid

• 🟥 execution_time [+787.726µs; +801.281µs] or [+470.175%; +478.266%]

scenario:signed_equals_matcher.valid

• 🟥 execution_time [+261.708ns; +289.650ns] or [+18.584%; +20.569%]

scenario:string_equals_matcher.valid

• 🟥 execution_time [+345.611ns; +370.163ns] or [+24.562%; +26.307%]

scenario:unsigned_equals_matcher.valid

• 🟥 execution_time [+278.045ns; +317.119ns] or [+19.904%; +22.701%]

[Anilm3]

Benchmark is failing since the test name has been changed (so no tests are common with master). This should be resolved post-merge.