Multi-Client Runner with pubsub-sub-bench Integration

[fcostaoliveira]

This PR introduces comprehensive multi-client execution capabilities to the redis-benchmarks-specification client runner, enabling simultaneous execution of multiple benchmark tools with unified results collection.

It includes a pubsub example by running publishers and subscribers simultaneously, providing complete visibility into both sides of the pubsub performance equation.

Here's a sample run:

$ poetry run redis-benchmarks-spec-client-runner --test memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers.yml --override-memtier-test-time 30 --container-timeout-buffer 3
/home/fco/.cache/pypoetry/virtualenvs/redis-benchmarks-specification-pNKGADFH-py3.10/lib/python3.10/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/home/fco/.cache/pypoetry/virtualenvs/redis-benchmarks-specification-pNKGADFH-py3.10/lib/python3.10/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "class": algorithms.TripleDES,
2025-06-28 20:02:50 INFO redis-benchmarks-spec-client-runner (solely client) 0.1.272
2025-06-28 20:02:50 INFO Using test-suites folder dir /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites
2025-06-28 20:02:50 INFO Running specific benchmark in 1 files: ['/home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers.yml']
2025-06-28 20:02:50 INFO There are a total of 1 test-suites in folder /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites
2025-06-28 20:02:50 INFO Test /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers.yml
2025-06-28 20:02:50 INFO Overriding memtier benchmark --test-time to 30 seconds
2025-06-28 20:02:50 INFO Running the benchmark specs.
2025-06-28 20:02:50 INFO Loading default specifications from file: /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites/defaults.yml
2025-06-28 20:02:50 INFO Found RedisTimeSeries default metrics specification. Will include the following metrics on all benchmarks $."BEST RUN RESULTS".Totals."Ops/sec" $."BEST RUN RESULTS".Totals."Latency" $."BEST RUN RESULTS".Totals."Misses/sec" $."BEST RUN RESULTS".Totals."Percentile Latencies"."p50.00" $."BEST RUN RESULTS".Totals."Percentile Latencies"."p99.00" $."WORST RUN RESULTS".Totals."Ops/sec" $."WORST RUN RESULTS".Totals."Latency" $."WORST RUN RESULTS".Totals."Misses/sec" $."WORST RUN RESULTS".Totals."Percentile Latencies"."p50.00" $."WORST RUN RESULTS".Totals."Percentile Latencies"."p99.00" $."AGGREGATED AVERAGE RESULTS (3 runs)".Totals."Ops/sec" $."AGGREGATED AVERAGE RESULTS (3 runs)".Totals."Latency" $."AGGREGATED AVERAGE RESULTS (3 runs)".Totals."Misses/sec" $."AGGREGATED AVERAGE RESULTS (3 runs)".Totals."Percentile Latencies"."p50.00" $."AGGREGATED AVERAGE RESULTS (3 runs)".Totals."Percentile Latencies"."p99.00" $."ALL STATS".Totals."Ops/sec" $."ALL STATS".Totals."Latency" $."ALL STATS".Totals."Misses/sec" $."ALL STATS".Totals."Percentile Latencies"."p50.00" $."ALL STATS".Totals."Percentile Latencies"."p99.00" $."MessageRate"
2025-06-28 20:02:50 INFO Found RedisTimeSeries default time metric specification. Will use the following JSON path to retrieve the test time $."ALL STATS".Runtime."Start time"
  0%|                                                                                                                                                                                        | 0/1 [00:00<?, ?it/s]2025-06-28 20:02:50 INFO Benchmark required memory: 2147483648 Bytes
2025-06-28 20:02:50 INFO  Using total system memory as max 33332305920
2025-06-28 20:02:50 INFO Resetting commmandstats for shard 0
2025-06-28 20:02:50 INFO Test memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers priority (23) is within the priority limit [0,100000]
2025-06-28 20:02:50 INFO Benchmark used memory at start of benchmark: 1g
2025-06-28 20:02:50 INFO Checking if there is a keyspace check being enforced
2025-06-28 20:02:50 INFO Will store benchmark json output to local file oss-standalone-2025-06-28-19-02-50-NA-memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers.json
2025-06-28 20:02:50 INFO Running test with 2 client configurations
2025-06-28 20:02:50 INFO Running multiple client configurations simultaneously
2025-06-28 20:02:50 INFO Preparing the benchmark parameters. ['memtier_benchmark', '--json-out-file', 'benchmark_output_0.json', '--port', '6379', '--server', 'localhost'].
2025-06-28 20:02:50 INFO Client 0: Set container timeout to 33s (test-time: 30s + 3s buffer)
2025-06-28 20:02:50 INFO Starting client 0 with docker image redislabs/memtier_benchmark:edge (cpuset=0,1,2,3,4,5) with args: memtier_benchmark --json-out-file benchmark_output_0.json --port 6379 --server localhost --test-time=30 --key-prefix "channel-" --pipeline 1 -d 128 --key-maximum 100 --command "PUBLISH __key__ __data__" --command-key-pattern="R" -c 50 -t 4 --hide-histogram
2025-06-28 20:02:51 INFO Preparing pubsub-sub-bench parameters: ['-json-out-file', 'benchmark_output_1.json', '-host', 'localhost', '-port', '6379', '-resp', '2']
2025-06-28 20:02:51 INFO Applied test-time override: 30s
2025-06-28 20:02:51 INFO Client 1: Set container timeout to 33s (test-time: 30s + 3s buffer)
2025-06-28 20:02:51 INFO Starting client 1 with docker image filipe958/pubsub-sub-bench:latest (cpuset=0,1,2,3,4,5) with args: -json-out-file benchmark_output_1.json -host localhost -port 6379 -resp 2 -test-time 30 -clients 100 -channel-minimum 1 -channel-maximum 100 -subscriber-prefix "channel-" -mode subscribe  -subscribers-per-channel 1
2025-06-28 20:02:51 INFO Waiting for 2 containers to complete...
2025-06-28 20:03:22 INFO Client 0 completed successfully with exit code: {'StatusCode': 0}
2025-06-28 20:03:23 INFO Client 1 completed successfully with exit code: {'StatusCode': 0}
2025-06-28 20:03:23 INFO Successfully completed 2 client configurations
2025-06-28 20:03:23 INFO Successfully read memtier JSON output from client 0
2025-06-28 20:03:23 INFO Successfully read JSON output from client 0 (memtier_benchmark)
2025-06-28 20:03:23 INFO Successfully read pubsub-sub-bench JSON output from client 1
2025-06-28 20:03:23 INFO Successfully read JSON output from client 1 (pubsub-sub-bench)
2025-06-28 20:03:23 INFO Using merged JSON results from memtier and pubsub-sub-bench clients
2025-06-28 20:03:23 INFO Completed 2 client configurations
2025-06-28 20:03:23 INFO Benchmark duration 32 secs.
2025-06-28 20:03:23 WARNING Benchmark duration of 32 secs is bellow the considered minimum duration for a stable run (60 secs).
2025-06-28 20:03:23 INFO Printing client tool stdout output
2025-06-28 20:03:23 INFO Benchmark used memory at end of benchmark: 1g
2025-06-28 20:03:23 INFO Converting redis-benchmark output to json. Storing it in: oss-standalone-2025-06-28-19-02-50-NA-memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers.json
2025-06-28 20:03:23 INFO Using aggregated JSON results from multi-client execution
# Results for memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers test-case on oss-standalone topology
|                 Metric JSON Path                 |Metric Value|
|--------------------------------------------------|-----------:|
|"ALL STATS".Totals."Ops/sec"                      |  136227.010|
|"ALL STATS".Totals."Latency"                      |       1.468|
|"ALL STATS".Totals."Misses/sec"                   |       0.000|
|"ALL STATS".Totals."Percentile Latencies"."p50.00"|       1.255|
|"ALL STATS".Totals."Percentile Latencies"."p99.00"|       2.815|
|MessageRate                                       |  129736.955|
2025-06-28 20:03:23 INFO Using datapoint_time_ms: 1751137370957. git_hash=None, git_branch=None, git_version=NA. gh_org=redis, gh_repo=redis
2025-06-28 20:03:23 WARNING there was no git hash information to push data brokedown by hash
2025-06-28 20:03:23 INFO Extending the by.hash None timeseries (0) with version info NA
2025-06-28 20:03:23 INFO Total of 6 cleaned metrics: [['"ALL STATS".Totals."Ops/sec"', 'Totals', 'ALL_STATS.Totals.Ops/sec', 136227.01, {}, False], ['"ALL STATS".Totals."Latency"', 'Totals', 'ALL_STATS.Totals.Latency', 1.468, {}, False], ['"ALL STATS".Totals."Misses/sec"', 'Totals', 'ALL_STATS.Totals.Misses/sec', 0.0, {}, False], ['"ALL STATS".Totals."Percentile Latencies"."p50.00"', 'Percentile Latencies', 'ALL_STATS.Totals.Percentile_Latencies.p50.00', 1.255, {}, False], ['"ALL STATS".Totals."Percentile Latencies"."p99.00"', 'Percentile Latencies', 'ALL_STATS.Totals.Percentile_Latencies.p99.00', 2.815, {}, False], ['"MessageRate"', '$', 'MessageRate', 129736.9554276396, {}, False]]
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/ALL_STATS.Totals.Ops/sec to time_series_dict.
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/ALL_STATS.Totals.Latency to time_series_dict.
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/ALL_STATS.Totals.Misses/sec to time_series_dict.
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/ALL_STATS.Totals.Percentile_Latencies.p50.00 to time_series_dict.
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/ALL_STATS.Totals.Percentile_Latencies.p99.00 to time_series_dict.
2025-06-28 20:03:23 INFO Adding timeserie named ci.benchmarks.redislabs/by.version/fco-laptop/redis/redis/memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers/NA/fco-laptop/oss-standalone/NA/MessageRate to time_series_dict.
2025-06-28 20:03:23 WARNING there was no git BRANCH information to push data brokedown by BRANCH
2025-06-28 20:03:23 INFO Collecting memory metrics
2025-06-28 20:03:23 INFO Adding a total of 38 server side metrics collected at the end of benchmark
2025-06-28 20:03:23 INFO Collecting commandstat metrics
2025-06-28 20:03:23 INFO Adding a total of 30 server side metrics collected at the end of benchmark
2025-06-28 20:03:23 INFO Tearing down setup
2025-06-28 20:03:23 INFO Removing temporary client dir /home/fco/tmpsa7nign3
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:32<00:00, 32.60s/it]

# Redis Server Information
|    Property    |            Value             |
|----------------|------------------------------|
|Redis Version   |255.255.255                   |
|Redis Git SHA1  |05eaf6e4                      |
|Redis Git Dirty |0                             |
|Redis Build ID  |9ff8e77d3d80abcd              |
|Redis Mode      |standalone                    |
|OS              |Linux 5.19.0-46-generic x86_64|
|Arch Bits       |64                            |
|GCC Version     |11.3.0                        |
|Process ID      |153953                        |
|TCP Port        |6379                          |
|Uptime (seconds)|9984                          |
2025-06-28 20:03:23 INFO Displayed Redis server information: Redis 255.255.255
# Results for entire test-suite
|                                      Test Name                                       |                 Metric JSON Path                 |Metric Value|
|--------------------------------------------------------------------------------------|--------------------------------------------------|-----------:|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|"ALL STATS".Totals."Ops/sec"                      |  136227.010|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|"ALL STATS".Totals."Latency"                      |       1.468|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|"ALL STATS".Totals."Misses/sec"                   |       0.000|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|"ALL STATS".Totals."Percentile Latencies"."p50.00"|       1.255|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|"ALL STATS".Totals."Percentile Latencies"."p99.00"|       2.815|
|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|MessageRate                                       |  129736.955|