Skip to content

BENCHMARKING.md

Latest commit

 

History

History
143 lines (100 loc) · 5.49 KB

BENCHMARKING.md

File metadata and controls

143 lines (100 loc) · 5.49 KB
Raw

Benchmarking

This document defines the current benchmark harness for Janus backend work.

Status: verified against the benchmark targets present in this checkout. It defines how to produce new evidence; it does not endorse older result snapshots as current performance guarantees.

Goals

The benchmark suite is split by engine path so regressions are easier to localize:

  • storage_write: ingest-only storage throughput
  • storage_footprint_benchmark: persistent on-disk storage footprint after ingestion for Janus versus Oxigraph
  • historical_fixed: fixed historical window execution
  • historical_sliding: sliding historical window execution
  • live_injection: live event ingestion to first emitted result
  • hybrid_baseline: Hybrid Coordination Benchmark, live window plus static baseline join
  • janusql_e2e: HTTP/WebSocket Janus-QL first-result latency for historical queries
  • janusql_live_mqtt_e2e: HTTP/WebSocket Janus-QL first-result latency for broker-backed live and hybrid queries

These benchmarks are intended for backend performance work on the current repository state. They are not a substitute for a public performance report.

Run Commands

cargo bench already uses Cargo's optimized bench profile, so there is no extra --release flag to pass here:

cargo bench --bench storage_write
cargo bench --bench historical_fixed
cargo bench --bench historical_sliding
cargo bench --bench live_injection
cargo bench --bench hybrid_baseline
cargo bench --bench janusql_e2e
cargo bench --bench janusql_live_mqtt_e2e

cargo run --release --bin storage_footprint_benchmark -- \
  --event-counts 10000,50000 \
  --iterations 1 \
  --cleanup-runs-after-measurement \
  --output-dir results/storage_footprint_local

To verify that all benchmark targets compile without running them:

cargo bench --no-run

What Each Benchmark Measures

storage_write

Measures StreamingSegmentedStorage::write_rdf_event throughput on fresh storage with large flush thresholds so the benchmark isolates write-path cost rather than background flush cost.

storage_footprint_benchmark

Measures persistent storage footprint, not RSS memory. The benchmark writes the same deterministic CityBench-style RDF events to:

  • Janus's real segmented RDF event-log storage path
  • Oxigraph's persistent on-disk store

It then flushes and closes the store, measures the full recursive directory size in bytes, and records ingest time, events per second, and bytes per event. --include-10m is required before running the 10,000,000-event case. If --cleanup-runs-after-measurement is enabled, each per-run store directory is deleted only after its raw CSV row has been written and flushed.

historical_fixed

Measures a single fixed historical window query over deterministic stored events in ex:graph1.

historical_sliding

Measures a sliding historical window scan over deterministic stored events placed in a recent time range so the windows stay valid relative to the executor's wall-clock based offset logic.

live_injection

Measures the live path from event injection into LiveStreamProcessing to the first emitted result. The harness injects a sentinel event at the end to close remaining windows deterministically.

hybrid_baseline

Measures the Hybrid Coordination Benchmark: live window bindings join against static baseline data already materialized into the live engine. This isolates hybrid join cost without requiring an MQTT broker or the HTTP server.

janusql_e2e

Measures end-to-end Janus-QL latency through the HTTP API:

  • historical_preloaded: register query, start query, subscribe over WebSocket, wait for first result
  • historical_via_replay: start replay, register query, start query, subscribe over WebSocket, wait for first result

This target currently covers historical Janus-QL end-to-end flows. Full live or hybrid HTTP e2e benchmarks are covered separately in janusql_live_mqtt_e2e, because Janus live execution is broker-driven at the API layer.

janusql_live_mqtt_e2e

Measures broker-backed Janus-QL latency through the HTTP API and WebSocket result stream:

  • mqtt_live_first_result: register a live query, start it, subscribe over WebSocket, publish MQTT events, wait for the first live result
  • mqtt_hybrid_first_result: register a hybrid query with historical baseline bootstrap, start it, wait for baseline warm-up, subscribe over WebSocket, publish MQTT events, wait for the first live result

This target requires an MQTT broker reachable from the benchmark process. By default it uses 127.0.0.1:1883 and topic sensors, and it can be overridden with:

  • JANUS_BENCH_MQTT_HOST
  • JANUS_BENCH_MQTT_PORT
  • JANUS_BENCH_MQTT_TOPIC
  • JANUS_BENCH_MQTT_STREAM_URI

Reproducibility Rules

When comparing numbers across commits:

  1. Run on a quiet machine.
  2. Use the same hardware, Rust toolchain, and bench profile.
  3. Avoid running multiple benchmark targets in parallel.
  4. Capture the exact command, branch, and commit SHA with the results.
  5. Record CPU model, RAM, OS version, and whether the system was thermally constrained.

Recommended Workflow

For routine optimization work:

cargo bench --no-run
cargo bench --bench storage_write
cargo bench --bench historical_fixed
cargo bench --bench historical_sliding
cargo bench --bench live_injection
cargo bench --bench hybrid_baseline
cargo bench --bench janusql_e2e
cargo bench --bench janusql_live_mqtt_e2e

For benchmark result reviews, attach the Criterion output directory from target/criterion/.