feat(bench): add benchmark harness to tests and tooling (closes #936)

[arcgod-design]

What

Adds a zero-dependency microbenchmark harness for retrieval pipeline performance, with tests and CLI tooling — closing issue #936.

The existing backend/tests/test_chromadb_benchmark.py measures real chromadb + sentence-transformer latency but can only run when those heavy deps are installed (sentence-transformers alone times out on the local Windows env, requires GPU for reasonable latency). This PR adds a deferred, dependency-free harness that:

• Works in any Python 3.10+ environment via injected callables + injected clock.
• Provides statistical rigor (warmup-runs-discarded, p95/p99, stdev, pooled-summary stats).
• Ships a compare_to_baseline() helper for CI regression gating.
• Persists results to JSON for offline diffing.

Why

Retrieval pipeline regressions are hard to catch in PRs because:

1. The existing benchmark needs the heavy-dep environment, so most contributors skip it.
2. There's no percentile math — only mean — so p95 regressions (the "slow tail" that hurts user experience) pass review invisibly.
3. There's no JSON output — every PR reviews raw stdout numbers, so regressions can slip past a tired reviewer.

benchmark_harness addresses all three:

• run_benchmark(spec, tasks, clock) — runs spec.callable_(task) once per task per runs iteration. Warmup runs are discarded. Per-invocation time.perf_counter() deltas are normalised to ms. Exceptions in callable_ are caught (task marked failed); the loop continues onto the next task so one bug doesn't kill the whole run.
• compare_to_baseline(result, baseline) — emits mean_ratio / median_ratio / p95_ratio and a regressed: bool flag set if any of the three exceeds REGRESSION_FACTOR=1.5. Drop this into a CI gate and you get automatic PR-block on 50%+ regressions without bolting onto the existing workflow's stdout scraping.
• write_report(results, path) — JSON-serialises the whole BenchResult (per-task latencies + pooled summary) so future PRs can diff against the committed baseline.

What changed

Backend utility — backend/utils/benchmark_harness.py (NEW)

Public surface:

@dataclass
class BenchmarkSpec:
    name: str
    callable_: Callable[[dict], Any]   # opaque task -> result
    warmup_runs: int = 0
    runs: int = 10
    timeout_seconds: float = 30.0
    baseline_ms: dict | None = None

@dataclass
class TaskResult:
    task_id: str
    succeeded: bool
    latencies_ms: list[float] = field(default_factory=list)
    error: str | None = None

@dataclass
class BenchResult:
    spec_name: str
    started_at: str
    ended_at: str
    total_runs: int
    fail_count: int
    task_results: list[TaskResult]
    min_ms/max_ms/mean_ms/median_ms/stdev_ms/p95_ms/p99_ms: float

def run_benchmark(spec, tasks, clock=time.perf_counter) -> BenchResult
def compare_to_baseline(result, baseline) -> dict
def write_report(results, path) -> Path

Internals:

• _percentile(sorted_values, p) — linear-interpolation percentile, indistinguishable from statistics.quantiles for monotonic inputs, but defined for p ∈ [0,100] inclusive.
• REGRESSION_FACTOR = 1.5 — the soft cap above which a benchmark is flagged as regressed.

Tooling — scripts/benchmark_harness_cli.py (NEW)

Reads a JSON file describing one or more benchmarks, each with a dotted module:callable import path, runs them, prints a JSON report, optionally writes a report file (--out) and emits baseline-drift comparison (--compare). Exit codes 0/1/2/3 per repo convention.

Example:

$ cat config.json | python scripts/benchmark_harness_cli.py --out bench.json
{
  "benchmarks": [
    {
      "spec_name": "noop",
      "started_at": "...",
      "ended_at": "...",
      "total_runs": 4,
      "fail_count": 0,
      "summary": {"min_ms": 0.0002, "max_ms": 0.001, "mean_ms": 0.0006, ...},
      "task_results": [...]
    }
  ]
}
(report also written to bench.json)
Exit: 0

Tests — backend/tests/test_benchmark_harness.py (NEW, 23 cases)

Class	Cases	Coverage
TestPercentile	6	empty, single, p0=min, p100=max, interpolate-p50, interpolate-p95
TestRunBenchmarkHappyPath	2	latencies recorded, summary stats populated
TestWarmupRuns	1	warmup invocations NOT recorded in latencies_ms
TestFailureHandling	2	exception marks task failed + remaining tasks continue
TestCustomClock	1	deterministic injected clock yields exactly expected ms latencies
TestEmptyTasks	1	zero-task path yields zeroed summary
TestCompareToBaseline	4	None baseline no-op, equal baseline no-op, regression flagged at >1.5x ratio, zero-baseline /0 protected
TestWriteReport	3	writes JSON, creates parent dirs, round-trips through JSON
TestDataclassShape	3	BenchmarkSpec defaults, TaskResult.to_dict, BenchResult.to_dict keys

Verification

$ cd backend && pytest tests/test_benchmark_harness.py -v
============================= 23 passed in 0.46s ==============================

$ ruff check backend/utils/benchmark_harness.py \
           backend/tests/test_benchmark_harness.py scripts/benchmark_harness_cli.py
All checks passed!

$ ruff format --check <same files>
3 files already formatted

$ python scripts/benchmark_harness_cli.py config.json --out bench.json
{...JSON report + report also written to bench.json...}
Exit: 0

The existing test_chromadb_benchmark.py is unchanged and continues to require chromadb + sentence-transformers (intentionally — it's the only way to benchmark the real stack under load). This new harness complements it: zero-dependent unit-tests in every env + PR-time regression gate.

Acceptance criteria

• Implement the change in the tests and tooling.
• Add or update tests, docs, or validation for the touched path.

Closes #936.

[vercel]

@arcgod-design is attempting to deploy a commit to the Darshan's projects Team on Vercel.

A member of the Team first needs to authorize it.

[imDarshanGK]

@arcgod-design
Please add a short video demo showcasing your key features for review.