Skip to content

ci(bench): publish benchmark time series to /bench on gh-pages#14

Merged
jack-champagne merged 5 commits into
mainfrom
bench/gh-pages-dashboard
Apr 18, 2026
Merged

ci(bench): publish benchmark time series to /bench on gh-pages#14
jack-champagne merged 5 commits into
mainfrom
bench/gh-pages-dashboard

Conversation

@jack-champagne

Copy link
Copy Markdown
Member

Summary

  • Publishes a live time-series benchmark dashboard at https://harmoniqs.github.io/CuQuantum.jl/bench/.
  • Coexists with Documenter on the same gh-pages branch — Documenter only writes /dev, /stable, /vX.Y.Z, /previews, so /bench is out of its way.
  • PR and branch runs stay out of the time series (per request).

How it works

  • `benchmark/run_benchmarks.jl` now emits `benchmark/results/bench.json` alongside the existing CSV, in `customSmallerIsBetter` shape (`{name, unit, value}`).
  • `gpu-benchmark.yml` runs `benchmark-action/github-action-benchmark@v1` after the GPU job.
    • `save-data-file` and `auto-push` are `${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}` — only push-to-main appends to the series.
    • PR / workflow_dispatch runs still fetch the baseline, render a comparison, and comment on regressions via `comment-on-alert`, but don't persist.
  • `docs/src/benchmarks.md` gets a dashboard link at the top.

Follow-ups to consider (separate PRs)

  • Add `push: branches: [main]` or a `schedule` trigger — today the series only grows when someone `workflow_dispatch`es the benchmark manually against main.
  • Prune `data.js` once it gets long (years out).

Test plan

  • Merge to main, then `workflow_dispatch` a benchmark run against main to seed `/bench/data.js` on `gh-pages`.
  • Visit https://harmoniqs.github.io/CuQuantum.jl/bench/ and confirm the chart renders.
  • Trigger a run against a PR branch; confirm no commit lands on `gh-pages`, but if the benchmark crosses 120% of baseline we get a PR comment.
  • Confirm Documenter `docs/deploy` still works and the dashboard link resolves from the rendered docs.

Adds a time-series dashboard at harmoniqs.github.io/CuQuantum.jl/bench/
that coexists with Documenter's existing gh-pages deploy (Documenter
only writes /dev, /stable, /vX.Y.Z, /previews).

- run_benchmarks.jl emits benchmark/results/bench.json alongside the
  existing CSV, in github-action-benchmark's customSmallerIsBetter
  schema.
- gpu-benchmark workflow runs benchmark-action/github-action-benchmark
  after the GPU run. save-data-file and auto-push are gated to
  push-to-main so PR or branch runs stay out of the published series;
  those runs still produce a comparison against the recorded baseline
  and comment if they cross the alert threshold.
- benchmarks.md links to the live dashboard.
Closes the M=5-6 "middle regime" gap in the dashboard — per docs this
is the best approach in that range. Reuses build_cpu_sparse_system
and transfers to CuSparseMatrixCSC (no new deps).

Emits as a 4th column in benchmark_results.csv and as a separate
"GPU cuSPARSE SpMV L[ρ] M=... D=..." series in bench.json.
Use QT.jl's liouvillian() + QuantumObject * state rather than hand-rolled
Kronecker builds + raw mul!. Tracks QT.jl code path so upstream regressions
in Liouvillian construction or dispatch appear in the /bench time series.

cuDensityMat path unchanged (direct library binding). QT.jl and hand-rolled
produce identical Liouvillians (max diff 7e-18). On CPU sparse the * overhead
vs raw mul! is ~4% at M=6 — kernel dominates; on GPU cuSPARSE the overhead is
more visible at small M where the cuSPARSE kernel itself is small.
- Remove unused SparseArrays and CUDA.CUSPARSE imports (QT.jl cu() handles
  sparse wrapping internally).
- Update file header to list all four tracked backends and both output files.
- Fix section numbering after QT.jl refactor (main() was 5, now 4).
@jack-champagne jack-champagne merged commit 15e3513 into main Apr 18, 2026
30 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant