Reproducible benchmarks/ scripts + committed benchmark outputs

[rosspeili]

Context

notebooks/02–04 produce one-off results inside Jupyter. OVERVIEW.md deliverables list benchmarks/ as reproducible scripts vs classical Echo State Networks. The benchmarks/ directory is empty (.gitkeep only).

spinq-vqe ships committed data/.csv and figures/.png from executed notebooks; spintronic-qrc should mirror that pattern for script-driven reruns without opening notebooks.

After Issues #1, #4, #5, #6, #7, #8, the package has:

• pipeline.run_qrc() — quantum reservoir
• esn.run_esn() — classical baseline
• memory.qrc_memory_capacity() / esn_memory_capacity() — Dambre metric
• tasks.narma10(), tasks.mackey_glass()

This issue wires them into CLI-runnable scripts that write standardized CSVs to data/.

Goal

Add benchmarks/ scripts that reproduce headline numbers from Notebooks 02–04 and can be run from the repo root:

pytest spintronic-qrc/tests/ -v
python benchmarks/run_narma10.py
python benchmarks/run_mackey_glass.py
python benchmarks/run_memory_capacity.py

Optional fourth script run_all.py that calls the three in sequence.

Requirements

1. Shared benchmark utilities

Add benchmarks/_common.py (or src/spintronic_qrc/benchmarks.py if you prefer package scope — scripts/ folder is fine for v1):

• parse_seed() from env or --seed flag (default 42)
• write_results_csv(path, rows: list[dict]) — append-safe or overwrite with timestamp column
• default QRCConfig and ESNConfig presets matching Notebook 02 committed values (document in module docstring)
• print_summary(table) for stdout

2. benchmarks/run_narma10.py

• Generate NARMA-10 train/test via tasks.narma10()

• Run run_qrc() and run_esn() with shared chronological split

• Write/append data/narma10_results.csv with columns:

  method, n_sites, n_reservoir, evolution_time, disorder, washout, ridge_alpha, train_length, test_length, train_rmse, test_rmse, test_nmse, seed

• Two rows minimum per run: QRC and ESN

• CLI: --seed, --train-length, --test-length, --quick (smaller lengths for CI smoke)

3. benchmarks/run_mackey_glass.py

• Build Mackey-Glass supervised pairs (same u(t)=x(t), y(t)=x(t+1) normalization as Notebook 03)
• Run QRC + ESN
• Write data/mackey_glass_results.csv (same column pattern + mg_tau)

4. benchmarks/run_memory_capacity.py

• Call qrc_memory_capacity() and esn_memory_capacity() with shared MemoryCapacityConfig

• Write data/memory_capacity.csv:

  method, n_sites, n_reservoir, k_max, total_mc, seed

• Two rows: QRC, ESN

5. benchmarks/run_all.py (optional but recommended)

• Runs all three scripts via subprocess or direct imports
• Exits non-zero if any script fails
• Prints combined summary table

6. Quick / CI mode

Each script supports --quick flag:

Script	--quick behavior
narma10	train_length=200, test_length=50
mackey_glass	total_length=500
memory_capacity	k_max=5, n_samples=100

Full defaults match notebook committed settings (document in script --help).

7. Documentation

• benchmarks/README.md — how to run, output files, --quick vs full
• Root README.md — add short Benchmarks section with three commands (mirror spinq-vqe scripts/ mention)
• docs/testing.md — note benchmark smoke can be run manually (not required in pytest by default)

8. Tests

Add tests/test_benchmarks_smoke.py:

• Import benchmark main functions or run with --quick via subprocess
• Verify CSV files created in tmp_path (use monkeypatch or chdir to temp dir)
• Keep under ~60 s total

Do not add benchmark runs to default pytest suite if too slow — mark @pytest.mark.slow or skip unless env RUN_BENCHMARKS=1.

Acceptance criteria

• benchmarks/run_narma10.py, run_mackey_glass.py, run_memory_capacity.py exist and run with --quick
• Full run produces/updates data/narma10_results.csv, mackey_glass_results.csv, memory_capacity.csv
• Each script compares QRC and ESN on identical splits
• benchmarks/README.md documents usage
• tests/test_benchmarks_smoke.py passes with --quick
• OVERVIEW.md deliverables: check off benchmarks/

Files likely touched

• benchmarks/_common.py (new)
• benchmarks/run_narma10.py (new)
• benchmarks/run_mackey_glass.py (new)
• benchmarks/run_memory_capacity.py (new)
• benchmarks/run_all.py (new, optional)
• benchmarks/README.md (new)
• tests/test_benchmarks_smoke.py (new)
• README.md (small Benchmarks section)
• data/*.csv (updated by full run — commit if changed)

Depends on

• Issue Implement end-to-end QRC pipeline (ApproxTimeEvolution + feature matrix builder) #1 (pipeline) — required
• Issue Notebook 02 — NARMA-10 end-to-end QRC + committed results #4 (NARMA defaults) — recommended
• Issue Classical Echo State Network baseline module + unit tests #5 (ESN) — required
• Issue Memory capacity metric (Dambre et al. 2012) in package #6 (memory) — required for memory script
• Issue Notebook 03 — Mackey-Glass chaotic attractor prediction #7, Notebook 04 — Memory capacity vs chain parameters + Optuna hyperparameter sweeps #8 (notebooks) — recommended for parameter alignment

Blocks

• spintronic-qbench task handoff (standardized CSV schema)
• CI workflow adding benchmark smoke job (future)