Refactor hardware integration to top-level provider layout

Author: DennisWayo

README.md

@@ -237,13 +237,19 @@ recommended data transport, and adapter API.
 Xanadu-style job conversion example:
 
 ```bash
-bash examples/hardware_integration/run.sh
+bash hardware_integration/xanadu/run.sh
 ```
 
 Aurora/QCA/GKP fixture conversions:
 
 ```bash
-bash examples/hardware_integration/run_public_datasets.sh
+bash hardware_integration/xanadu/run_public_datasets.sh
+```
+
+One-command real-data slice (download + convert + replay):
+
+```bash
+bash hardware_integration/xanadu/xandau_hardware_data.sh --install-deps
 ```
 
 For large real datasets, use converter streaming/chunk flags:
@@ -264,6 +270,7 @@ Replay converted NDJSON through the C++ adapter:
 include/   # public headers and interfaces
 src/       # simulator and decoder implementations
 examples/  # reproducible runs and plotting scripts
+hardware_integration/ # provider-oriented hardware data ingestion
 ```
 
 ## Release Notes

docs/hardware-integration.md

@@ -6,6 +6,9 @@ LiDMaS+ supports three data ingestion modes for hardware integration:
 2. File batch (NDJSON)
 3. In-process C++ adapter API
 
+Provider-specific examples live under `hardware_integration/<provider>/`.
+Current provider implementation: `hardware_integration/xanadu/`.
+
 ## Recommended default
 
 - Use **sparse time-stamped syndrome events** by default.
@@ -47,7 +50,7 @@ Supported source modes:
 Legacy job JSON:
 
 ```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format xanadu_job_json \
   --input /path/to/xanadu_job.json \
   --mapping /path/to/your_mapping.json \
@@ -57,25 +60,37 @@ python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
 Quick demo:
 
 ```bash
-bash examples/hardware_integration/run.sh
+bash hardware_integration/xanadu/run.sh
 ```
 
 Aurora / QCA / GKP fixture demos:
 
 ```bash
-bash examples/hardware_integration/run_public_datasets.sh
+bash hardware_integration/xanadu/run_public_datasets.sh
+```
+
+One-command real-data slices (download + convert + replay):
+
+```bash
+bash hardware_integration/xanadu/xandau_hardware_data.sh --install-deps
+
+# QCA fig3b
+bash hardware_integration/xanadu/xandau_hardware_data.sh \
+  --dataset qca_fig3b \
+  --max-shots 200000 \
+  --install-deps
 ```
 
 Real Aurora decoder-demo batch:
 
 ```bash
 python3 -m pip install numpy
 
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format aurora_switch_dir \
   --stream \
   --input /path/to/decoder_demo/signal/batch_0 \
-  --mapping examples/hardware_integration/xanadu_aurora_mapping_example.json \
+  --mapping hardware_integration/xanadu/xanadu_aurora_mapping_example.json \
   --out examples/results/hardware_integration/decoder_requests_aurora.ndjson \
   --aurora-binarize \
   --max-shots 20000 \
@@ -85,11 +100,11 @@ python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
 Real QCA sample matrix:
 
 ```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format shot_matrix \
   --stream \
   --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
+  --mapping hardware_integration/xanadu/xanadu_qca_mapping_example.json \
   --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
   --max-shots 50000 \
   --progress-every 10000
@@ -98,20 +113,20 @@ python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
 Chunk large QCA files by repeating conversion with shifted `--shot-start` and `--append-out`:
 
 ```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format shot_matrix \
   --stream \
   --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
+  --mapping hardware_integration/xanadu/xanadu_qca_mapping_example.json \
   --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
   --shot-start 0 \
   --max-shots 200000
 
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format shot_matrix \
   --stream \
   --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
+  --mapping hardware_integration/xanadu/xanadu_qca_mapping_example.json \
   --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
   --append-out \
   --shot-start 200000 \
@@ -121,15 +136,15 @@ python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
 Count-compressed GKP outcomes:
 
 ```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
+python3 hardware_integration/xanadu/convert_xanadu_job_to_decoder_io.py \
   --source-format count_table_json \
   --input /path/to/gkp_outcome_counts.json \
-  --mapping examples/hardware_integration/xanadu_gkp_mapping_example.json \
+  --mapping hardware_integration/xanadu/xanadu_gkp_mapping_example.json \
   --out examples/results/hardware_integration/decoder_requests_gkp.ndjson
 ```
 
 The mapping file controls how measured modes are converted to syndrome events.
-See `examples/hardware_integration/xanadu_syndrome_mapping_example.json`.
+See `hardware_integration/xanadu/xanadu_syndrome_mapping_example.json`.
 
 Large-data controls:
 

examples/README.md

@@ -27,7 +27,7 @@ LIDMAS_SKIP_PY_DEPS=1 ./examples/hybrid_threshold/run.sh
 - `decoder_comparison/`: same sweep across multiple decoders.
 - `failure_debug/`: stress run and failure-dump capture workflow.
 - `plot_only/`: publication-grade plotting from existing CSV files.
-- `hardware_integration/`: convert Xanadu datasets (Aurora/QCA/GKP/job JSON) to LiDMaS+ decoder IO NDJSON.
+- `../hardware_integration/`: provider-oriented hardware integration entry point; current provider: `hardware_integration/xanadu/` (Aurora/QCA/GKP/job JSON) plus one-command real-data downloader.
 
 
 ## Central Results Folder

examples/hardware_integration/README.md

@@ -1,209 +1,17 @@
-# hardware_integration
+# Hardware Integration (Moved)
 
-This example converts Xanadu datasets to LiDMaS+ `decoder_io` NDJSON.
-The same converter now supports:
+Hardware integration entry points now live at:
 
-- legacy Xanadu job JSON payloads,
-- Aurora decoder-demo switch-setting directories,
-- QCA/Borealis-style shot matrices (`samples.npy`),
-- count-compressed outcomes (useful for GKP-style exports).
+- `hardware_integration/xanadu/`
 
-## Files
-
-- `convert_xanadu_job_to_decoder_io.py`: converter script.
-- `xanadu_job_result_example.json`: minimal sample job payload.
-- `xanadu_syndrome_mapping_example.json`: mode-to-syndrome parity mapping.
-- `aurora_batch_example/`: tiny Aurora-style batch with `switch_settings_qpu_*.json`.
-- `xanadu_aurora_mapping_example.json`: Aurora mapping.
-- `xanadu_qca_samples_example.json`: QCA-like shot-matrix fixture.
-- `xanadu_qca_mapping_example.json`: QCA mapping.
-- `xanadu_gkp_counts_example.json`: count-compressed outcome fixture.
-- `xanadu_gkp_mapping_example.json`: GKP mapping.
-- `run.sh`: one-command local demo.
-- `run_aurora.sh`: Aurora conversion demo.
-- `run_qca.sh`: QCA conversion demo.
-- `run_gkp.sh`: GKP count-table conversion demo.
-- `run_public_datasets.sh`: run Aurora + QCA + GKP demos.
-- `replay.sh`: decode generated requests via `./build/lidmas --decoder_io_replay`.
-
-## Quick Run
-
-```bash
-bash examples/hardware_integration/run.sh
-```
-
-Output:
-
-- `examples/results/hardware_integration/decoder_requests.ndjson`
-
-Each NDJSON line is a `DecodeRequest` compatible with `schemas/decoder_io.proto`.
-
-Run all public-dataset fixtures:
-
-```bash
-bash examples/hardware_integration/run_public_datasets.sh
-```
-
-Outputs:
-
-- `examples/results/hardware_integration/decoder_requests_aurora.ndjson`
-- `examples/results/hardware_integration/decoder_requests_qca.ndjson`
-- `examples/results/hardware_integration/decoder_requests_gkp.ndjson`
-
-Replay those requests through the C++ adapter:
-
-```bash
-./build/lidmas --decoder_io_replay \
-  --decoder_io_in=examples/results/hardware_integration/decoder_requests.ndjson \
-  --decoder_io_out=examples/results/hardware_integration/decoder_responses.ndjson \
-  --decoder_io_config=schemas/surface_decoder_adapter_config.json \
-  --decoder_io_continue_on_error
-```
-
-Helper script (auto-derives response filename):
-
-```bash
-bash examples/hardware_integration/replay.sh \
-  examples/results/hardware_integration/decoder_requests_aurora.ndjson
-```
-
-## Real Xanadu Job Data
-
-### Aurora decoder_demo batch (switch settings)
-
-Requires NumPy when reading `.npy` files:
-
-```bash
-python3 -m pip install numpy
-```
-
-Convert one Aurora batch directory:
-
-```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format aurora_switch_dir \
-  --input /path/to/decoder_demo/signal/batch_0 \
-  --mapping examples/hardware_integration/xanadu_aurora_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_aurora.ndjson \
-  --aurora-binarize \
-  --max-shots 20000 \
-  --meta hardware=xanadu \
-  --meta dataset=aurora_decoder_demo
-```
-
-For large Aurora batches, enable streaming and progress logs:
-
-```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format aurora_switch_dir \
-  --stream \
-  --input /path/to/decoder_demo/signal/batch_0 \
-  --mapping examples/hardware_integration/xanadu_aurora_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_aurora.ndjson \
-  --aurora-binarize \
-  --max-shots 200000 \
-  --progress-every 50000
-```
-
-### QCA / Borealis samples.npy
-
-Convert `samples.npy` (shape typically `n x 1 x M`):
-
-```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format shot_matrix \
-  --stream \
-  --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
-  --max-shots 50000 \
-  --meta hardware=xanadu \
-  --meta dataset=qca
-```
-
-Chunked QCA conversion (memory-safe over very large files):
-
-```bash
-# chunk 1
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format shot_matrix \
-  --stream \
-  --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
-  --shot-start 0 \
-  --max-shots 200000 \
-  --progress-every 50000
-
-# chunk 2 (append)
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format shot_matrix \
-  --stream \
-  --input /path/to/fig3a/samples.npy \
-  --mapping examples/hardware_integration/xanadu_qca_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_qca.ndjson \
-  --append-out \
-  --shot-start 200000 \
-  --max-shots 200000 \
-  --progress-every 50000
-```
-
-### GKP outcome counts
-
-For count-compressed outcomes exported from your analysis notebook:
+Run from repo root:
 
 ```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --source-format count_table_json \
-  --input /path/to/gkp_outcome_counts.json \
-  --mapping examples/hardware_integration/xanadu_gkp_mapping_example.json \
-  --out examples/results/hardware_integration/decoder_requests_gkp.ndjson \
-  --max-shots 100000 \
-  --meta hardware=xanadu \
-  --meta dataset=gkp
-```
-
-`gkp_outcome_counts.json` entries should look like:
-
-```json
-{"counts":[{"sample":[0,1,0],"count":12},{"sample":[1,0,1],"count":5}]}
+bash hardware_integration/xanadu/run.sh
+bash hardware_integration/xanadu/run_public_datasets.sh
+bash hardware_integration/xanadu/xandau_hardware_data.sh --install-deps
 ```
 
-### Legacy Xanadu job JSON
-
-Use the converter with exported job JSON payloads:
-
-```bash
-python3 examples/hardware_integration/convert_xanadu_job_to_decoder_io.py \
-  --input /path/to/xanadu_job.json \
-  --mapping /path/to/your_mapping.json \
-  --out examples/results/hardware_integration/decoder_requests.ndjson \
-  --sigma 0.18 \
-  --gate-error-rate 0.0007 \
-  --meas-error-rate 0.0009 \
-  --idle-error-rate 0.0003 \
-  --meta hardware=xanadu \
-  --meta backend=X8_01
-```
-
-## Mapping Notes
-
-`stabilizers` entries define syndrome-event generation by parity:
-
-- `index`: stabilizer index in LiDMaS+ space.
-- `type`: `X`, `Z`, or `UNKNOWN`.
-- `modes`: list of measured-mode indices used for parity.
-- `mod`: modulus (default 2).
-- `trigger_on`: event if `(sum(modes) % mod) == trigger_on` (default 1).
-- `time_offset_ns`: optional per-stabilizer event timestamp offset.
-
-This mapping is hardware- and experiment-specific.
-Illustrative mappings may produce non-physical syndromes; use `--decoder_io_continue_on_error` during early integration.
-
-## Large-Data Flags
+Outputs remain in:
 
-- `--stream`: uses NumPy memory-mapped loading when available (`.npy`), avoiding full in-memory expansion.
-- `--shot-start N`: skip the first `N` expanded shots before writing.
-- `--max-shots K`: write at most `K` shots this run.
-- `--append-out`: append NDJSON to an existing output file.
-- `--progress-every M`: emit progress every `M` written shots to stderr.
+- `examples/results/hardware_integration/`

hardware_integration/README.md

@@ -0,0 +1,19 @@
+# Hardware Integration
+
+Provider-oriented hardware integration entry point for LiDMaS+.
+
+## Providers
+
+- `xanadu/`: converters, mappings, demos, and real-data download/replay scripts for Aurora/QCA/GKP workflows.
+
+## Quick Start
+
+```bash
+bash hardware_integration/xanadu/run.sh
+bash hardware_integration/xanadu/run_public_datasets.sh
+bash hardware_integration/xanadu/xandau_hardware_data.sh --install-deps
+```
+
+Outputs stay centralized in:
+
+- `examples/results/hardware_integration/`