perf(ptl): replay native CUDA oriented networks

CHANGELOG.md

@@ -6,6 +6,8 @@ All notable changes to lucid-yolo are documented here, following the Keep a Chan
 
 ### Added
 
+- **Opt-in CUDA network replay for oriented training.** `--model.capture_network true` records the existing backbone, neck and both oriented head branches, including their angle outputs. The native objective and state updates remain outside capture. Accumulation and overridden objectives keep the native route; the default stays off. The speed claim is limited to a small-batch workload (Nano OBB at batch 4, BF16, one L4): a single batch-8 pair changed fit time by only 4.10%, and replay raised peak reserved memory from 3974 to 4300 MiB, so check throughput and memory on your own batch size. See [WP-200](docs/work_packages/wp-200.md) for complete-fit measurements and limits.
+
 - **Opt-in CUDA replay of detection assignment and loss.** `--model.capture_loss true` records the existing loss forward/backward with a masked target workspace that grows from observed counts. Current label checks and schedules stay live; returned values own their storage and fit exit releases the recording. Default off, compatible with network and orthogonalization replay. See [WP-199](docs/work_packages/wp-199.md) for measurements and limits.
 
 - **Opt-in shared-feature CUDA replay for pose training.** `--model.capture_network true` can record the existing backbone and neck for `task: keypoints`; the head stays native so a batch with no assigned positives keeps absent point-stem gradients. Accumulated pose runs, runs with a fully frozen backbone and neck, and heads with changed topology, hooks or forwards stay native. The default remains off. The speed claim is limited to a small-batch workload (batch 4, one L4): a single batch-16 pair changed fit time by 2.50%, and replay lowered peak allocated memory while raising peak reserved memory, so check throughput and both memory figures on your own batch size. See [WP-193d](docs/work_packages/wp-193d.md) for complete-fit measurements and validation limits.

docs/ROADMAP.md

@@ -1,6 +1,6 @@
 # 🗺️ Work-Package Roadmap
 
-The agent's work queue: 199 numbered work packages, one commit each, executed in dependency order per the AGENTS.md loop, plus lettered rows recording follow-up work that landed after its package had closed. Transcribed from the governing blueprint (sec. 15) with a live status column.
+The agent's work queue: 200 numbered work packages, one commit each, executed in dependency order per the AGENTS.md loop, plus lettered rows recording follow-up work that landed after its package had closed. Transcribed from the governing blueprint (sec. 15) with a live status column.
 
 **Scope says what a package does, not what executing it taught.** Measurements, rejected approaches and negative results live in RESEARCH_LOG.md, linked per row as `· log`; choices the papers left open live in ASSUMPTIONS.md and are cited by id; release results live in REPRODUCTION_REPORT.md. A row that restates any of those is duplicating a record that has an owner.
 
@@ -364,6 +364,7 @@ Opened by issue 2 on the tracker: R1 sec. 3.4.2 publishes the pose objective thi
 | 197c | `perf(optim): replay CUDA update preparation` [DATA][GPU] | Opt-in group preparation replay retains native arithmetic, records momentum with rollback-safe setup and applies changing learning rates outside capture; layout alignment enables final foreach writes (D29) · [log](work_packages/wp-197c.md#wp-197c) | `tests/optim/test_update_replay.py`; `tests/ptl/test_cli.py::test_capture_updates_flag_reaches_optimizer`; focused optimizer/replay/CLI CPU/CUDA checks, mutation proof, five paired public-COCO fits with byte parity and 10.01% median complete-fit reduction (nano, batch 4, BF16, L4; batch-32 screen showed no benefit; batch 16 unmeasured); mypy and full precommit; full `make gate` not run | 197b | 🔄 |
 | 198 | `perf(data): overlap epoch preparation and fuse colour restore` [DATA][GPU] | Persistent workers prepare up to `num_workers x prefetch_factor` next-epoch batches during validation; an optional Triton pass restores the existing colour boundary without intermediate batches. Fixed sampler ownership, finite epochs and exact pixel codes remain the contract · [log](work_packages/wp-198.md#wp-198) | `tests/ptl/test_epoch_prefetch.py`; `tests/ptl/test_fused_device_hsv.py`; isolated-L4 complete-fit comparisons; full `make gate` not run | 197 | 🔄 |
 | 199 | `perf(ptl): replay detection assignment and loss` [DATA][GPU] | Opt-in loss forward/backward replay calls the existing assignment and scoring expressions with masked target capacity. One recording grows from observed counts; checks and live alpha stay current, and optimizer work is independent · [log](work_packages/wp-199.md#wp-199) | `tests/ptl/test_loss_capture.py`; focused CPU/CUDA, mutation checks and repeated complete public-COCO fits; full `make gate` and accelerator goldens not run | 198 | 🔄 |
+| 200 | `perf(ptl): replay native CUDA oriented networks` [DATA][GPU] | Opt-in network replay retains all six native oriented outputs, while classification, rotated objectives and state updates stay native. Accumulation and changed objectives retain the original route · [log](work_packages/wp-200.md#wp-200) | `tests/ptl/test_oriented_network_capture.py`; actual graph execution, empty-angle gradients and validation transitions; five paired public-DOTA complete fits above 8%; focused CPU/CUDA and full pre-commit; full `make gate`, accelerator goldens and long-run accuracy not run | 193d, 199 | 🔄 |
 
 ## ⛓️ Critical path and parallelism
 

docs/TRAINING.md

@@ -143,7 +143,9 @@ Budget about `12 * sum(height * width)` bytes for the retained RGB pixels, plus
 
 For `task: keypoints`, the same flag records only backbone and neck; the complete dual head and the current RLE/OKS or Laplace objectives stay native. That preserves absent point-stem gradients on batches with no assigned positives. Gradient accumulation, a fully frozen backbone and neck, changed head topology or forwards, and head hooks/compilation keep pose native; a partial freeze that leaves any backbone or neck parameter trainable still replays. The trainable set is fixed per fit at the first recording, so a fully frozen start, or a later freeze or unfreeze of any backbone or neck parameter, keeps the rest of the fit native after one warning: a recipe that starts with backbone and neck fully frozen trains natively throughout, even after it unfreezes them. The head's ordinary dense classification path supplies gradients to all three shared features. The pose speed-up is likewise limited to a small-batch workload (batch 4): a single batch-16 pair changed fit time by 2.50%, and replay lowered peak allocated memory while raising peak reserved memory (see the memory note below), so check throughput and memory on your own batch size. The pose measurements and remaining validation limits are recorded in [WP-193d](work_packages/wp-193d.md).
 
-This path requires single-device CUDA automatic optimisation and `task: detect` or `task: keypoints`. BF16 replay requires a final PyTorch 2.9 or newer release; 2.9 pre-release and NGC `a0` builds are excluded by design. 2.9 is the first verified runtime: older BF16 runs warn and stay eager because the 2.4 comparison did not meet the exact eager oracle. FP32 and FP16 replay are checked on 2.4 as well. CPU, MPS and other strategies warn and stay eager; other tasks or `compile_step` raise at construction. One image signature is recorded per fit. A partial batch uses eager execution without growing a graph cache. Changes to the recorded stages’ parameter/buffer storage, module modes, hooks or precision retire replay for the rest of that fit. The supplied network must keep its ordinary stage implementations and static Python configuration. Root forward overrides, stage instance-forward overrides and registered hooks use the eager path. Anomaly detection (`--trainer.detect_anomaly true`) also stays eager so backward NaN checks run normally. A failure while recording raises instead of falling back to eager (RNG state and buffers are restored first), so disable `--model.capture_network` if recording fails on your model. Higher-order differentiation is not supported.
+For `task: obb`, the flag records backbone, neck and the complete dual head, including both angle outputs. The native classification and oriented losses stay outside capture. Their empty-target expressions retain zero angle gradients. Overridden training consumers, classification losses or oriented helpers keep the complete native call; gradient accumulation also stays native. The oriented speed-up is likewise limited to a small-batch workload (Nano OBB at batch 4, BF16, one L4): a single batch-8 pair changed fit time by only 4.10%, and replay raised peak reserved memory from 3974 to 4300 MiB, so check throughput and memory on your own batch size. The measurements and remaining limits are in [WP-200](work_packages/wp-200.md).
+
+This path requires single-device CUDA automatic optimisation and `task: detect`, `task: keypoints` or `task: obb`. BF16 replay requires a final PyTorch 2.9 or newer release; 2.9 pre-release and NGC `a0` builds are excluded by design. 2.9 is the first verified runtime: older BF16 runs warn and stay eager because the 2.4 comparison did not meet the exact eager oracle. FP32 and FP16 replay are checked on 2.4 as well. CPU, MPS and other strategies warn and stay eager; other tasks or `compile_step` raise at construction. One image signature is recorded per fit. A partial batch uses eager execution without growing a graph cache. Changes to the recorded stages’ parameter/buffer storage, module modes, hooks or precision retire replay for the rest of that fit. The supplied network must keep its ordinary stage implementations and static Python configuration. Root forward overrides, stage instance-forward overrides and registered hooks use the eager path. Anomaly detection (`--trainer.detect_anomaly true`) also stays eager so backward NaN checks run normally. A failure while recording raises instead of falling back to eager (RNG state and buffers are restored first), so disable `--model.capture_network` if recording fails on your model. Higher-order differentiation is not supported.
 
 Recording warmup restores RNG and BatchNorm buffers before the first real batch. Accumulation or custom optimiser/zero-grad hooks preserve live gradients outside the graph-owned backward buffers before they can be overwritten. AMP uses an uncached autocast scope as the PyTorch API requires. Validation, prediction and public `forward` calls stay eager. Checkpoints retain the ordinary parameter names and the flag, with no graph state. Each fit records afresh and drops its recording on completion or an exception. CUDA graph pools can increase reserved memory even when peak allocated memory falls; measure both on the intended workload. The measured scope and remaining validation gaps are in [WP-193c](work_packages/wp-193c.md).
 

docs/engineering/phase-17-pose-objective-parity.md

@@ -37,3 +37,5 @@
 --8<-- "work_packages/wp-198.md:engineering"
 
 --8<-- "work_packages/wp-199.md:engineering"
+
+--8<-- "work_packages/wp-200.md:engineering"

docs/work_packages/wp-193d.md

@@ -2,7 +2,7 @@
 
 > Engineering log · [Phase 17 — R1 pose-objective parity](../engineering/phase-17-pose-objective-parity.md)
 >
-> 🔗 Depends on [WP-193c](../work_packages/wp-193c.md) · Needed by none
+> 🔗 Depends on [WP-193c](../work_packages/wp-193c.md) · Needed by [WP-200](../work_packages/wp-200.md)
 >
 > ⬅️ [WP-193c](../work_packages/wp-193c.md) · [Roadmap](../ROADMAP.md) · [WP-193e](../work_packages/wp-193e.md) ➡️
 

docs/work_packages/wp-199.md

@@ -2,9 +2,9 @@
 
 > Engineering log · [Phase 17 — R1 pose-objective parity](../engineering/phase-17-pose-objective-parity.md)
 >
-> 🔗 Depends on [WP-198](../work_packages/wp-198.md) · Needed by none
+> 🔗 Depends on [WP-198](../work_packages/wp-198.md) · Needed by [WP-200](../work_packages/wp-200.md)
 >
-> ⬅️ [WP-198](../work_packages/wp-198.md) · [Roadmap](../ROADMAP.md)
+> ⬅️ [WP-198](../work_packages/wp-198.md) · [Roadmap](../ROADMAP.md) · [WP-200](../work_packages/wp-200.md) ➡️
 
 <!-- --8<-- [start:engineering] -->
 

docs/work_packages/wp-200.md

@@ -0,0 +1,43 @@
+# WP-200 — replay native CUDA oriented networks
+
+> Engineering log · [Phase 17 — R1 pose-objective parity](../engineering/phase-17-pose-objective-parity.md)
+>
+> 🔗 Depends on [WP-193d](../work_packages/wp-193d.md), [WP-199](../work_packages/wp-199.md) · Needed by none
+>
+> ⬅️ [WP-199](../work_packages/wp-199.md) · [Roadmap](../ROADMAP.md)
+
+<!-- --8<-- [start:engineering] -->
+
+<a id="wp-200"></a>
+
+## WP-200 — six native outputs, one recorded network
+
+**The oriented head is part of the recording.** The existing `capture_network` option now accepts `task: obb`. Backbone, neck and both head branches run their existing expressions through R43's graphed-callable API. The adapter retains the four dense detection fields and both angle tensors, rebuilding them by name. Assignment, rotated decoding, classification, ProbIoU, rotated L1, angle scoring, schedules, clipping, optimizer, EMA and validation stay in their current loop. No method arithmetic, gain, convention or frozen golden changes.
+
+**The complete native objective owns the gradient contract.** Its empty-target path retains zero gradients to both angle stems. An overridden objective can instead leave those gradients absent, so a changed training consumer, classification loss or oriented helper keeps the complete native call. Accumulation also stays native, as it does for pose: the existing shared recorder has no established accumulation contract. The ordinary network storage, hooks, precision, single-device CUDA and first-order constraints remain unchanged. One image signature is retained per fit; a temporary partial batch stays native and can return to the existing recording. The default stays off.
+
+**The stage was priced before implementation.** Four native CUDA steps occupied a 1.466 s instrumented window, with about 24.1% covered by recorded device activity. Network forward dispatch took 378.113 ms, including 174.956 ms in backbone and neck and 202.463 ms in the head. A separate complete fit with lightweight host timers took 68.263 s: network forwards took 15.943 s, or 23.35%, across 208 training and 32 validation calls. That includes validation, which remains native, and is an upper bound rather than a speed claim. The 21.261 s backward timer includes loss work as well as network work and is not wholly removable. Profiler runs are separate from the final timing comparisons.
+
+**The smaller recording did not clear the investigation's target.** A shared-backbone/neck-only screen preserved model, optimizer, scheduler, EMA and loss bytes, but changed complete-fit time from 68.277 to 63.800 s, a single-pair 6.56% reduction. It was set aside before the final cohort. The complete six-output screen preserved those same states and validation values and changed 68.742 to 59.853 s, a preliminary 12.93%; the repeated cohort, rather than that screen, determines the final claim.
+
+**Workload.** The registered nano oriented model trains on 104 public DOTA-v1.0 tiles with 2,075 annotations and validates on 13 tiles with 616 annotations. Training and validation use disjoint original source images, 16 and 4 respectively. Eight epochs, batch 4, 1024 px, BF16, seed 123, deterministic kernels, four persistent loader workers and one NVIDIA L4 on a g2-standard-8. Both arms enable prepared source caching, device HSV, epoch prefetch and the existing orthogonalization/update-preparation replay. The comparison uses separate upstream and candidate source trees in fresh processes. Timing brackets include model/trainer construction, cache and recording setup, augmentation, training, EMA, validation and checkpoint writes, with device synchronization at both boundaries.
+
+**[MEASURED] Five alternating fresh-process pairs.** One excluded warmup pair precedes the five measured pairs. Native and replay ranges do not overlap; every paired reduction is positive (11.669–13.572%). The exact model, optimizer, scheduler, EMA and per-step loss byte hashes agree in each pair and across native repeats, as do validation values. The package import path and hashes bind each arm to its separate source tree. The complete-fit median drops **12.011%**.
+
+| Metric, median (range across five runs) | Native | Replay |
+| -- | -- | -- |
+| Complete fit, seconds | 68.642 (68.213–69.333) | 60.397 (59.924–60.784) |
+| Whole process, seconds | 73.795 (73.379–74.561) | 65.608 (65.169–65.987) |
+| Warmed training step, ms | 183.275 (179.156–184.821) | 139.500 (138.453–141.266) |
+
+Warmed step diagnostics discard the first 64 steps and use CUDA events, synchronized before reading. Peak allocated memory is 2,572,018,688 bytes native and 1,889,886,208 bytes replay; peak reserved memory is 4,167,041,024 and 4,508,876,800 bytes, respectively, constant across the five runs. Extra graph-pool reservation is a tradeoff, not a capacity improvement. Dependency/compiler and OS caches are warm; each fit reconstructs its prepared image cache and CUDA recording. Package import and post-fit byte hashing are outside the fit timer and included in the separately reported process time. Runtime: Python 3.11.17, torch 2.9.1+cu129, Lightning 2.6.6, L4 driver 580.178.04.
+
+**The removed work is observed at its owner.** Separate four-step CUDA traces show 31,193 native `cudaLaunchKernel` calls and 12,609 with replay, while graph launches rise from 4 to 12, adding the network forward/backward recordings beside the existing optimizer replay. Device-event union is 353.374 and 358.045 ms; these instrumented windows support the dispatch diagnosis, not a kernel-speed claim.
+
+**A larger-batch probe limits the claim.** One batch-8 pair, with the same eight epochs and 1024 px inputs, changes complete-fit time from 60.494 to 58.014 s, only 4.099%. State and validation parity still hold, but that one pair is below the investigation's 8% target and is not a repeated performance claim. The five-pair headline remains scoped to batch 4.
+
+**Validation.** Thirty oriented/constructor cases passed on both torch 2.9.1 and 2.4.0; the floor retains its existing BF16 eager fallback, while FP32 and FP16 exercise real replay. A silent eager reversion and an unsafe objective-guard bypass each make the committed Trainer tests fail. Five additional helper-replacement cases pass on CPU after review. Local CPU checks and source doctests, with those cases included, report 17 passed and 19 CUDA skips. Strict mypy and the full pre-commit hooks pass. No frozen golden is changed.
+
+**Coverage and limits.** Focused tests compare all six fields, actual CUDA launches, empty-target gradient presence, temporary shape fallback, gradient clipping, accumulation retirement (native fallback), FP32/FP16/BF16, scaler overflow and recovery, native-objective fallback, EMA validation and fit cleanup. FP16 fixtures use a unit initial scale plus one injected overflow so at least one real optimizer update must occur. A silent native reversion must fail the launch assertions; bypassing the objective guard must be rejected by the absent-angle-gradient case. Full `make gate`, accelerator goldens and long-run acceptance accuracy are outside this validation run.
+
+<!-- --8<-- [end:engineering] -->