Repository navigation
perf(ptl): replay native CUDA oriented networks - #23
Conversation
Replay the native oriented network without changing arithmetic. Preserve six output fields and retain native execution for unsupported consumers and accumulation. WP: 200 Provenance: R43; native repository stages and measured CUDA profiles Assumptions: none Gate: tests/ptl/test_oriented_network_capture.py Signed-off-by: Jesús Royeth <JESUSROYETH@users.noreply.github.com>
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (16 files)
Previous Review Summaries (3 snapshots, latest commit ec54fee)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit ec54fee)Status: No Issues Found | Recommendation: Merge Files Reviewed (20 files)
Previous review (commit 97703ed)This review did not finish. The model reached its output limit before it Previous review (commit 97703ed)Status: No Issues Found | Recommendation: Merge Files Reviewed (12 files)
Reviewed by free · Input: 151.3K · Output: 16.9K · Cached: 823.8K |
Derive the valid roadmap and header expectations from the configured floor. Check rejection exactly one package below the floor. WP: 200 Provenance: none (test fixture maintenance; no new sources) Assumptions: none Gate: scripts/_tests/test_audit_docs_present.py Signed-off-by: Jesús Royeth <JESUSROYETH@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
CUDA forward/backward replay needs human review while the full gate, accelerator goldens and long-run accuracy remain unverified.
No open issues 🎉
What changed in this PR
Extends opt-in CUDA network replay to oriented detection training, preserving native loss computation and state updates.
Changes:
- Records all six oriented head outputs, with native fallback for accumulation and modified objectives.
- Adds replay parity, gradient, precision and fallback tests.
- Documents measured performance, memory costs and validation limits.
| File | Description |
|---|---|
| tests/ptl/test_oriented_network_capture.py | Adds oriented replay and fallback coverage. |
| tests/ptl/test_network_capture.py | Updates supported-option expectations. |
| src/lucid_yolo/ptl/network_capture.py | Adds six-output oriented adapter. |
| src/lucid_yolo/ptl/module.py | Enables guarded oriented replay. |
| scripts/lint/audit_docs_present.py | Raises work-package floor to 200. |
| scripts/_tests/test_audit_docs_present.py | Makes floor fixtures dynamic. |
| docs/work_packages/wp-200.md | Records implementation, measurements and limits. |
| docs/work_packages/wp-199.md | Links dependent work package. |
| docs/work_packages/wp-193d.md | Links dependent work package. |
| docs/TRAINING.md | Documents oriented replay requirements. |
| docs/ROADMAP.md | Adds WP-200 tracking. |
| docs/PROVENANCE.md | Records API provenance. |
| docs/engineering/phase-17-pose-objective-parity.md | Includes WP-200 engineering log. |
| CHANGELOG.md | Announces oriented replay support. |
Review effort: Balanced
--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Oriented network replay now asks only whether the loss objects and the callables they use are the native ones. A new _native_loss helper in loss_capture.py holds that identity check, and _source_signature calls it before its scalar setting checks, so its result is unchanged. The oriented guard calls _native_loss instead of wrapping _source_signature in a try/except, so a changed scalar loss setting such as an assigner epsilon no longer retires replay, and the retirement reason reads "the native loss or an assignment implementation changed". The five oriented helper bindings are read by one _oriented_helpers function, used for both the import-time _ORIENTED_HELPERS snapshot and the per-call comparison. [resolve group] PR Borda#23 — items 4 5 WP: 200 Provenance: none (review remediation; no new source) Assumptions: none Gate: tests/ptl/test_oriented_network_capture.py::test_oriented_loss_setting_change_keeps_replay; tests/ptl/test_oriented_network_capture.py::test_oriented_classification_override_stays_native; tests/ptl/test_oriented_network_capture.py::test_oriented_helper_replacement_stays_native --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
CPU tests now reach the CUDA-gated oriented route through CPU images that report is_cuda: an unbroken module hands all three stages to run_oriented, accumulation and an overridden objective each retire once and return the native forward, and plain CPU images never reach the recorder. A subclass override retires replay, the objective-override parametrization covers all five guarded methods, the packaged OBB recipe must pass the replay guard, and setup warns about the ignored flag only for the fit stage while make_graphed_callables is patched to fail. The GPU trainer tests pin each retirement's reason, and the oriented parity oracle also compares LR-scheduler state, with lrf=0.1 so both arms run a decaying schedule. The empty-target OBB training step now runs backward and asserts that every angle_stems parameter of both branches receives a gradient, and the network-replay constructor error is matched in full. [resolve group] PR Borda#23 — items 3 9 6 7 8 10 11 WP: 200 Provenance: none (review remediation; no new source) Assumptions: none Gate: tests/ptl/test_oriented_network_capture.py; tests/ptl/test_obb_training.py::test_image_without_instances_gives_finite_loss; tests/ptl/test_network_capture.py::test_incompatible_options_are_explicit --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
TRAINING.md and the CHANGELOG entry limit the oriented speed claim to Nano OBB at batch 4, BF16 on one L4, and cite the 4.10% batch-8 fit-time change and the rise in peak reserved memory from 3974 to 4300 MiB. The network-capture docstrings cover the oriented path alongside detection and pose, and the WP-200 coverage list names gradient clipping and accumulation retirement instead of clipped accumulation. [resolve group] PR Borda#23 — items 13 14 16 WP: 200 Provenance: none (wording of existing WP-200 measurements; no new source) Assumptions: none Gate: make gate (docs site and docstring doctests) --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
The 2026-10-20 R43 row cited the PyTorch 2.9 graphed-callable contract, which R43 does not register (only the 2.4 API documentation and the v2.4.0 LICENSE), and R43 already has its first-use row on 2026-10-16 in a log defined as first-use only. The row is removed; WP-200's use of R43 stays traced through its commit trailer. [resolve group] PR Borda#23 — items 15 WP: 200 Provenance: none (usage-log correction; R43 register row unchanged) Assumptions: none Gate: pre-commit run --files docs/PROVENANCE.md; scripts/_tests/test_audit_docs_present.py --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
--model.capture_network truenow supports native OBB training, replaying the existing backbone, neck and all six head outputs. Assignment, losses and state updates keep their current arithmetic, and the flag defaults off. Accumulation and changed objectives still run the native path, so the absent-gradient semantics stay the same.I ran five alternating pairs on an isolated L4, and complete-fit time dropped 12.01%:
Setup: Nano OBB, 104 public DOTA tiles plus 13 validation tiles from disjoint source images, eight epochs, batch 4, 1024 px, BF16, deterministic kernels, on a g2-standard-8 with an L4, driver 580.178.04, torch 2.9.1+cu129, Lightning 2.6.6. Both arms have prepared caching, device HSV, epoch prefetch and optimizer replay enabled, and the timing covers setup, recording, training, EMA, validation and checkpoints.
Model, optimizer, scheduler, EMA and every loss match by byte hash, and validation agrees too. Thirty focused cases passed on torch 2.4 and 2.9 (actual replay, empty/partial batches and AMP recovery/fallback), plus five added CPU cases covering the replaced decode/loss helpers. Full pre-commit and strict mypy pass. Reserved CUDA memory goes up from 3974 to 4300 MiB. A single batch-8 probe only saved 4.10%, so the claim stays at batch 4. Full
make gate, accelerator goldens and long-run accuracy are still unverified — more detail in WP-200.Clean-room acknowledged: unchanged repository expressions, our own CUDA profiles and the registered PyTorch API R43, no external model implementation consulted.
WP: 200
Provenance: R43; native repository stages and measured CUDA profiles
Assumptions: none
Gate: tests/ptl/test_oriented_network_capture.py