Skip to content

perf(ptl): replay native CUDA oriented networks - #23

Merged
Borda merged 7 commits into
Borda:mainfrom
JESUSROYETH:codex/obb-network-replay
Oct 20, 2026
Merged

Borda merged 7 commits into
Borda:mainfrom
JESUSROYETH:codex/obb-network-replay

Conversation

@JESUSROYETH

Copy link
Copy Markdown
Collaborator

--model.capture_network true now supports native OBB training, replaying the existing backbone, neck and all six head outputs. Assignment, losses and state updates keep their current arithmetic, and the flag defaults off. Accumulation and changed objectives still run the native path, so the absent-gradient semantics stay the same.

I ran five alternating pairs on an isolated L4, and complete-fit time dropped 12.01%:

Seconds, median (range) Native Replay
Complete fit 68.642 (68.213–69.333) 60.397 (59.924–60.784)

Setup: Nano OBB, 104 public DOTA tiles plus 13 validation tiles from disjoint source images, eight epochs, batch 4, 1024 px, BF16, deterministic kernels, on a g2-standard-8 with an L4, driver 580.178.04, torch 2.9.1+cu129, Lightning 2.6.6. Both arms have prepared caching, device HSV, epoch prefetch and optimizer replay enabled, and the timing covers setup, recording, training, EMA, validation and checkpoints.

Model, optimizer, scheduler, EMA and every loss match by byte hash, and validation agrees too. Thirty focused cases passed on torch 2.4 and 2.9 (actual replay, empty/partial batches and AMP recovery/fallback), plus five added CPU cases covering the replaced decode/loss helpers. Full pre-commit and strict mypy pass. Reserved CUDA memory goes up from 3974 to 4300 MiB. A single batch-8 probe only saved 4.10%, so the claim stays at batch 4. Full make gate, accelerator goldens and long-run accuracy are still unverified — more detail in WP-200.

Clean-room acknowledged: unchanged repository expressions, our own CUDA profiles and the registered PyTorch API R43, no external model implementation consulted.

WP: 200
Provenance: R43; native repository stages and measured CUDA profiles
Assumptions: none
Gate: tests/ptl/test_oriented_network_capture.py

Replay the native oriented network without changing arithmetic. Preserve six output fields and retain native execution for unsupported consumers and accumulation.

WP: 200
Provenance: R43; native repository stages and measured CUDA profiles
Assumptions: none
Gate: tests/ptl/test_oriented_network_capture.py
Signed-off-by: Jesús Royeth <JESUSROYETH@users.noreply.github.com>
@JESUSROYETH
JESUSROYETH requested a review from Borda as a code owner October 20, 2026 17:05
@kilo-code-bot

kilo-code-bot Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (16 files)
  • CHANGELOG.md
  • docs/PROVENANCE.md
  • docs/ROADMAP.md
  • docs/TRAINING.md
  • docs/engineering/phase-17-pose-objective-parity.md
  • docs/work_packages/wp-193d.md
  • docs/work_packages/wp-199.md
  • docs/work_packages/wp-200.md
  • scripts/lint/audit_docs_present.py
  • scripts/_tests/test_audit_docs_present.py
  • src/lucid_yolo/ptl/loss_capture.py
  • src/lucid_yolo/ptl/module.py
  • src/lucid_yolo/ptl/network_capture.py
  • tests/ptl/test_network_capture.py
  • tests/ptl/test_obb_training.py
  • tests/ptl/test_oriented_network_capture.py
Previous Review Summaries (3 snapshots, latest commit ec54fee)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit ec54fee)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (20 files)
  • src/lucid_yolo/models/heads/detect.py
  • src/lucid_yolo/ptl/loss_capture.py
  • src/lucid_yolo/ptl/module.py
  • src/lucid_yolo/ptl/network_capture.py
  • src/lucid_yolo/ptl/segment_head_capture.py
  • scripts/lint/audit_docs_present.py
  • scripts/_tests/test_audit_docs_present.py
  • tests/ptl/test_network_capture.py
  • tests/ptl/test_obb_training.py
  • tests/ptl/test_oriented_network_capture.py
  • tests/ptl/test_segment_head_capture.py
  • tests/ptl/test_segment_head_capture_guards.py
  • docs/CHANGELOG.md
  • docs/ROADMAP.md
  • docs/TRAINING.md
  • docs/work_packages/wp-193c.md
  • docs/work_packages/wp-193d.md
  • docs/work_packages/wp-193e.md
  • docs/work_packages/wp-200.md
  • docs/engineering/phase-17-pose-objective-parity.md

Previous review (commit 97703ed)

This review did not finish. The model reached its output limit before it
could write the review — a reasoning model can spend the whole budget thinking.
Re-run the review, or lower the model's thinking effort, and it should get
further. Any inline comments below are from an earlier review.

Previous review (commit 97703ed)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (12 files)
  • CHANGELOG.md
  • docs/PROVENANCE.md
  • docs/ROADMAP.md
  • docs/TRAINING.md
  • docs/engineering/phase-17-pose-objective-parity.md
  • docs/work_packages/wp-193d.md
  • docs/work_packages/wp-199.md
  • docs/work_packages/wp-200.md
  • scripts/lint/audit_docs_present.py
  • src/lucid_yolo/ptl/module.py
  • src/lucid_yolo/ptl/network_capture.py
  • tests/ptl/test_network_capture.py
  • tests/ptl/test_oriented_network_capture.py

Reviewed by free · Input: 151.3K · Output: 16.9K · Cached: 823.8K

Derive the valid roadmap and header expectations from the configured floor.
Check rejection exactly one package below the floor.

WP: 200
Provenance: none (test fixture maintenance; no new sources)
Assumptions: none
Gate: scripts/_tests/test_audit_docs_present.py

Signed-off-by: Jesús Royeth <JESUSROYETH@users.noreply.github.com>
@Borda
Borda requested a balanced review from Copilot October 20, 2026 18:31
@Borda Borda added the enhancement New feature or request label Oct 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

CUDA forward/backward replay needs human review while the full gate, accelerator goldens and long-run accuracy remain unverified.

No open issues 🎉

What changed in this PR

Extends opt-in CUDA network replay to oriented detection training, preserving native loss computation and state updates.

Changes:

  • Records all six oriented head outputs, with native fallback for accumulation and modified objectives.
  • Adds replay parity, gradient, precision and fallback tests.
  • Documents measured performance, memory costs and validation limits.
File Description
tests/​ptl/​test_oriented_network_capture.py Adds oriented replay and fallback coverage.
tests/​ptl/​test_network_capture.py Updates supported-option expectations.
src/​lucid_yolo/​ptl/​network_capture.py Adds six-output oriented adapter.
src/​lucid_yolo/​ptl/​module.py Enables guarded oriented replay.
scripts/​lint/​audit_docs_present.py Raises work-package floor to 200.
scripts/​_tests/​test_audit_docs_present.py Makes floor fixtures dynamic.
docs/​work_packages/​wp-200.md Records implementation, measurements and limits.
docs/​work_packages/​wp-199.md Links dependent work package.
docs/​work_packages/​wp-193d.md Links dependent work package.
docs/​TRAINING.md Documents oriented replay requirements.
docs/​ROADMAP.md Adds WP-200 tracking.
docs/​PROVENANCE.md Records API provenance.
docs/​engineering/​phase-17-pose-objective-parity.md Includes WP-200 engineering log.
CHANGELOG.md Announces oriented replay support.

Review effort: Balanced

Borda added 4 commits October 20, 2026 23:24
---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Oriented network replay now asks only whether the loss objects and the callables they use are the native ones. A new _native_loss helper in loss_capture.py holds that identity check, and _source_signature calls it before its scalar setting checks, so its result is unchanged. The oriented guard calls _native_loss instead of wrapping _source_signature in a try/except, so a changed scalar loss setting such as an assigner epsilon no longer retires replay, and the retirement reason reads "the native loss or an assignment implementation changed".

The five oriented helper bindings are read by one _oriented_helpers function, used for both the import-time _ORIENTED_HELPERS snapshot and the per-call comparison.

[resolve group] PR Borda#23 — items 4 5

WP: 200
Provenance: none (review remediation; no new source)
Assumptions: none
Gate: tests/ptl/test_oriented_network_capture.py::test_oriented_loss_setting_change_keeps_replay; tests/ptl/test_oriented_network_capture.py::test_oriented_classification_override_stays_native; tests/ptl/test_oriented_network_capture.py::test_oriented_helper_replacement_stays_native

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
CPU tests now reach the CUDA-gated oriented route through CPU images that report is_cuda: an unbroken module hands all three stages to run_oriented, accumulation and an overridden objective each retire once and return the native forward, and plain CPU images never reach the recorder. A subclass override retires replay, the objective-override parametrization covers all five guarded methods, the packaged OBB recipe must pass the replay guard, and setup warns about the ignored flag only for the fit stage while make_graphed_callables is patched to fail.

The GPU trainer tests pin each retirement's reason, and the oriented parity oracle also compares LR-scheduler state, with lrf=0.1 so both arms run a decaying schedule.

The empty-target OBB training step now runs backward and asserts that every angle_stems parameter of both branches receives a gradient, and the network-replay constructor error is matched in full.

[resolve group] PR Borda#23 — items 3 9 6 7 8 10 11

WP: 200
Provenance: none (review remediation; no new source)
Assumptions: none
Gate: tests/ptl/test_oriented_network_capture.py; tests/ptl/test_obb_training.py::test_image_without_instances_gives_finite_loss; tests/ptl/test_network_capture.py::test_incompatible_options_are_explicit

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
TRAINING.md and the CHANGELOG entry limit the oriented speed claim to Nano OBB at batch 4, BF16 on one L4, and cite the 4.10% batch-8 fit-time change and the rise in peak reserved memory from 3974 to 4300 MiB. The network-capture docstrings cover the oriented path alongside detection and pose, and the WP-200 coverage list names gradient clipping and accumulation retirement instead of clipped accumulation.

[resolve group] PR Borda#23 — items 13 14 16

WP: 200
Provenance: none (wording of existing WP-200 measurements; no new source)
Assumptions: none
Gate: make gate (docs site and docstring doctests)

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
The 2026-10-20 R43 row cited the PyTorch 2.9 graphed-callable contract, which R43 does not register (only the 2.4 API documentation and the v2.4.0 LICENSE), and R43 already has its first-use row on 2026-10-16 in a log defined as first-use only. The row is removed; WP-200's use of R43 stays traced through its commit trailer.

[resolve group] PR Borda#23 — items 15

WP: 200
Provenance: none (usage-log correction; R43 register row unchanged)
Assumptions: none
Gate: pre-commit run --files docs/PROVENANCE.md; scripts/_tests/test_audit_docs_present.py

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@Borda
Borda merged commit 0c84549 into Borda:main Oct 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants