Skip to content

fix(ptl): guard segment replay drain on failed fit - #27

Merged
Borda merged 3 commits into
mainfrom
fix/segment-replay-held-work
Oct 21, 2026
Merged

Borda merged 3 commits into
mainfrom
fix/segment-replay-held-work

Conversation

@Borda

@Borda Borda commented Oct 21, 2026

Copy link
Copy Markdown
Owner

_SegmentHeadCapture.clear() now resets the holder before draining the recorded streams, so a failing synchronize still leaves no recordings behind. _SegmentHeadCaptureLifecycle.on_exception reports that drain failure as a warning, so the trainer's own exception reaches the caller and its teardown runs; on_fit_end still raises it. _holder returns None for a module without a segmentation holder and every hook is then a no-op, as in network_capture.py.

Guard test names drop filler words, and the stage-helper certificate test lists proto_fusion among the certified stages.

WP: 193e
Provenance: none (review remediation of replay lifecycle error handling; repository code only)
Assumptions: none
Gate: tests/ptl/test_segment_head_capture_guards.py::TestRecordingLifetime, tests/ptl/test_segment_head_capture.py::test_stage_helpers_and_shared_storage_are_not_certified

`_SegmentHeadCapture.clear()` now resets the holder before draining the recorded streams, so a failing synchronize still leaves no recordings behind. `_SegmentHeadCaptureLifecycle.on_exception` reports that drain failure as a warning, so the trainer's own exception reaches the caller and its teardown runs; `on_fit_end` still raises it. `_holder` returns None for a module without a segmentation holder and every hook is then a no-op, as in `network_capture.py`.

Guard test names drop filler words, and the stage-helper certificate test lists `proto_fusion` among the certified stages.

WP: 193e
Provenance: none (review remediation of replay lifecycle error handling; repository code only)
Assumptions: none
Gate: tests/ptl/test_segment_head_capture_guards.py::TestRecordingLifetime, tests/ptl/test_segment_head_capture.py::test_stage_helpers_and_shared_storage_are_not_certified

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@Borda
Borda requested a balanced review from Copilot October 21, 2026 20:03
@Borda Borda added the bug Something isn't working label Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Promoted warnings can still mask the original training exception and interrupt teardown.

1 open issue
What changed in this PR

This PR hardens segmentation replay cleanup after failed training while preserving drain errors at normal fit completion.

Changes:

  • Reset capture state before draining streams and warn on failed-fit drain errors.
  • Make lifecycle hooks no-ops for modules without a segmentation holder.
  • Add lifecycle regression tests, simplify test names, and include proto_fusion in certification coverage.
File Description
tests/​ptl/​test_segment_head_capture.py Include proto_fusion in stage certification.
tests/​ptl/​test_segment_head_capture_guards.py Cover drain failures and missing holders; simplify names.
src/​lucid_yolo/​ptl/​segment_head_capture.py Harden capture reset and lifecycle cleanup.

Review effort: Balanced

Comment thread src/lucid_yolo/ptl/segment_head_capture.py Outdated
Borda and others added 2 commits October 21, 2026 22:11
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
`_SegmentHeadCaptureLifecycle.on_exception` replaces the `try`/`except UserWarning: pass` around its drain-failure warning with `contextlib.suppress(UserWarning)`; a warning filter that turns the warning into an error still cannot replace the trainer's original exception.

WP: 193e
Provenance: none (behaviour-preserving restructuring; repository code only)
Assumptions: none
Gate: tests/ptl/test_segment_head_capture_guards.py::TestRecordingLifetime

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@Borda
Borda merged commit 8276de7 into main Oct 21, 2026
13 of 14 checks passed
@Borda
Borda deleted the fix/segment-replay-held-work branch October 21, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants