Skip to content

fix(cc): retain sealed selections across infrastructure retries - #4628

Open
Epsilon006 wants to merge 1 commit into
NVIDIA-NeMo:aroshanghias/cc-capture-redesign-20260914from
Epsilon006:fix/4557-cc-infra-retry
Open

Epsilon006 wants to merge 1 commit into
NVIDIA-NeMo:aroshanghias/cc-capture-redesign-20260914from
Epsilon006:fix/4557-cc-infra-retry

Conversation

@Epsilon006

@Epsilon006 Epsilon006 commented Oct 9, 2026 •

Copy link
Copy Markdown

Transient forest-training rollout failures drop the prompt group on the first infrastructure error despite a larger retry budget. Retain selections for sealed siblings across live retries, discard abandoned selections, and honor the existing budget and backoff. Prompt-group recovery reruns the abandoned cohort; sibling recovery reruns only missing siblings.

Closes #4557. Stacked on #4124; the target remains aroshanghias/cc-capture-redesign-20260914.

This PR is now a single signed-off commit based directly on the #4124 foundation head (2f3649d75bbe85724984a49fb6a2da575777a157). The diff against that foundation changes only the retry logic, tests, and documentation (4 files, +339/-15); no dependency, lockfile, or Gym pin changes.

Validation

  • Affected dispatch and CPU pipeline tests: 49 passed, 4 failed. All four padding/router-validation failures reproduce on the unmodified foundation.
  • Expanded capture/finalization/selection/ingestion/exporter tests: 258 passed, 9 skipped, 6 failed. The same four padding failures and two Gym OpenCode test-double failures (SimpleNamespace.timeout) reproduce on the unmodified foundation.
  • Both recovery granularities crossed with both returned_history and all_captured pass the regression: complete owners and all four rows reach the controller train pump with exact tokens and logical GRPO advantages.
  • Pinned Ruff 0.9.9 lint, import ordering and formatting pass. Pyrefly 0.24.2 on the modified source reports 0 errors and the same 12 ignored diagnostics as the foundation. The PR diff has no whitespace errors.

Tests ran in an isolated Linux Docker container with Python 3.13.14, CPU torch 2.13.0, Lens b0f977d4, and Gym infrastructure #4128 (b53e7b7e) combined locally with the native client from #3368 (507d15c3). CPU tests exclude repository-wide Ray cluster/monitor fixtures and use generation, storage and learner doubles. No GPU training or full distributed qualification is claimed. Final paired-dependency integration and NVIDIA runner approval remain maintainer gates.

Live retries can repeat external agent side effects. Forest-training checkpoint/resume remains unsupported; an exhausted group still prevents a complete training step.

Hosted checks and remaining gates

For the current head 1f01a10f008e14f1aece46c36709ed2434e4d5ca, DCO has passed, and the lightweight semantic PR-title, triage, and submodule-comment checks have passed. Other hosted workflow runs still report action_required; NVIDIA runner vetting and approval to run heavyweight CI remain outstanding. GPU training and full distributed validation have not been completed. This PR remains stacked on and dependent on the unmerged #4124 foundation. Final paired-dependency integration and maintainer approval remain merge gates.

@copy-pr-bot

copy-pr-bot Bot commented Oct 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 9, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 11, 2026
@aroshanghias-nvd
aroshanghias-nvd force-pushed the aroshanghias/cc-capture-redesign-20260914 branch from 41f2622 to dae5a4a Compare October 13, 2026 22:46
Signed-off-by: Epsilon006 <199138677+Epsilon006@users.noreply.github.com>
@Epsilon006
Epsilon006 force-pushed the fix/4557-cc-infra-retry branch from 2898a6e to 1f01a10 Compare October 18, 2026 07:13
@Epsilon006

Copy link
Copy Markdown
Author

@terrykong — this PR addresses #4557 and is stacked on #4124. Could you advise on the next validation step and help route the NVIDIA runner vetting request to the appropriate maintainer?

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Oct 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants