Repository navigation
Users/jha shweta1/rccl emfile kernel version - #13034
Open
JhaShweta1 wants to merge 9 commits into
Open
JhaShweta1 wants to merge 9 commits into
JhaShweta1 wants to merge 9 commits into
Conversation
ncclGetUniqueId() crashed in strstr when a failed open left the kernel version string null. Co-authored-by: Cursor <cursoragent@cursor.com>
The previous notes described a failed fopen as an empty string and treated a later retry as general. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
The empty-/proc/version test only checked the first failure, so a latched initResult would still pass. The changelog now names every public caller of ncclInit. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
ARGUS
Bottom line: Looks approval ready. The fix and the unlatched-retry test both check out; the open question from last round about failing init outright on an unreadable /proc/version is a design call worth a reply.
Comment @argus review to re-run, or @argus re-review after pushing.
The retry call was outside those hooks, so it read the host /proc/cmdline. Both calls now use the same fakes as the neighboring init tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
ARGUS
Bottom line: LGTM once the comment at init-test.cc:4570 is corrected: NCCLCHECKGOTO assigns unconditionally, so the second ncclInit() does not catch the poisoned initResult the comment claims it does.
Comment @argus review to re-run, or @argus re-review after pushing.
The second ncclInit() shows initOnceFlag was left unlatched. NCCLCHECKGOTO overwrites initResult, so that call does not catch a poisoned error. Co-authored-by: Cursor <cursoragent@cursor.com>
…ccl-emfile-kernel-version Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # projects/rccl/CHANGELOG.md
Contributor
pr-rocm-systems-13034-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13034-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ncclGetUniqueId() crashed in strstr when a failed open left the kernel version string null.
Motivation
ncclGetUniqueId() terminated with SIGSEGV when /proc/version could not be opened, including when the process had exhausted its file descriptors. Initialization should return an ncclResult_t so the caller can recover.
Technical Details
ncclOsTopoGetStrFromSys() reports a failed read as success plus an empty string. ncclInit() then tokenized that buffer, got a null version string, and passed it to strstr. The null check now returns ncclSystemError before the strstr call and before call_once, so a later initialization can succeed once file descriptors are available.
Issue Tracking
JIRA ID: ROCM-31881
Test Plan
Test Result
All four InitMicrotestIsolated.NcclInit_* tests passed, including the empty-version case and a following successful ncclInit(). On unmodified develop, the empty-version case terminated with signal 11.
Submission Checklist