Repository navigation
ci(rccl): build the coco PR baseline at the PR's merge-base - #14105
Open
speriaswamy-amd wants to merge 1 commit into
Open
speriaswamy-amd wants to merge 1 commit into
speriaswamy-amd wants to merge 1 commit into
Conversation
The coco PR gate built its develop baseline at develop tip on every run, so changes merged after a PR branched read as the PR's own. Resolve the PR's merge-base with its base branch and pass it as --baseline-sha when the coco harness supports the flag, warning when it does not. Signed-off-by: speriaswamy-amd <surya.periaswamy@amd.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The coco PR gate built its develop baseline at develop tip on every run, because each run uses a fresh output directory. A PR was therefore measured against code it did not contain. In 18 replayed ruby64 runs, seven unrelated PRs showed another commit's 1-node reduce_scatter change (#12928) as their own.
Technical Details
rccl_resolve_coco_run.pynow, for PR runs:GET /pulls/{n}) and its merge-base with the head (GET /compare/{base}...{head}), using the job'sGITHUB_TOKEN;--baseline-sha <merge-base>tococo_args, which already flows into both the build-only and the testcoco prcalls;coco pr --baseline-sha, ROCm/tRCCL), and otherwise emits a workflow warning. This change is therefore safe to land before the harness release that adds the flag;Periodic runs are unchanged. The flag ships in coco 1.4, so pinning starts when the gate's default
coco_refmoves from1.3.xto1.4.x. Until then, runs log the warning and behave as today.Issue Tracking
JIRA ID: AICOMRCCL-2204
Test Plan
.github/scripts/tests/rccl_resolve_coco_run_test.py(21 pass):a22b7e7847, matchinggh api compare.Test Result
All pass. A ruby64 gate run on a PR, once the harness has the flag, should show
--baseline-shain thecoco prcommand line and an RCCL version line at the merge-base on the baseline stack.Submission Checklist