[MINOR][CI] Stop unrelated PR comments from cancelling Delta and ANSI runs - #12906
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes an Actions concurrency edge case in delta_spark_ut.yml where any issue_comment event (including normal review comments) could land in the same concurrency group as a PR’s in-flight Delta Spark UT run and cancel it due to cancel-in-progress: true. The change makes cancellation conditional so only a real /delta-test command comment can cancel an in-flight run for that PR, preserving the intended “new push cancels old run” and “explicit rerun command cancels old run” behavior.
Changes:
- Replace unconditional
concurrency.cancel-in-progress: truewith a conditional expression that only enables cancellation for real/delta-testcomment commands. - Keep the concurrency group key unchanged, but align the cancellation condition with the existing
jobs.delta-test-requested.iflogic to avoid unintended cancellations.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Thanks for fixing it @LuciferYang. I wonder if the same happens to I think we don't have a good way to tests your changes before merging it. But it looks good to me |
|
🔄 ANSI mode analysis started by @felipepessoto. View run |
|
Test checking if ansi is cancelled UPDATE: yes it does: https://github.com/apache/gluten/actions/runs/33043165559 |
ANSI Mode Test Analysis Report (Spark 4.1)Note Expression-level ANSI mode offload coverage analysis.
ANSI Offload suites: 0 tests, 0 records | Other suites: 0 tests ANSI OffloadOverview (ANSI Offload Expression Records)
|
Conditioning cancel-in-progress instead would leave the run pending until the in-flight suite ends, up to timeout-minutes: 350, showing as an unresolved check the whole time. Taking it out of the group keeps today's behaviour for the two cases the group exists for and removes only the collateral cancellation.
|
Created the issue #12908 in case you are interested in fixing it as well |
Keeps the group name byte-identical to main for every other trigger, so PRs open at merge time do not land in a window where a push run and a /delta-test run sit in different groups and stop superseding each other.
Same shape as delta_spark_ut.yml: any comment on a PR joined the ANSI group and cancelled an in-flight /ansi-test or /ansi-analyze run, then skipped in check-comment. Fixes the case reported in apache#12908.
|
Good catch, thanks. Fixed in this PR too, at I took the second of your two options for both files, for the reason you gave: a unique group fully isolates the unrelated comment, while conditioning The suffix is appended only when the comment is not a command, so for The negated condition is Three properties are checkable without running anything, and I verified them for both files by parsing the YAML rather than reading it: You are right that neither fix can be exercised before merging. |
|
🔄 ANSI mode analysis started by @LuciferYang. View run |
Two complete eight-shard runs on this branch report the same 32 now-passing tests, byte-identical lists, with Stale and Skipped both zero. Deleted by hand rather than committing the regenerated artifact, which also replaces the hand-written header with an auto-generated stub.
| org.apache.spark.sql.delta.ConvertToDeltaSQLSuite#external tables use correct path scheme | ||
| org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#external tables use correct path scheme | ||
| org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row per file - Partitioned = false, cdfEnabled = false | ||
| org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row per file - Partitioned = false, cdfEnabled = true | ||
| org.apache.spark.sql.delta.DeltaAllFilesInCrcSuite#test all-files-in-crc verification failure also triggers and logs incremental-commit verification result |
| # Cancel older `pull_request` and `/delta-test` runs for the same PR. | ||
| # | ||
| # `issue_comment` fires on every comment posted on an issue or a PR, and | ||
| # `github.event.issue.number` is that issue's number, which for a comment on a | ||
| # PR is the PR's own number. Such a run would therefore join the PR's group and | ||
| # cancel an in-flight suite while `delta-test-requested` below skips it, losing | ||
| # the suite and replacing it with nothing. Give only that case a group of its | ||
| # own, so every other trigger keeps the exact group name it has today. The | ||
| # negated condition is that job's `if` verbatim. |
|
cc @jackylee-ch CI paased |
|
Thank you @jackylee-ch |
What changes are proposed in this pull request?
delta_spark_ut.ymltriggers onpull_requestand onissue_comment: types: [created], which fires for every comment posted on an issue or a pull request (review comments on the diff and commit comments use other events). For anissue_commenteventgithub.event.issue.numberis that issue's number, which for a comment on a PR is the PR's own number, so the comment run lands in the same concurrency group as that PR's in-flightpull_requestrun andcancel-in-progress: truekills it. The comment run then does not replace it:delta-test-requestedskips anything that is not a/delta-testcommand, so the suite is lost and nothing takes its place.The same shape is in
velox_backend_ansi.yml: its group is${{ github.repository }}-ansi-${{ github.event.issue.number || inputs.pr_number }}withcancel-in-progress: true, and its command test lives injobs.check-comment.if, which is again evaluated after the group. Both files are fixed here the same way.Fixes #12908.
Neither case is hypothetical. On #12902 I posted an unrelated review comment at 03:37:39Z. It started
issue_commentrun 33036933287 at 03:37:43Z, which cancelledpull_requestrun 33034774777 forty-three minutes into the tests. From shard 7's log:All eight shards and the aggregate job died within thirty seconds of each other, and the run that killed them concluded
skipped. The signal lost was a real one: every failing test in that shard up to the cancellation was in the baseline at the time, so nothing had regressed and the shard was on course to pass.For ANSI, @felipepessoto reproduced it deliberately on this PR while reviewing it: a
/ansi-testcomment at 05:38:58Z started run 33043165559, an unrelated follow-up comment at 05:39:40Z created run 33043202388 in the same group two seconds later, and the original run'sbuild-native-libwas cancelled at 05:39:59Z while every job in the replacement was skipped. #12908 has the full trace.The fix appends a discriminator to the group for that case only. When the comment is not a command the group gains a
-comment-<run_id>suffix and the run therefore neither cancels nor queues; for every other trigger the appended segment evaluates to the empty string, so the group name stays byte-identical to what it is today andcancel-in-progressstaystrue.workflow_dispatchkeeps its group too, so a manual ANSI re-dispatch still supersedes an older one for the same PR, which is one of the behaviours #12908 asked to preserve.In each file the negated condition is the gate job's
ifrepeated verbatim:delta-test-requestedfor Delta,check-commentfor ANSI. One difference between the two is worth naming:check-commenttests the body withcontains, not a first-token match, so a comment that merely mentions/ansi-testcounts as a command. Copying the condition verbatim keeps the grouping and the gate in agreement on that, rather than making them disagree in a new way. Whether thecontainstest itself should be tightened is a separate question and not changed here.Two alternatives I discarded, both of which I had implemented before settling here, in case a reviewer wonders why the expression is shaped this way.
Making
cancel-in-progressconditional instead leaves the group alone but does not let the comment run proceed: runs in one group never overlap, so it sitspendinguntil the whole suite finishes, then starts and skips anyway. It also occupies the group's single pending slot in the meantime.Emitting a
-sharedsuffix for the normal cases rather than an empty one reads better but renames the group for every trigger.pull_requestruns use the workflow file from the PR head whileissue_commentruns always use the default branch's, so for PRs already open at merge time a push run and a/delta-testrun would sit in differently-named groups and stop superseding each other until the PR picks the change up. Appending nothing in those cases avoids the window entirely.How was this patch tested?
The behaviour cannot be exercised from this PR. Per the
issue_commentdocs, "This event will only trigger a workflow run if the workflow file exists on the default branch", withGITHUB_REFset to the default branch, so a comment posted here runs main's copy of the workflow no matter what this branch says. That is also why the run that cancelled #12902's suite used the unfixed version. It takes effect once merged.Three properties are worth checking mechanically rather than by eye, and I checked them for both files by parsing the YAML rather than reading it:
concurrency.cancel-in-progresstrue, unchanged from main!( ... )vs the gate job'sifResolved group per trigger, worked through by hand from the precedence rules (
&&binds tighter than||, and both yield values):pull_request,workflow_dispatchand a real command comment all take the|| ''branch and keep today's group; any other comment takesformat('-comment-{0}', github.run_id). A comment on a plain issue also gets the suffix, sincegithub.event.issue.pull_requestis absent, which is harmless because such a run skips too.I cannot run GitHub's expression evaluator locally, so evaluation itself rests on the docs plus the fact that each file's gate job already evaluates the identical condition on every comment. A malformed expression fails the run with a workflow syntax error rather than mis-grouping silently. If it evaluated to the wrong branch for
pull_requestevents, the symptom would be a group suffixed with-comment-<id>, visible in the Actions UI, and the cost would be pushes no longer superseding older runs rather than a wrong result.The Delta suite is green on this head: run 33148690783 on
6eec515cd, all eight shards plus the aggregate. Earlier heads of this branch showed four red shards for a reason unrelated to the change, recorded here so the history reads correctly:known-failures.txtlisted 32 column-mapping tests as expected failures that had started passing, andfail_on_fixedresolves totrueforpull_requestevents by design, so the gate reported them asNOW-PASSING. @felipepessoto removed exactly those 32 entries on main in #12907, so this branch does not touch the baseline.One flake to know about, seen once in four complete runs on this branch:
DeltaCDCStreamWithCatalogManagedBatch100Suite#maxFilesPerTrigger with Trigger.AvailableNow respects read limitswas reported as a regression by run 33044870485 and by none of 33043547622, 33141742435 or 33148690783.flaky-tests.txthas no active entries, so if it recurs it will red a shard.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5