[SPARK-52895][SQL] Don't add duplicate elements in resolveExprsWithAggregate
#51567
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mihailotim-db
force-pushed
the
mihailotim-db/deduplicate_agg_exprs
branch
2 times, most recently
from
7a473ce to
c34456e
Compare
mihailotim-db
changed the title
resolveExprsWithAggregate
mihailotim-db
force-pushed
the
mihailotim-db/deduplicate_agg_exprs
branch
from
c34456e to
3a3b774
Compare
cloud-fan
reviewed
Jul 21, 2025
| @@ -2919,21 +2919,21 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor | |||
| def resolveExprsWithAggregate( | |||
| exprs: Seq[Expression], | |||
| agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = { | |||
| val extraAggExprs = ArrayBuffer.empty[NamedExpression] | |||
| val extraAggExprs = new LinkedHashSet[NamedExpression] | |||
There was a problem hiding this comment.
shall we use ExpressionSet which deduplicates expressions by their semantics instead of object equality?
There was a problem hiding this comment.
This is a good point, but we can't actually use ExpressionSet since we need to keep the ordering in aggregate list deterministic. I actually found a separate bug here: #51557 and if we use a LinkedHashMap with cannonicalized expression as key, we can solve both issues.
mihailotim-db
force-pushed
the
mihailotim-db/deduplicate_agg_exprs
branch
from
3a3b774 to
d423f26
Compare
mihailotim-db
force-pushed
the
mihailotim-db/deduplicate_agg_exprs
branch
from
d423f26 to
4c24e34
Compare
cloud-fan
approved these changes
Jul 21, 2025
|
thanks, merging to master! |
haoyangeng-db
pushed a commit
to haoyangeng-db/apache-spark
that referenced
this pull request
Jul 22, 2025
…ggregate` ### What changes were proposed in this pull request? Don't add duplicate elements in `resolveExprsWithAggregate`. ### Why are the changes needed? This is needed in order to resolve plan mismatches between fixed-point and single-pass analyzer. At the moment fixed-point duplicates columns if there are duplicate columns missing in HAVING/ORDER BY. However, if there are LCAs, fixed-point will deduplicate these columns because LCA resolution uses a set (and LCA resolution runs after ORDER BY/HAVING resolution in fixed-point). In single-pass LCA resolution is done first and only after comes ORDER BY/HAVING resolution which adds duplicates. This PR makes behavior consistent across all cases by never adding duplicates. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new test cases to golden files. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51567 from mihailotim-db/mihailotim-db/deduplicate_agg_exprs. Authored-by: Mihailo Timotic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Don't add duplicate elements in
resolveExprsWithAggregate.Why are the changes needed?
This is needed in order to resolve plan mismatches between fixed-point and single-pass analyzer. At the moment fixed-point duplicates columns if there are duplicate columns missing in HAVING/ORDER BY. However, if there are LCAs, fixed-point will deduplicate these columns because LCA resolution uses a set (and LCA resolution runs after ORDER BY/HAVING resolution in fixed-point). In single-pass LCA resolution is done first and only after comes ORDER BY/HAVING resolution which adds duplicates. This PR makes behavior consistent across all cases by never adding duplicates.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added new test cases to golden files.
Was this patch authored or co-authored using generative AI tooling?
No