-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Support UNKNOWN type for Spark collect_set aggregate function #12013
base: main
Are you sure you want to change the base?
Conversation
✅ Deploy Preview for meta-velox canceled.
|
@zhli1142015 @rui-mo Could you please help review this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
velox/functions/sparksql/aggregates/tests/CollectSetAggregateTest.cpp
Outdated
Show resolved
Hide resolved
velox/functions/sparksql/aggregates/tests/CollectSetAggregateTest.cpp
Outdated
Show resolved
Hide resolved
velox/functions/sparksql/aggregates/tests/CollectSetAggregateTest.cpp
Outdated
Show resolved
Hide resolved
@WangGuangxin, please revise the title to replace "null type" with "UNKNOWN type", since null type is not belonging to velox. |
Thanks, fixed |
@@ -260,5 +260,23 @@ TEST_F(CollectSetAggregateTest, rowWithNestedNull) { | |||
{data}, {}, {"collect_set(c0)"}, {"spark_array_sort(a0)"}, {expected}); | |||
} | |||
|
|||
TEST_F(CollectSetAggregateTest, nullType) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: nullType->unknownType
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, Thanks.
@WangGuangxin Could you rebase this PR to get rid of the fuzzer failure? |
f8059aa
to
b6ba8b2
Compare
done |
Gluten maps Spark NullType as UNKNOWN type in Velox. This PR supports the
UNKNOWN type for the Spark collect_set aggregate function.