[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

wengh · 2025-03-26T17:59:46Z

Continuing @allisonwang-db's work on #50373 and #49471

What changes were proposed in this pull request?

This PR adds support for DEFAULT arguments in SQL UDF. Examples:

CREATE FUNCTION foo1d1(a INT DEFAULT 10) RETURNS INT RETURN a;
SELECT foo1d1();   -- 10
SELECT foo1d1(20); -- 20

CREATE FUNCTION foo1d6(a INT, b INT DEFAULT 7) RETURNS TABLE(a INT, b INT) RETURN SELECT a, b;
SELECT * FROM foo1d6(5);    -- 5, 7
SELECT * FROM foo1d6(5, 2); -- 5, 2

See sql-udf.sql for more valid and invalid examples.

Why are the changes needed?

To support default arguments in SQL UDFs.

Does this PR introduce any user-facing change?

Yes. Now SQL UDFs support DEFAULT arguments.

A side effect of the grammar change is that some invalid function parameter definitions are now no longer rejected by the grammar, but instead rejected by the parser logic.

Examples:

-- multiple COMMENT or multiple NOT NULL
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a;

-- before:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'COMMENT'. SQLSTATE: 42601
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- after:
[CREATE_TABLE_COLUMN_DESCRIPTOR_DUPLICATE] CREATE TABLE column a specifies descriptor "COMMENT" more than once, which is invalid. SQLSTATE: 42710
== SQL (line 1, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world')...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- GENERATED ALWAYS AS
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;

-- before:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'GENERATED'. SQLSTATE: 42601
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- after:
[INVALID_SQL_SYNTAX.CREATE_FUNC_WITH_GENERATED_COLUMNS_AS_PARAMETERS] Invalid SQL syntax: CREATE FUNCTION with generated columns as parameters is not allowed. SQLSTATE: 42000
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This doesn't change the behavior of existing valid SQL.

How was this patch tested?

End-to-end regression tests in sql-udf.sql and simple tests in SQLFunctionSuite.

Was this patch authored or co-authored using generative AI tooling?

No

wengh · 2025-03-27T15:55:05Z

@cloud-fan @allisonwang-db Please take a look at this follow up of [SPARK-50763][SQL] Add Analyzer rule for resolving SQL table functions!

allisonwang-db

Looks great!

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserInterface.scala

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

wengh

explain refactor

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

cloud-fan · 2025-03-30T12:47:03Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

@@ -45,6 +45,70 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
    withOrigin(ctx)(StructType(visitColTypeList(ctx.colTypeList)))
  }

+  override def visitSingleRoutineParamList(ctx: SingleRoutineParamListContext): StructType = {


where do we fail this method if unsupported feature such as generated column is specified?

It's checked in SparkSqlParser.visitCreateUserDefinedFunction. I should add comment in visitStructField to explain

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala

cloud-fan · 2025-04-01T01:24:16Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

+   * Create a [[StructField]] from a column definition which allows options like COMMENT and
+   * DEFAULT.
+   *
+   * Don't handle generation expression since this function is currently only used for creating


why don't we reject it here within this function?

we reject it in SparkSqlParser to report the right ParserRuleContext

cloud-fan · 2025-04-01T08:28:51Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

+   * SQL functions which don't support generation expressions. The rejection logic is in
+   * [[SqlBaseParserVisitor#visitCreateUserDefinedFunction()]] implementation.
+   */
+  private def visitStructField(


This method looks very similar to visitColDefinition. shall we invoke visitColDefinition here, fail if ColumnDefinition#generationExpression/identityColumnSpec is defined, and call ColumnDefinition#toV1Column to get StructField.

cloud-fan · 2025-04-01T08:29:38Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala

@@ -50,6 +50,10 @@ abstract class AbstractParser extends DataTypeParserInterface with Logging {
    astBuilder.visitSingleTableSchema(parser.singleTableSchema())
  }

+  override def parseRoutineParam(sqlText: String): StructType = parse(sqlText) { parser =>


so the main difference between this and parseTableSchema is: it supports default expression, and it fails for unsupported fields.

github-actions bot added SQL CONNECT labels Mar 26, 2025

wengh changed the title ~~[WIP][SPARK-51439] Support SQL UDF with DEFAULT argument~~ [WIP][SPARK-51439][SQL] Support SQL UDF with DEFAULT argument Mar 26, 2025

wengh marked this pull request as ready for review March 27, 2025 15:05

wengh changed the title ~~[WIP][SPARK-51439][SQL] Support SQL UDF with DEFAULT argument~~ [SPARK-51439][SQL] Support SQL UDF with DEFAULT argument Mar 27, 2025

allisonwang-db and others added 6 commits March 27, 2025 09:01

fix

24a697d

Implement SQL UDF default argument

2a8fa60

fix

b1b35a1

fix test

18a4a05

fix

b2173ad

fix

aee7338

wengh force-pushed the sql-udf-default branch from ecfb9ce to aee7338 Compare March 27, 2025 16:01

allisonwang-db approved these changes Mar 28, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserInterface.scala Outdated Show resolved Hide resolved

wengh marked this pull request as draft March 28, 2025 00:31

allisonwang-db reviewed Mar 28, 2025

View reviewed changes

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 Outdated Show resolved Hide resolved

cloud-fan reviewed Mar 28, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Outdated Show resolved Hide resolved

wengh added 4 commits March 28, 2025 08:25

WIP

fdfbd2c

use colDefinitionList

90c6b8a

fix

983f057

fix tests

8c73a14

wengh force-pushed the sql-udf-default branch from 8f51e83 to 8c73a14 Compare March 28, 2025 16:48

add tests and fix CREATE_FUNC_WITH_GENERATED_COLUMNS_AS_PARAMETERS

8722fc6

wengh requested review from allisonwang-db and cloud-fan March 28, 2025 17:13

wengh marked this pull request as ready for review March 28, 2025 17:13

refactor

66635dc

wengh commented Mar 28, 2025

View reviewed changes

comment

9478651

cloud-fan reviewed Mar 30, 2025

View reviewed changes

zhengruifeng reviewed Mar 31, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala Show resolved Hide resolved

add comment

8fcadae

wengh requested review from cloud-fan and zhengruifeng April 1, 2025 00:22

cloud-fan reviewed Apr 1, 2025

View reviewed changes

wengh added 2 commits April 1, 2025 09:01

add comment

93f1422

reuse parseColDefinition

b61bd41

wengh requested a review from cloud-fan April 2, 2025 23:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

wengh commented Mar 26, 2025 •

edited

Loading

wengh commented Mar 27, 2025

allisonwang-db left a comment

wengh left a comment •

edited

Loading

cloud-fan Mar 30, 2025

wengh Mar 31, 2025 •

edited

Loading

wengh Mar 31, 2025

cloud-fan Apr 1, 2025

wengh Apr 1, 2025 •

edited by cloud-fan

Loading

cloud-fan Apr 1, 2025

wengh Apr 2, 2025

cloud-fan Apr 1, 2025

wengh Apr 1, 2025

[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

Are you sure you want to change the base?

[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

Conversation

wengh commented Mar 26, 2025 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

wengh commented Mar 27, 2025

allisonwang-db left a comment

Choose a reason for hiding this comment

wengh left a comment • edited Loading

Choose a reason for hiding this comment

cloud-fan Mar 30, 2025

Choose a reason for hiding this comment

wengh Mar 31, 2025 • edited Loading

Choose a reason for hiding this comment

wengh Mar 31, 2025

Choose a reason for hiding this comment

cloud-fan Apr 1, 2025

Choose a reason for hiding this comment

wengh Apr 1, 2025 • edited by cloud-fan Loading

Choose a reason for hiding this comment

cloud-fan Apr 1, 2025

Choose a reason for hiding this comment

wengh Apr 2, 2025

Choose a reason for hiding this comment

cloud-fan Apr 1, 2025

Choose a reason for hiding this comment

wengh Apr 1, 2025

Choose a reason for hiding this comment

wengh commented Mar 26, 2025 •

edited

Loading

wengh left a comment •

edited

Loading

wengh Mar 31, 2025 •

edited

Loading

wengh Apr 1, 2025 •

edited by cloud-fan

Loading