[SPARK-53687][SQL][SS][SDP] Introduce WATERMARK clause in SQL statement #52428

HeartSaVioR · 2025-09-24T01:46:58Z

What changes were proposed in this pull request?

This PR proposes to introduce WATERMARK clause in SQL statement. WATERMARK clause is to define the watermark against the relation, especially streaming relation where STREAM keyword is added to the relation (or table valued function).

Please refer to the SQL reference doc for WATERMARK clause about its definition.
https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-watermark

The PR also contains new tests which show how to use it.

Why are the changes needed?

This is needed to unblock the stateful workload in Streaming Table & Flow being described as SQL statement.

Does this PR introduce any user-facing change?

Yes, users can define the watermark for stateful workloads with SQL statement.

How was this patch tested?

New UTs.

Was this patch authored or co-authored using generative AI tooling?

No.

HeartSaVioR · 2025-09-24T01:50:43Z

cc. @sryza @cloud-fan @viirya
Could you please take a look? Thanks!

Also, do we have test purposed TVF definition for "streaming source"? I couldn't add E2E test for STREAM TVF + WATERMARK and it'd be ideal if we could add one.

HeartSaVioR · 2025-09-24T01:53:06Z

cc. @anishshri-db as well

HeartSaVioR · 2025-09-24T01:53:22Z

common/utils/src/main/resources/error/error-conditions.json

      "Doesn't support month or year interval: <interval>"
    ]
  },
+  "_LEGACY_ERROR_TEMP_3263" : {


TODO: Need to assign one error class

TODO: remove this

cloud-fan · 2025-09-24T08:07:50Z

...g/apache/spark/sql/execution/streaming/EventTimeWatermarkWithWatermarkDefInSelectSuite.scala

+        |FROM
+        |    stream_left WATERMARK leftTime DELAY OF INTERVAL 0 second
+        |FULL OUTER JOIN
+        |    stream_right WATERMARK rightTime DELAY OF INTERVAL 0 second


do we require the watermark to be the same between two join sides? what's the corresponding dataframe query?

No, that's just for simplification. You can have different definition of watermark including delay threshold. e.g. you can define watermark delay to 10 minutes in stream_left while defining delay to 5 minutes in stream_right.

The corresponding DataFrame query is following:

stream_left <- streaming DF
stream_right <- streaming DF

val leftDf = stream_left.withWatermark("leftTime, "0 second") val rightDf = stream_right.withWatermark("rightTime", "0 second") val query = leftDf.join( rightDf, expr("leftKey = rightKey AND leftTime BETWEEN rightTime - INTERVAL 5 SECONDS"), "full_outer") query.writeStream.blabla.start()

I have the impression that, with DataFrame API, we can set watermark for all streaming operators, not only the streaming scans. Is this true?

You can set watermark "before" the first stateful operator has appeared. But conceptually, it is advised to define watermark closer to stream scan. The concept of watermark is to mark that the engine has seen (processed) all the data before the timestamp ts, which is closely related to the input data from stream scan.

viirya · 2025-09-24T16:23:48Z

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

-      optionsClause? sample? tableAlias                     #tableName
-    | LEFT_PAREN query RIGHT_PAREN sample? tableAlias       #aliasedQuery
-    | LEFT_PAREN relation RIGHT_PAREN sample? tableAlias    #aliasedRelation
+      optionsClause? sample? watermarkClause? tableAlias    #tableName


watermarkClause is already defined in streamRelationPrimary. Do we still need it here? Is it also applied for non-stream relation?

relation here can be a temp view which could be technically streaming without STREAM keyword

viirya · 2025-09-24T16:41:02Z

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

    : funcName=functionName LEFT_PAREN
      (functionTableArgument (COMMA functionTableArgument)*)?
-      RIGHT_PAREN tableAlias
+      RIGHT_PAREN watermarkClause? tableAlias


From the doc https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-watermark, it looks like table_valued_function has no watermark_clause support, but we want to have it here?

Ouch I think it's missed in that doc.

cloud-fan · 2025-09-25T10:29:57Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

+    }
+
+    // TODO: This seems to need to find the new expression by index. We can't rely on exprId
+    //  due to UnresolvedAlias. Are there better ways to do?


I think we need a new logical plan like UnresolvedEventTimeWatermark, which will be rewritten into EventTimeWatermark + Project during analysis.

Addressed. PTAL!

HeartSaVioR · 2025-09-26T02:12:35Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveEventTimeWatermark.scala

+        case e: Expression => UnresolvedAlias(e)
+      }
+
+      val isAttributeReference = namedExpression.isInstanceOf[AttributeReference]


@cloud-fan
Please help verify what I'm doing is correct.

What EventTimeWatermark node does is simply to attach metadata marker against the column (produced by child.output) referred by the expression. The node itself does not do any projection and just updates the accumulators and does the passthrough. If the eventTimeColExpr has to be evaluated, we have to inject Project to evaluate the expression, which we expect the result to be explicitly named.

Hence I skip injecting Project only if the expression is an AttributeReference and the column is available in the child.output, otherwise I simply add Project.

it doesn't matter. We can also always add the Project, as optimizer can remove it.

cloud-fan · 2025-09-26T12:10:58Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveEventTimeWatermark.scala

+  override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning(
+    _.containsPattern(TreePattern.UNRESOLVED_EVENT_TIME_WATERMARK), ruleId) {
+
+    case u: UnresolvedEventTimeWatermark if u.childrenResolved =>


shall we wait for u.eventTimeColExpr.resolved?

cloud-fan · 2025-09-26T12:11:52Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveEventTimeWatermark.scala

+      }
+
+      val isAttributeReference = namedExpression.isInstanceOf[AttributeReference]
+      val exprInChildOutput = u.child.output.exists(_.name == namedExpression.name)


it's weird to search by name for resolved attributes, we can just do u.child.outputSet.contains(namedExpression)

cloud-fan · 2025-09-26T12:13:31Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala

+
+  final override val nodePatterns: Seq[TreePattern] = Seq(UNRESOLVED_EVENT_TIME_WATERMARK)
+
+  private val delay = IntervalUtils.fromIntervalString(delayInterval.toString)


can we do it in the parser? This logical plan should take CalendarInterval directly.

cloud-fan · 2025-09-26T12:14:03Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/EventTimeWatermark.scala

  // This is not allowed by default - WatermarkPropagator will throw an exception. We keep the
  // logic here because we also maintain the compatibility flag. (See
  // SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE for details.)
+  // Should be defined as lazy so that attributes can be resolved before calling this.


do we still need this hack?

[SPARK-53687][SQL][SS] Introduce WATERMARK clause in SQL statement

e0b1f5d

github-actions bot added SQL STRUCTURED STREAMING labels Sep 24, 2025

HeartSaVioR changed the title ~~[SPARK-53687][SQL][SS] Introduce WATERMARK clause in SQL statement~~ [SPARK-53687][SQL][SS][SDP] Introduce WATERMARK clause in SQL statement Sep 24, 2025

HeartSaVioR commented Sep 24, 2025

View reviewed changes

HeartSaVioR added 2 commits September 24, 2025 11:20

addressed error class

65d9f21

fix CI failures

988cc60

github-actions bot added the DOCS label Sep 24, 2025

cloud-fan reviewed Sep 24, 2025

View reviewed changes

HeartSaVioR added 2 commits September 24, 2025 18:20

removed unused error class

b8602bb

another fix

20a6a32

viirya reviewed Sep 24, 2025

View reviewed changes

cloud-fan reviewed Sep 25, 2025

View reviewed changes

HeartSaVioR added 2 commits September 26, 2025 09:32

Move the logic of resolving EventTimeWatermark to separate analysis rule

a161007

maybe correct one

eaf357c

HeartSaVioR commented Sep 26, 2025

View reviewed changes

cloud-fan reviewed Sep 26, 2025

View reviewed changes


		final override val nodePatterns: Seq[TreePattern] = Seq(UNRESOLVED_EVENT_TIME_WATERMARK)

		private val delay = IntervalUtils.fromIntervalString(delayInterval.toString)

[SPARK-53687][SQL][SS][SDP] Introduce WATERMARK clause in SQL statement #52428

Are you sure you want to change the base?

[SPARK-53687][SQL][SS][SDP] Introduce WATERMARK clause in SQL statement #52428

Conversation

HeartSaVioR commented Sep 24, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

HeartSaVioR commented Sep 24, 2025

Uh oh!

HeartSaVioR commented Sep 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HeartSaVioR Sep 24, 2025 •

edited

Loading