fix : implement_try_eval_mode_arithmetic #2073

coderfender · 2025-08-06T00:38:20Z

Which issue does this PR close?

Closes #2021

Rationale for this change

PR to support Try eval mode in native . Unfortunately neither DataFusion not Arrow crates support null on overflow ops which is the desired outcome in Spark (when Eval mode is set to Try)

What changes are included in this PR?

New UDF called checked_arithmetic which would perform checked_add / checked_subtract/ checked_multiply over operands and wraps overflow in a NULL . (There aren't direct DataFusion options / Arrow Kernel APIs which would provide this functionality out of the box and hence the need for custom kernel + UDF based solution)
On the Spark side, the check to fallback to Spark when the EVAL Mode is set to Try is removed for above arithmetic ops.

How are these changes tested?

Implemented unit tests with various overflow edge cases (add , subtract , multiple , divide etc)

coderfender · 2025-08-07T15:50:05Z

Hello @andygrove , I Implemented custom arrow kernels to perform checked_add , checked_sub and checked_mul (registered as UDFS) supporting Integral types only (similar to spark's behavior) . My hope is to repeat / reuse this for other ops (in future), now that there is a framework established.

Please take a look whenever you get a chance and I can make changes (if any) to support try eval mode. Thank you very much

codecov-commenter · 2025-08-07T19:38:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.45%. Comparing base (f09f8af) to head (bd002fa).
⚠️ Report is 381 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2073      +/-   ##
============================================
+ Coverage     56.12%   58.45%   +2.33%     
- Complexity      976     1253     +277     
============================================
  Files           119      143      +24     
  Lines         11743    13192    +1449     
  Branches       2251     2370     +119     
============================================
+ Hits           6591     7712    +1121     
- Misses         4012     4256     +244     
- Partials       1140     1224      +84

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

coderfender · 2025-08-08T01:29:34Z

@andygrove ,
Here is the summary of changes :

Spark's Try eval mode returns NULL in case there is a computation failure. Note that this is only supported / useful when the operands are integer (Int / Long) type since overflow on Float , Double and Decimal are non deterministic and/or return Nan/ Inf.
Since neither data fusion nor Arrow kernels have native implementation of Spark's Try eval mode , I went ahead and implemented custom UDFs (with custom Arrow kernels) which perform checked_add, checked_sub, checked_mul , checked_div which return None when overflow occurs.
I also verified that div and integer_div do work with the Try mode and added tests to check all possible edge cases.
fail_on_overflow param is added to create_physical_expr function to fork code to call UDF based on the Eval Option selected.
Once this PR gets approved / merged, I will go ahead and continue to use this framework to implement the EVAL mode for other ops such as cast while also refactoring abs , modulo operations

andygrove · 2025-08-08T01:37:37Z

@coderfender looks like there are some clippy issues to be resolved

andygrove · 2025-08-08T01:38:04Z

native/core/src/execution/planner.rs

@@ -231,7 +231,7 @@ impl PhysicalPlanner {
    ) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
        match spark_expr.expr_struct.as_ref().unwrap() {
            ExprStruct::Add(expr) => {
-                // TODO respect eval mode
+                // TODO respect ANSI eval mode
                // https://github.com/apache/datafusion-comet/issues/2021
                // https://github.com/apache/datafusion-comet/issues/536
                let _eval_mode = from_protobuf_eval_mode(expr.eval_mode)?;


Please remove the leading _ from the variable name now that we are using the variable

andygrove · 2025-08-08T01:41:40Z

native/spark-expr/src/math_funcs/checked_arithmetic.rs

+            match op {
+                "checked_add" => builder.append_option(l.add_checked(r).ok()),


Performing this match operation on every row will be expensive. It would be better to invert this and to the match once and then have a different for loop for each operation, something like this:

match op { "checked_add" => for i in 0..len { ... } "checked_sub" => for i in 0..len { ... }

andygrove · 2025-08-08T01:43:20Z

@coderfender I took a first pass through this, and I think this is looking good 👍

andygrove · 2025-08-08T03:39:10Z

native/core/src/execution/planner.rs

@@ -878,6 +879,7 @@ impl PhysicalPlanner {
        return_type: Option<&spark_expression::DataType>,
        op: DataFusionOperator,
        input_schema: SchemaRef,
+        fail_on_overflow: bool,


Maybe consider passing in the eval mode here instead of a boolean? We'll eventually need to support all three modes.

Sure thats a great idea!

coderfender · 2025-08-08T19:35:49Z

One of the TPC-H check failed with a network exception. @andygrove could you please re trigger that workflow whenever you get a chance?
Thank you

coderfender · 2025-08-08T20:48:06Z

@andygrove , thank you for restarting the failed job and glad to see that the checks have all passed. Please review once you get a chance and let me know if you think we need further changes.

Thank you

native/spark-expr/src/math_funcs/checked_arithmetic.rs

andygrove · 2025-08-11T15:17:32Z

native/spark-expr/src/math_funcs/checked_arithmetic.rs

+            (l.to_array_of_size(r.len())?, Arc::clone(r))
+        }
+        (ColumnarValue::Array(l), ColumnarValue::Scalar(r)) => {
+            (Arc::clone(l), r.to_array_of_size(l.len())?)


We may eventually want to have a specialized version of the kernel for the scalar case to avoid the overhead of creating an array from the scalar. This does not need to happen as part of this PR, though.

Sure ! I will create a follow up enhancement to track changes for a scalar impl. Thank you for the feed back @andygrove .

andygrove

LGTM. Thanks @coderfender

andygrove · 2025-08-12T13:27:08Z

@coderfender could you fix the conflicts?

coderfender · 2025-08-12T13:30:00Z

Thank you very much for the approval @andygrove . I am going to push a fix shortly after fixing conflicts.

coderfender · 2025-08-12T19:29:29Z

@andygrove , There is a test failure with the below error after rebase with main branch. I am currently investigating the failure and patch a potential fix

coderfender · 2025-08-12T21:53:07Z

@andygrove the checks have all passed. Thank you for your approval .Please merge once you get a chance

coderfender · 2025-08-13T01:12:26Z

Thank you very much for merging feature branch Andy . I created a new issue to extend on these changes and support ANSI mode for above arithmetic operations : #2137 (and raised a WIP PR #2136 )

coderfender marked this pull request as draft August 6, 2025 00:38

coderfender marked this pull request as ready for review August 7, 2025 16:37

coderfender force-pushed the fix_eval_try_mode_spark branch from 2f735f2 to e539695 Compare August 7, 2025 21:35

andygrove reviewed Aug 8, 2025

View reviewed changes

andygrove reviewed Aug 11, 2025

View reviewed changes

native/spark-expr/src/math_funcs/checked_arithmetic.rs Outdated Show resolved Hide resolved

andygrove reviewed Aug 11, 2025

View reviewed changes

andygrove approved these changes Aug 12, 2025

View reviewed changes

coderfender added 12 commits August 12, 2025 12:13

init_commit

0e041ca

try_arithmetic_impl

3f7e0e8

try_arithmetic_impl

4c8f5d6

try_arithmetic_impl

b7b94f0

try_arithmetic_impl

ce72d79

try_arithmetic_impl

956d6e9

try_arithmetic_impl

cf3d30c

try_arithmetic_impl

f6908a1

try_arithmetic_impl

8d6501d

try_arithmetic_impl

d80a18a

try_arithmetic_impl

c8a212f

try_arithmetic_impl

e0373d4

coderfender added 2 commits August 12, 2025 12:16

try_arithmetic_impl

dc01c9f

try_arithmetic_impl

bd002fa

coderfender force-pushed the fix_eval_try_mode_spark branch from cfba67a to bd002fa Compare August 12, 2025 19:42

andygrove merged commit 7976b94 into apache:main Aug 12, 2025
91 checks passed

coderfender mentioned this pull request Aug 13, 2025

docs: Update to support try arithmetic functions #2143

Merged

davidlghellin mentioned this pull request Aug 18, 2025

More types in try_* #2179

Open

		match op {
		"checked_add" => builder.append_option(l.add_checked(r).ok()),

fix : implement_try_eval_mode_arithmetic #2073

fix : implement_try_eval_mode_arithmetic #2073

Uh oh!

Conversation

coderfender commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

coderfender commented Aug 7, 2025

Uh oh!

codecov-commenter commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coderfender commented Aug 8, 2025

Uh oh!

andygrove commented Aug 8, 2025

Uh oh!

andygrove Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

andygrove Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

andygrove commented Aug 8, 2025

Uh oh!

andygrove Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

coderfender Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderfender commented Aug 8, 2025

Uh oh!

coderfender commented Aug 8, 2025

Uh oh!

Uh oh!

andygrove Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

coderfender Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove commented Aug 12, 2025

Uh oh!

coderfender commented Aug 12, 2025

Uh oh!

coderfender commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderfender commented Aug 12, 2025

Uh oh!

Uh oh!

coderfender commented Aug 13, 2025

Uh oh!

Uh oh!

coderfender commented Aug 6, 2025 •

edited

Loading

codecov-commenter commented Aug 7, 2025 •

edited

Loading

coderfender Aug 8, 2025 •

edited

Loading

coderfender commented Aug 12, 2025 •

edited

Loading