Commit 9375f7d
[GLUTEN-12753][CORE] Mark natively-handled PushedFilters with
Gluten's file-source scan transformers push every filter conjunct into the native
scan via PushDownFilterToScan and evaluate them as exact row-level filters, so the
paired FilterExecTransformer becomes a no-op (FilterExecTransformerBase.isNoop) and
no separate post-scan Filter is needed. This is exactly the state Spark marks with
a leading `*` in a scan's `PushedFilters` metadata (RowDataSourceScanExec's
convention for a filter the source fully handles itself).
Gluten inherited FileSourceScanLike's rendering, which leaves `PushedFilters`
unmarked, so its plan output under-reported what the native scan actually does.
Mark each pushed-filter entry with `*` in the rendered plan string of
FileSourceScanExecTransformerBase. `metadata` is a lazy val and cannot be
super-overridden, so the marking is applied to the rendered node string in both
paths that print it:
- simpleString (executedPlan.toString), and
- verboseStringWithOperatorId (FormattedMode / plan-stability golden files).
The mark is gated on the backend actually accepting Gluten's full filter pushdown
(SparkPlanExecApi.supportPushDownFilterToScan). ClickHouse deliberately declines it
for Parquet to keep vanilla-Spark best-effort semantics, and there
BasicScanExecTransformer.filterExprs() silently drops filters the backend cannot
evaluate, leaving a real (non-no-op) FilterExecTransformer above the scan -- marking
those would be a false claim. Velox does not override the default (true), so its
rendering is unaffected.
Only FileSourceScanExecTransformer nodes are marked. AQE FormattedMode also prints
an Initial Plan whose scans are vanilla FileSourceScanExec (`Scan parquet`); those
are left unmarked, matching runtime output and the semantics of `*` (the vanilla
Initial-Plan scan still has a real Filter above it).
The change lives in the common gluten-substrait module (hence [CORE]). Golden plans
updated accordingly (TPC-H approved-plan, TPC-DS plan-stability, and
gluten-tpch-plan-stability across the supported Spark versions): every offloaded
FileSourceScanExecTransformer PushedFilters entry now carries `*`, while vanilla
Initial-Plan scan entries stay unmarked. The goldens contain no DataSourceV2
BatchScan PushedFilters, so no V2 golden is affected.
Add FileSourceScanExecTransformerPushedFiltersSuite covering the rendering helpers
directly: entry marking, column names containing spaces, commas nested inside an
entry (e.g. `In(id, [1,2,3])`), empty/non-list values, that only the PushedFilters
list is rewritten, and that an unbalanced list degrades to unchanged text rather
than being corrupted. It needs no native library, so it runs in the regular JVM
test job.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02286396-395f-41db-ad89-ad69a192cded* for all file-source scans1 parent 413901f commit 9375f7d
910 files changed
Lines changed: 5017 additions & 4822 deletions
File tree
- backends-velox/src/test/resources/tpch-approved-plan
- v1-bhj-ras
- spark33
- spark34
- spark35
- v1-bhj
- spark33
- spark34
- spark35
- v1-ras
- spark33
- spark34
- spark35
- v1
- spark33
- spark34
- spark35
- gluten-substrait/src
- main/scala/org/apache/gluten/execution
- test/scala/org/apache/gluten/execution
- gluten-ut
- spark40/src/test/resources/backends-velox
- gluten-tpch-plan-stability
- q10
- q11
- q12
- q13
- q14
- q15
- q16
- q17
- q18
- q19
- q1
- q20
- q21
- q22
- q2
- q3
- q4
- q5
- q7
- q8
- q9
- tpcds-plan-stability
- gluten-approved-plans-modified
- q10.sf100
- q10
- q19.sf100
- q19
- q27.sf100
- q27
- q3.sf100
- q34.sf100
- q34
- q3
- q42.sf100
- q42
- q43.sf100
- q43
- q46.sf100
- q46
- q52.sf100
- q52
- q53.sf100
- q53
- q55.sf100
- q55
- q59.sf100
- q59
- q63.sf100
- q63
- q65.sf100
- q65
- q68.sf100
- q68
- q7.sf100
- q73.sf100
- q73
- q79.sf100
- q79
- q7
- q89.sf100
- q89
- q98.sf100
- q98
- gluten-approved-plans-v1_4
- q1.sf100
- q10.sf100
- q10
- q11.sf100
- q11
- q12.sf100
- q12
- q13.sf100
- q13
- q14a.sf100
- q14a
- q14b.sf100
- q14b
- q15.sf100
- q15
- q16.sf100
- q16
- q17.sf100
- q17
- q18.sf100
- q18
- q19.sf100
- q19
- q1
- q2.sf100
- q20.sf100
- q20
- q21.sf100
- q21
- q22.sf100
- q22
- q23a.sf100
- q23a
- q23b.sf100
- q23b
- q24a.sf100
- q24a
- q24b.sf100
- q24b
- q25.sf100
- q25
- q26.sf100
- q26
- q27.sf100
- q27
- q28.sf100
- q28
- q29.sf100
- q29
- q2
- q3.sf100
- q30.sf100
- q30
- q31.sf100
- q31
- q32.sf100
- q32
- q33.sf100
- q33
- q34.sf100
- q34
- q35.sf100
- q35
- q36.sf100
- q36
- q37.sf100
- q37
- q38.sf100
- q38
- q39a.sf100
- q39a
- q39b.sf100
- q39b
- q3
- q4.sf100
- q40.sf100
- q40
- q41.sf100
- q41
- q42.sf100
- q42
- q43.sf100
- q43
- q44.sf100
- q44
- q45.sf100
- q45
- q46.sf100
- q46
- q47.sf100
- q47
- q48.sf100
- q48
- q49.sf100
- q49
- q4
- q5.sf100
- q50.sf100
- q50
- q51.sf100
- q51
- q52.sf100
- q52
- q53.sf100
- q53
- q54.sf100
- q54
- q55.sf100
- q55
- q56.sf100
- q56
- q57.sf100
- q57
- q58.sf100
- q58
- q59.sf100
- q59
- q5
- q6.sf100
- q60.sf100
- q60
- q61.sf100
- q61
- q62.sf100
- q62
- q63.sf100
- q63
- q64.sf100
- q64
- q65.sf100
- q65
- q66.sf100
- q66
- q67.sf100
- q67
- q68.sf100
- q68
- q69.sf100
- q69
- q6
- q7.sf100
- q70.sf100
- q70
- q71.sf100
- q71
- q72.sf100
- q72
- q73.sf100
- q73
- q74.sf100
- q74
- q75.sf100
- q75
- q76.sf100
- q76
- q77.sf100
- q77
- q78.sf100
- q78
- q79.sf100
- q79
- q7
- q8.sf100
- q80.sf100
- q80
- q81.sf100
- q81
- q82.sf100
- q82
- q83.sf100
- q83
- q84.sf100
- q84
- q85.sf100
- q85
- q86.sf100
- q86
- q87.sf100
- q87
- q88.sf100
- q88
- q89.sf100
- q89
- q8
- q9.sf100
- q90.sf100
- q90
- q91.sf100
- q91
- q92.sf100
- q92
- q93.sf100
- q93
- q94.sf100
- q94
- q95.sf100
- q95
- q96.sf100
- q96
- q97.sf100
- q97
- q98.sf100
- q98
- q99.sf100
- q99
- q9
- gluten-approved-plans-v2_7
- q10a.sf100
- q10a
- q11.sf100
- q11
- q12.sf100
- q12
- q14.sf100
- q14a.sf100
- q14a
- q14
- q18a.sf100
- q18a
- q20.sf100
- q20
- q22.sf100
- q22a.sf100
- q22a
- q22
- q24.sf100
- q24
- q27a.sf100
- q27a
- q34.sf100
- q34
- q35.sf100
- q35a.sf100
- q35a
- q35
- q36a.sf100
- q36a
- q47.sf100
- q47
- q49.sf100
- q49
- q51a.sf100
- q51a
- q57.sf100
- q57
- q5a.sf100
- q5a
- q6.sf100
- q64.sf100
- q64
- q67a.sf100
- q67a
- q6
- q70a.sf100
- q70a
- q72.sf100
- q72
- q74.sf100
- q74
- q75.sf100
- q75
- q77a.sf100
- q77a
- q78.sf100
- q78
- q80a.sf100
- q80a
- q86a.sf100
- q86a
- q98.sf100
- q98
- spark41/src/test/resources/backends-velox
- gluten-tpch-plan-stability
- q10
- q11
- q12
- q13
- q14
- q15
- q16
- q17
- q18
- q19
- q1
- q20
- q21
- q22
- q2
- q3
- q4
- q5
- q7
- q8
- q9
- tpcds-plan-stability
- gluten-approved-plans-modified
- q10.sf100
- q10
- q19.sf100
- q19
- q27.sf100
- q27
- q3.sf100
- q34.sf100
- q34
- q3
- q42.sf100
- q42
- q43.sf100
- q43
- q46.sf100
- q46
- q52.sf100
- q52
- q53.sf100
- q53
- q55.sf100
- q55
- q59.sf100
- q59
- q63.sf100
- q63
- q65.sf100
- q65
- q68.sf100
- q68
- q7.sf100
- q73.sf100
- q73
- q79.sf100
- q79
- q7
- q89.sf100
- q89
- q98.sf100
- q98
- gluten-approved-plans-v1_4
- q1.sf100
- q10.sf100
- q10
- q11.sf100
- q11
- q12.sf100
- q12
- q13.sf100
- q13
- q14a.sf100
- q14a
- q14b.sf100
- q14b
- q15.sf100
- q15
- q16.sf100
- q16
- q17.sf100
- q17
- q18.sf100
- q18
- q19.sf100
- q19
- q1
- q2.sf100
- q20.sf100
- q20
- q21.sf100
- q21
- q22.sf100
- q22
- q23a.sf100
- q23a
- q23b.sf100
- q23b
- q24a.sf100
- q24a
- q24b.sf100
- q24b
- q25.sf100
- q25
- q26.sf100
- q26
- q27.sf100
- q27
- q28.sf100
- q28
- q29.sf100
- q29
- q2
- q3.sf100
- q30.sf100
- q30
- q31.sf100
- q31
- q32.sf100
- q32
- q33.sf100
- q33
- q34.sf100
- q34
- q35.sf100
- q35
- q36.sf100
- q36
- q37.sf100
- q37
- q38.sf100
- q38
- q39a.sf100
- q39a
- q39b.sf100
- q39b
- q3
- q4.sf100
- q40.sf100
- q40
- q41.sf100
- q41
- q42.sf100
- q42
- q43.sf100
- q43
- q44.sf100
- q44
- q45.sf100
- q45
- q46.sf100
- q46
- q47.sf100
- q47
- q48.sf100
- q48
- q49.sf100
- q49
- q4
- q5.sf100
- q50.sf100
- q50
- q51.sf100
- q51
- q52.sf100
- q52
- q53.sf100
- q53
- q54.sf100
- q54
- q55.sf100
- q55
- q56.sf100
- q56
- q57.sf100
- q57
- q58.sf100
- q58
- q59.sf100
- q59
- q5
- q6.sf100
- q60.sf100
- q60
- q61.sf100
- q61
- q62.sf100
- q62
- q63.sf100
- q63
- q64.sf100
- q64
- q65.sf100
- q65
- q66.sf100
- q66
- q67.sf100
- q67
- q68.sf100
- q68
- q69.sf100
- q69
- q6
- q7.sf100
- q70.sf100
- q70
- q71.sf100
- q71
- q72.sf100
- q72
- q73.sf100
- q73
- q74.sf100
- q74
- q75.sf100
- q75
- q76.sf100
- q76
- q77.sf100
- q77
- q78.sf100
- q78
- q79.sf100
- q79
- q7
- q8.sf100
- q80.sf100
- q80
- q81.sf100
- q81
- q82.sf100
- q82
- q83.sf100
- q83
- q84.sf100
- q84
- q85.sf100
- q85
- q86.sf100
- q86
- q87.sf100
- q87
- q88.sf100
- q88
- q89.sf100
- q89
- q8
- q9.sf100
- q90.sf100
- q90
- q91.sf100
- q91
- q92.sf100
- q92
- q93.sf100
- q93
- q94.sf100
- q94
- q95.sf100
- q95
- q96.sf100
- q96
- q97.sf100
- q97
- q98.sf100
- q98
- q99.sf100
- q99
- q9
- gluten-approved-plans-v2_7
- q10a.sf100
- q10a
- q11.sf100
- q11
- q12.sf100
- q12
- q14.sf100
- q14a.sf100
- q14a
- q14
- q18a.sf100
- q18a
- q20.sf100
- q20
- q22.sf100
- q22a.sf100
- q22a
- q22
- q24.sf100
- q24
- q27a.sf100
- q27a
- q34.sf100
- q34
- q35.sf100
- q35a.sf100
- q35a
- q35
- q36a.sf100
- q36a
- q47.sf100
- q47
- q49.sf100
- q49
- q51a.sf100
- q51a
- q57.sf100
- q57
- q5a.sf100
- q5a
- q6.sf100
- q64.sf100
- q64
- q67a.sf100
- q67a
- q6
- q70a.sf100
- q70a
- q72.sf100
- q72
- q74.sf100
- q74
- q75.sf100
- q75
- q77a.sf100
- q77a
- q78.sf100
- q78
- q80a.sf100
- q80a
- q86a.sf100
- q86a
- q98.sf100
- q98
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments