Skip to content

Commit 9375f7d

Browse files
[GLUTEN-12753][CORE] Mark natively-handled PushedFilters with * for all file-source scans
Gluten's file-source scan transformers push every filter conjunct into the native scan via PushDownFilterToScan and evaluate them as exact row-level filters, so the paired FilterExecTransformer becomes a no-op (FilterExecTransformerBase.isNoop) and no separate post-scan Filter is needed. This is exactly the state Spark marks with a leading `*` in a scan's `PushedFilters` metadata (RowDataSourceScanExec's convention for a filter the source fully handles itself). Gluten inherited FileSourceScanLike's rendering, which leaves `PushedFilters` unmarked, so its plan output under-reported what the native scan actually does. Mark each pushed-filter entry with `*` in the rendered plan string of FileSourceScanExecTransformerBase. `metadata` is a lazy val and cannot be super-overridden, so the marking is applied to the rendered node string in both paths that print it: - simpleString (executedPlan.toString), and - verboseStringWithOperatorId (FormattedMode / plan-stability golden files). The mark is gated on the backend actually accepting Gluten's full filter pushdown (SparkPlanExecApi.supportPushDownFilterToScan). ClickHouse deliberately declines it for Parquet to keep vanilla-Spark best-effort semantics, and there BasicScanExecTransformer.filterExprs() silently drops filters the backend cannot evaluate, leaving a real (non-no-op) FilterExecTransformer above the scan -- marking those would be a false claim. Velox does not override the default (true), so its rendering is unaffected. Only FileSourceScanExecTransformer nodes are marked. AQE FormattedMode also prints an Initial Plan whose scans are vanilla FileSourceScanExec (`Scan parquet`); those are left unmarked, matching runtime output and the semantics of `*` (the vanilla Initial-Plan scan still has a real Filter above it). The change lives in the common gluten-substrait module (hence [CORE]). Golden plans updated accordingly (TPC-H approved-plan, TPC-DS plan-stability, and gluten-tpch-plan-stability across the supported Spark versions): every offloaded FileSourceScanExecTransformer PushedFilters entry now carries `*`, while vanilla Initial-Plan scan entries stay unmarked. The goldens contain no DataSourceV2 BatchScan PushedFilters, so no V2 golden is affected. Add FileSourceScanExecTransformerPushedFiltersSuite covering the rendering helpers directly: entry marking, column names containing spaces, commas nested inside an entry (e.g. `In(id, [1,2,3])`), empty/non-list values, that only the PushedFilters list is rewritten, and that an unbalanced list degrades to unchanged text rather than being corrupted. It needs no native library, so it runs in the regular JVM test job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02286396-395f-41db-ad89-ad69a192cded
1 parent 413901f commit 9375f7d

910 files changed

Lines changed: 5017 additions & 4822 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/1.txt

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/10.txt

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/11.txt

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/12.txt

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/13.txt

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/14.txt

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/15.txt

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/16.txt

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/18.txt

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backends-velox/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/19.txt

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)