[GLUTEN][VL] Move Delta deletion vector reads off the driver

[malinjawi]

What changes are proposed in this pull request?

Note

This draft is stacked on #12612. The first three commits are the authoritative TahoeFileIndex table-root work from that PR; the final two commits are this change. The stack should be rebased onto main after #12612 is merged.

Move on-disk Delta deletion-vector payload reads off the Spark driver and add an opt-in path that reads the stored DV range directly through Velox.

The current native scan eagerly opens every DV sidecar, reads and checksum-validates the bitmap, and embeds the payload bytes into the split before Spark submits tasks. On a 2,461-file S3 table this added approximately 100 seconds to each physical scan while the executors were idle.

This patch introduces two compatible execution paths:

• The default deferred path carries a serializable DV source in the Spark input partition and materializes it on an executor. Successful task-local materialization is memoized; failed reads remain retryable.
• The opt-in native path carries the JVM-resolved absolute DV URI, stored-entry offset, payload size, and cardinality. Velox reads the exact range through FileHandleFactory and BufferedInputBuilder, allowing existing split preloading, I/O concurrency, and AsyncDataCache to apply.

The native reader validates the four-byte stored length, payload bounds, CRC32, roaring bitmap encoding, and expected cardinality before applying row filtering. Inline DVs continue to use the existing JVM-materialized byte field. The original protobuf field is retained for compatibility and rollback.

The native path is gated by:

spark.gluten.sql.columnar.delta.deletionVector.nativeRangeRead.enabled=true

It is disabled by default and takes precedence over deferPayloadRead.enabled. Setting it back to false returns to executor-JVM materialization; setting spark.gluten.sql.columnar.filescan=false remains the broader rollback.

The patch also adds driver/task DV metrics and native runtime counters for descriptor preparation, read attempts, bytes, and elapsed read time.

How was this patch tested?

• ./dev/format-scala-code.sh
• ./dev/format-cpp-code.sh with clang-format 15.0.7
• ./dev/gen-all-config-docs.sh
• git diff --check
• Compiled the Delta modules and modified test sources for Spark 3.3, 3.4, 3.5, 4.0, and 4.1 profiles.
• Built and linked the Velox backend and velox_delta_read_test target.
• DeltaDeletionVectorScanInfoSuite: 7 tests passed, including concurrent deferred materialization, failed-read retry, authoritative native descriptor handoff, and zero JVM payload reads.
• GlutenRuntimeConfigSuite: 5 tests passed.
• Added native tests for stored-range length, CRC32, bitmap/cardinality validation, and descriptor propagation.
• Added Spark/Delta handoff tests for native filtering with zero JVM DV payload reads.

The native test executable cannot run on the development macOS host because an unrelated Folly F14 assertion aborts during static initialization before GoogleTest begins. The target compiles and links successfully; Linux CI and the S3 benchmark are required before enabling the opt-in native path by default.

Matched S3 deletion-vector benchmark

The fix was validated on Spark 3.5.4 using one unchanged SF2500 store_sales Delta snapshot with 2,461 data files and 2,461 active deletion vectors. The snapshot contains 7,199,920,789 original rows, 720,032,919 deleted rows, and 6,479,887,870 remaining rows.

All arms used the same 8 workers, 15 executor cores per worker, 240 default/shuffle parallelism, snapshot, and query order:

1. metadata count(*);
2. physical sum(ss_ext_sales_price);
3. physical grouped sum(ss_net_profit).

Arm	Native file scan	Deferred JVM read	Native range read	Total
Vanilla Spark	No	—	—	76.316s
Gluten file-scan fallback	No	true	false	128.310s
Same-image legacy driver control	Yes	false	false	257.976s
Fixed native R1	Yes	true	true	21.821s
Fixed native R2 + metrics	Yes	true	true	22.110s
Fixed native R3	Yes	true	true	21.366s
Fixed native R4	Yes	true	true	21.674s

The fixed median is 21.748s: 11.86x faster than the same-image legacy control, 3.51x faster than vanilla Spark, and 5.90x faster than the Gluten file-scan fallback end to end. Considering only the two physical scans, the fixed median is 14.264s versus 251.053s for the legacy control, a 17.60x speedup. The four fixed physical-scan sums have a 1.0% range.

Correctness and activation gates passed:

• Every arm returned identical canonical result digests.
• Both physical native queries contained DeltaScanTransformer.
• The metrics repetition recorded 4,922 native DV range-read attempts, exactly two physical scans multiplied by 2,461 DV descriptors, and approximately 623 MiB read.
• A full SF2500 TPC-DS integration run on the fixed image, with active DVs on the three sales fact tables, passed all 103 query variants with result row counts and SHA-256 digests recorded.

The fixed path has four repetitions. The vanilla, fallback, and same-image legacy controls currently have one repetition each; randomized control repetitions are still recommended before treating the ratios as final external performance claims.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86