Skip to content

[GLUTEN-12628][VL] Support Iceberg V3 deletion vectors in Velox - #12629

Draft
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:xuanyili/dv3read
Draft

[GLUTEN-12628][VL] Support Iceberg V3 deletion vectors in Velox#12629
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:xuanyili/dv3read

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Jul 26, 2026

Copy link
Copy Markdown

What changes are proposed in this pull request?

Closes #12628.

Add native Iceberg format-version 3 deletion-vector support to the Velox backend:

  • Serialize Puffin DV metadata through Substrait and convert it to Velox FileContent::kDeletionVector.
  • Replace blanket V3 scan fallback with per-delete validation while retaining fallback for equality deletes and incomplete metadata.
  • Offload Iceberg WriteDeltaExec for DELETE, UPDATE, and MERGE behind enhanced native writes.
  • Preserve Iceberg's existing DeltaBatchWrite coordinator for atomic commit, validation, conflict handling, and abort cleanup.
  • Seed native writers with existing DVs so repeated mutations union positions and replace the prior DV.
  • Preserve Spark 3.3 behavior and add Spark 3.4–4.1 compatibility shims, including Spark 4.1 MERGE summaries.
  • Add native, Scala, and enhanced end-to-end coverage, and run targeted native DV tests in enhanced CI.

How was this patch tested?

Passed:

  • ./dev/format-scala-code.sh
  • git diff --check
  • Focused Spark 3.5 Iceberg tests: 2 JUnit and 8 Scala tests
  • Focused Spark 3.5 Velox backend tests: 11 tests
  • Spark 3.4 clean test-compile
  • Spark 4.0 / Scala 2.13 / Java 17 clean test-compile
  • Direct compatibility compilation of the V2 write shims against Spark 3.3, 3.4, 3.5, 4.0, and 4.1
  • Direct compilation of the Spark 4.1 columnar V2 commit path and Spark 4.0 enhanced DV suite
  • Enhanced workflow YAML parsing

Not completed locally:

  • Native C++ and enhanced end-to-end execution require a rebuilt enhanced Velox backend.
  • Spark 4.1 Maven tests are blocked by HTTP 403 responses for the Iceberg 1.11.0 test artifacts.
  • The license-header command is blocked because dev/license-header.py is absent from this checkout.
  • C++ formatting passed earlier with clang-format 15; it could not be rerun after the final JNI cleanup because clang-format is no longer available in the environment.

Add native Puffin deletion-vector reads and enhanced DELETE, UPDATE, and MERGE writes while preserving Iceberg commit coordination and Spark 3.3 fallback behavior.\n\nAdd compatibility shims, native and Scala unit coverage, enhanced end-to-end tests, and targeted native CI execution.
@github-actions github-actions Bot added CORE works for Gluten Core VELOX INFRA DATA_LAKE labels Jul 26, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] Support Iceberg V3 deletion vectors in Velox

1 participant