[VL][Iceberg] Add read evolved Iceberg schemas using recursive field identity support - #12900
Open
infvg wants to merge 1 commit into
Open
[VL][Iceberg] Add read evolved Iceberg schemas using recursive field identity support#12900infvg wants to merge 1 commit into
infvg wants to merge 1 commit into
Conversation
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Iceberg schema-evolution support for the Velox backend by propagating recursive Iceberg field-identity trees (struct/array/map) through the Substrait read extension and converting them to Velox ParquetFieldId mappings, enabling identity-based reads (instead of name-based) for renamed/reordered/reused nested fields.
Changes:
- Replace flat (root-only) Iceberg field-ID passing with a recursive
IcebergFieldIdtree and corresponding protobuf encoding (children). - Update Velox plan conversion to build recursive
ParquetFieldIdmappings from the extension and requested types. - Add Scala and C++ test coverage for evolved schemas (nested projections, collections, time travel, mixed files).
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/TransformerApi.scala | Updates Iceberg read-extension API to accept recursive field identity trees. |
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala | Adds a feature flag for Iceberg field-id based reads. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/IcebergFieldId.java | Introduces a recursive field identity container (name/id/children). |
| gluten-iceberg/src/test/java/org/apache/gluten/substrait/rel/IcebergLocalFilesNodeBoundsTest.java | Adjusts test construction for the new field-ids parameter type. |
| gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala | Builds recursive field-id trees from Iceberg’s expected schema. |
| gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala | Uses backend capability to decide whether to attach field-id trees and relaxes rename validation when supported. |
| gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesNode.java | Switches field IDs from map to recursive list and packs extension when needed. |
| gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesBuilder.java | Updates builder signature to accept recursive field-id lists. |
| cpp/velox/tests/iceberg/IcebergPlanConverterTest.cc | Adds unit tests for recursive field-id mapping behavior. |
| cpp/velox/tests/CMakeLists.txt | Registers the new Iceberg plan converter test. |
| cpp/velox/substrait/SubstraitToVeloxPlan.cc | Uses recursive ParquetFieldId mappings when building Iceberg column handles. |
| cpp/velox/compute/iceberg/IcebergPlanConverter.h | Adds recursive IcebergFieldIdInfo and the toParquetFieldId API. |
| cpp/velox/compute/iceberg/IcebergPlanConverter.cc | Parses recursive field-id trees and converts them into ParquetFieldId hierarchies. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxTransformerApi.scala | Serializes recursive field-id trees into the Iceberg read extension protobuf. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala | Enables supportIcebergFieldIdRead() for Velox. |
| backends-velox/src/main/resources/org/apache/gluten/proto/IcebergReadExtension.proto | Extends ColumnFieldId to include recursive children. |
| backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxIcebergFieldIdSuite.scala | Adds end-to-end Spark/Iceberg evolution verification for native reads. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
73
to
78
| private lazy val icebergFieldIds = | ||
| if (icebergInitialDefaults.isEmpty) { | ||
| new JHashMap[String, Integer]() | ||
| } else { | ||
| if (BackendsApiManager.getSettings.supportIcebergFieldIdRead()) { | ||
| GlutenIcebergSourceUtil.getFieldIds(scan) | ||
| } else { | ||
| new JArrayList[IcebergFieldId]() | ||
| } |
Comment on lines
+78
to
83
| /** Packs Iceberg field identities and initial defaults into a backend-specific read extension. */ | ||
| def packIcebergReadExtension( | ||
| fieldIds: util.Map[String, Integer], | ||
| fieldIds: util.List[IcebergFieldId], | ||
| initialDefaults: util.Map[String, String]): Any = { | ||
| throw new UnsupportedOperationException("Iceberg initial-default reads are not supported") | ||
| throw new UnsupportedOperationException("Iceberg field-ID reads are not supported") | ||
| } |
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gluten previously passed only top-level Iceberg field IDs to Velox. Consequently, native reads could not reliably resolve renamed, reordered, deleted/re-added, or projected nested fields and fell back for evolved schemas.
This change propagates recursive field-ID trees for structs, arrays, and maps. Velox now builds recursive ParquetFieldId mappings and reads evolved schemas by identity instead of name.