[GLUTEN-12597][CORE] Migrate Substrait extension referencing from URI to URN - #12604
Merged
zhouyuan merged 1 commit intoJul 27, 2026
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
… to URN Adopt the 0.98 URN-based simple-extension referencing model (substrait-io/substrait#971): - extensions.proto: SimpleExtensionURI -> SimpleExtensionURN; extension_uri_anchor -> extension_urn_anchor; uri -> urn; and in ExtensionType/ExtensionTypeVariation/ExtensionFunction, extension_uri_reference (field 1) is reserved and replaced by extension_urn_reference (field 4). - plan.proto / extended_expression.proto: extension_uris (field 1) is reserved, replaced by extension_urns = 8 (repeated SimpleExtensionURN). Velox producer (SubstraitExtensionCollector, VeloxToSubstraitPlan) now emits extension_urns / extension_urn_reference. Gluten still maps every function to one catch-all anchor and resolves by name, so the emitted URN is a single placeholder in the required extension:<OWNER>:<ID> format (extension:org.apache.gluten:functions) rather than an empty string; consuming the upstream io.substrait function extensions is a follow-up. Gluten's JVM producer never emitted extension_uris, so it needs no changes beyond recompiling against the regenerated classes. Test fixtures updated to URN field names and proper spec URN values (extension:io.substrait:<file>). AdvancedExtension.optimization (also repeated in 0.98) and the additive Plan fields (parameter_bindings/type_aliases/execution_behavior) are handled in follow-up increments. Part of apache#12597. Validated locally: gluten-substrait JVM build + Velox native build (libgluten/libvelox).
nielspardon
force-pushed
the
feat/substrait-0.98-inc1-uri-to-urn
branch
from
July 23, 2026 14:25
4e261ff to
9f0fa46
Compare
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
approved these changes
Jul 24, 2026
zhouyuan
left a comment
zhouyuan
left a comment
Member
There was a problem hiding this comment.
LGTM, thanks for the fix @nielspardon !
nielspardon
added a commit
to nielspardon/gluten
that referenced
this pull request
Jul 28, 2026
Substrait 0.98 changed extensions.proto AdvancedExtension.optimization from a singular to a repeated google.protobuf.Any (substrait-io/substrait). Update all producers and consumers accordingly: - Producers: AdvancedExtensionNode (JVM) and CHFormatWriterInjects use addOptimization(...) instead of setOptimization(...); VeloxToSubstraitPlan uses add_optimization() instead of mutable_optimization(). - Consumers: has_optimization() -> optimization_size() > 0 and optimization() -> optimization(0) in the Velox SubstraitParser / SubstraitToVeloxPlan and in the ClickHouse RelParsers (Read/Cross/Aggregate/Join/GroupLimit/Write) and local_engine_jni. The old singular optimization() accessor returned a default-constructed Any when unset, so consumers could read optimization().value()/UnpackTo() unconditionally. optimization(0) on an empty repeated field is out-of-bounds and crashes. The Velox reads are all guarded by optimization_size() > 0; the ClickHouse reads relied on that null-safe behavior, so route them through a firstOptimizationOrDefault() helper (SubstraitParserUtils.h) that returns Any::default_instance() when no optimization is present, preserving the prior semantics. Validated locally: proto (protoc), gluten-substrait JVM build, and Velox native build (libgluten/libvelox). The ClickHouse native parsers are covered by CI (local libch build requires a Linux/Docker toolchain). Part of apache#12597. Depends on apache#12604 (URI->URN).
zhouyuan
pushed a commit
that referenced
this pull request
Jul 29, 2026
…2642) Substrait 0.98 changed extensions.proto AdvancedExtension.optimization from a singular to a repeated google.protobuf.Any (substrait-io/substrait). Update all producers and consumers accordingly: - Producers: AdvancedExtensionNode (JVM) and CHFormatWriterInjects use addOptimization(...) instead of setOptimization(...); VeloxToSubstraitPlan uses add_optimization() instead of mutable_optimization(). - Consumers: has_optimization() -> optimization_size() > 0 and optimization() -> optimization(0) in the Velox SubstraitParser / SubstraitToVeloxPlan and in the ClickHouse RelParsers (Read/Cross/Aggregate/Join/GroupLimit/Write) and local_engine_jni. The old singular optimization() accessor returned a default-constructed Any when unset, so consumers could read optimization().value()/UnpackTo() unconditionally. optimization(0) on an empty repeated field is out-of-bounds and crashes. The Velox reads are all guarded by optimization_size() > 0; the ClickHouse reads relied on that null-safe behavior, so route them through a firstOptimizationOrDefault() helper (SubstraitParserUtils.h) that returns Any::default_instance() when no optimization is present, preserving the prior semantics. Validated locally: proto (protoc), gluten-substrait JVM build, and Velox native build (libgluten/libvelox). The ClickHouse native parsers are covered by CI (local libch build requires a Linux/Docker toolchain). Part of #12597. Depends on #12604 (URI->URN).
25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Second increment of the Substrait 0.98 proto rebase (#12597). Stacked on #12598 (approved, not yet merged) — until #12598 lands, this PR's diff includes its commit; the change owned by this PR is the top commit (URI→URN). I'll rebase onto
mainonce #12598 merges, which reduces the diff to just this increment. (The two increments touch disjoint files.)Adopts the 0.98 URN-based simple-extension referencing model (substrait-io/substrait#971), replacing URI-based references with URNs of the form
extension:<OWNER>:<ID>:extensions.proto:SimpleExtensionURI→SimpleExtensionURN;extension_uri_anchor→extension_urn_anchor;uri→urn; and inExtensionType/ExtensionTypeVariation/ExtensionFunction,extension_uri_reference(field 1) is reserved and replaced byextension_urn_reference(field 4).plan.proto/extended_expression.proto:extension_uris(field 1) reserved, replaced byextension_urns = 8(repeated SimpleExtensionURN).SubstraitExtensionCollector,VeloxToSubstraitPlan): now emitextension_urns/extension_urn_reference. Gluten maps all functions to a single catch-all anchor and resolves them by name, so it emits one valid-format placeholder URNextension:org.apache.gluten:functionsinstead of an empty string; consuming the real upstreamio.substraitfunction-extension URNs is a follow-up. Gluten's JVM producer never emittedextension_uris, so it needs no changes beyond recompiling against the regenerated classes.extension:io.substrait:<file>).AdvancedExtension.optimization(alsorepeatedin 0.98) and the additivePlanfields (parameter_bindings/type_aliases/execution_behavior) are handled in follow-up increments.How was this patch tested?
No behavioral change — this is a referencing-mechanism migration. Verified locally: the vendored proto compiles (
protoc); thegluten-substraitJVM build succeeds (regenerating theSimpleExtensionURNclasses); and the Velox native library builds (libgluten/libvelox, recompilingSubstraitExtensionCollector/VeloxToSubstraitPlan/SubstraitParseragainst the regenerated proto). The ClickHouse backend has noextension_uri*references and is unaffected. Existing Velox conversion tests exercise the updated fixtures.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
🤖 Generated with AI