Rebase vendored Substrait proto from v0.23.0 to 0.98.0

[nielspardon]

Background

Gluten vendors its Substrait .proto files (gluten-substrait/src/main/resources/substrait/proto/substrait/) pinned near Substrait v0.23.0 with local patches, while upstream Substrait is now at 0.98.0. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is no wire-compatibility constraint and the rebase is a pure code migration.

Goal

Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.

Approach

Land as small, independently-green PRs, one per message family (each keeps protoc + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.

Increments

One PR per message family; each keeps protoc + JVM + both native builds green. Progress: 3 merged · 6 in review · 1 draft · 2 to do · 1 resolved upstream.

• 0 — Remove unused vendored proto files + dead proto-based type derivation + Expression.Enum — merged ([GLUTEN-12597][CORE] Remove unused vendored Substrait proto files and dead proto-based type derivation #12598)
• 1a — plan.proto / extensions.proto → 0.98 (URI→URN) — merged ([GLUTEN-12597][CORE] Migrate Substrait extension referencing from URI to URN #12604)
• 1b — AdvancedExtension.optimization singular → repeated — merged ([GLUTEN-12597][CORE] Make AdvancedExtension.optimization repeated #12642)
• 2 — Type system: temporal → PrecisionTimestamp*; relocate Type.Nothing (kind 33→40, kept) — in review ([GLUTEN-12597][CORE] Migrate Substrait type system to 0.98 temporal types #12650)
• 3 — CrossRel → NestedLoopJoinRel (+ join-type enum remap) — in review ([GLUTEN-12597][CORE] Migrate nested loop joins from CrossRel to NestedLoopJoinRel (Substrait 0.98) #12700)
• 4 — Windowing: WindowRel → ConsistentPartitionWindowRel; WindowType → BoundsType — in review ([GLUTEN-12597][CORE] Migrate WindowRel to ConsistentPartitionWindowRel + BoundsType (Substrait 0.98) #12727)
• 5a — FetchRel offset/count → expressions — in review ([GLUTEN-12597][CORE] Migrate FetchRel offset/count to expressions (Substrait 0.98) #12704)
• 5b — AggregateRel.Grouping → expression references — in review ([GLUTEN-12597][CORE] Migrate AggregateRel.Grouping to expression references (Substrait 0.98) #12724)
• 5c — ExpandRel.advanced_extension — converged upstream via substrait-io/substrait#1167 (merged, ships 0.100.0); Gluten's graft already matches, no Gluten PR needed
• 6a — TopNRel → Expression count + offset + FetchMode — in review ([GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98) #12728)
• 6b — WriteRel: relocate bucket_spec (→ 1000); add create_mode / advanced_extension; OUTPUT_MODE_MODIFIED_TUPLES → _RECORDS — draft ([GLUTEN-12597][CORE] Migrate WriteRel to Substrait 0.98 (relocate bucket_spec; add create_mode/advanced_extension) #12746)
• 7 — ReadRel: relocate stream_kafka; text-options redesign; VirtualTable.values → expressions — to do
• 8 — Rel oneof reconciliation to exact 0.98 layout + drop RelRoot.output_schema — to do

Upstream Substrait convergence PRs

Some kept forks converge by landing the equivalent field/type in Substrait core (substrait-io/substrait) rather than by dropping the fork. These are tracked under the apache-gluten label:

• substrait-io/substrait#1167 — advanced_extension on ExpandRel — merged, ships 0.100.0 → drives increment 5c; Gluten's graft already matches verbatim
• substrait-io/substrait#1147 — core Type.Null for untyped nulls — open → increment 2 relocates Type.Nothing to kind 40 to match; converge by renaming Nothing→Null once released (see note below)
• substrait-io/substrait#1138 — JSON read options on ReadRel.LocalFiles — open → upstreams Gluten's json read-options fork; increment 7 keeps the vendored json = 15 until this merges
• substrait-io/substrait#1139 — partition fields on ReadRel.LocalFiles — open → upstreams Gluten's partition-columns fork; the field-16 collision with the iceberg fork is a post-merge concern
• substrait-io/substrait#1140 — docs on typing untyped nulls — closed, superseded by substrait-io/substrait#1147

Tracked for later extraction waves (not driving a rebase increment):

• substrait-io/substrait#917 — GenerateRel for lateral view / unnest (@EpsilonPrime) — upstreams the pattern of Gluten's GenerateRel fork
• substrait-io/substrait#1105 — expression-valued window-aggregate bounds (@yongchul) — future convergence for Gluten's window frame bounds, mirroring the FetchRel offset→expression change

Note on Type.Nothing

Following the Substrait community sync (2026-07-29), the earlier plan to drop the Type.Nothing fork is superseded: upstream is adding a dedicated core null type (substrait-io/substrait#1147 — Type.Null at kind field 40). Increment 2 relocates Nothing to field 40 to match, so once that PR releases the fork converges by renaming the message Nothing→Null — no field renumbering, no consumer-mapping changes. Gluten keeps Type.Nothing until then.

Related: upstream consolidation work is tracked under the apache-gluten label in substrait-io/substrait.

[nielspardon]

Update — Type.Nothing direction changed (Increment 2)

The Increment 2 checklist item previously read "drop Type.Nothing". That's been corrected to relocate (kind 33→40, kept).

Rationale: in the Substrait community sync on 2026-07-29 the PMCs decided upstream should add a dedicated core null type rather than have producers avoid a bottom type. That proposal is up as substrait-io/substrait#1147 (Type.Null at kind field 40). Increment 2 deliberately relocates Gluten's Nothing fork to field 40 to match, so once #1147 is merged and released upstream the fork converges by a pure message rename Nothing→Null — no field renumbering and no changes to the Velox/ClickHouse consumer mappings. Gluten keeps Type.Nothing as-is until then.

Net effect: Increment 2 is semantically neutral for Nothing (relocate only), and the former "drop" step becomes a future rename gated on the upstream Type.Null release.