[GLUTEN-12597][CORE] Remodel Kafka read onto ReadRel.ExtensionTable (Substrait 0.98) by nielspardon · Pull Request #12841 · apache/gluten

nielspardon · 2026-08-20T15:55:12Z

What changes are proposed in this pull request?

Part of #12597.

This makes ReadRel.read_type verbatim upstream Substrait 0.98 by removing Gluten's last graft on it -- the bool stream_kafka = 1000 discriminator and the nested ReadRel.StreamKafka message -- and remodels Gluten's ClickHouse Kafka streaming read onto the official extension_table = 8 path, the same mechanism MergeTree and Range already use. read_type now matches upstream exactly, which never allocated field 1000, so the retired tag is not reserved.

The StreamKafka payload moves out of algebra.proto into a new Gluten-owned kafka.proto (package gluten, org.apache.gluten.proto), packed into a google.protobuf.Any and carried in ReadRel.ExtensionTable.detail. The native consumer discriminates a Kafka read by the detail's type_url via detail().Is<gluten::StreamKafka>() -- mirroring the already-merged Velox Iceberg idiom enhancement().Is<gluten::IcebergReadExtension>(). This Any-in-an-official-extension-field pattern is the preferred way to carry Gluten payloads going forward, rather than grafting new fields onto the vendored Substrait messages; WriteRel.bucket_spec (field 1000) remains the one legacy graft.

Compatibility: this is a JVM-producer + ClickHouse-consumer change with no wire-compatibility constraint -- Gluten plans are transient, and the JAR and native library are generated from one proto source and ship together. Two side effects are worth calling out. The Kafka split-info payload changes from a bare StreamKafka to a ReadRel.ExtensionTable wrapping it; and read_type, which the unconditional stream_kafka flag used to set on every ReadRel, is now left unset for non-Kafka scans (no native code reads read_type_case). A JAR and native library must therefore be rebuilt together. This path is ClickHouse-only -- Kafka has no Velox path.

Stacked on #12832 (the iceberg_table half of the ReadRel.read_type migration), whose commit appears in this diff until it merges; review this PR's second commit (Remodel Kafka read onto ReadRel.ExtensionTable) in isolation. It will be rebased onto main once #12832 lands.

How was this patch tested?

Locally: standalone protoc confirms the post-delete read_type oneof is well-formed and that the relocated gluten.StreamKafka generates cleanly under both native codegen styles (flat kafka.pb.h). The JVM producer builds green -- mvn -Pspark-3.5 -Pkafka -pl gluten-core,gluten-substrait,gluten-kafka -am -DskipTests clean install (-Pkafka is required; clean clears stale generated .java from the removed message) -- with no lingering references to ReadRel.StreamKafka / setStreamKafka and scalastyle/checkstyle passing in-phase.

The native ClickHouse local-engine build (ReadRelParser.cpp, StreamKafkaRelParser.cpp, generated kafka.pb.*) needs the Linux/Docker toolchain and is verified here by inspection only. The Kafka runtime suites (GlutenKafkaScanSuite / ClickhouseGlutenKafkaScanSuite) require -Pkafka plus an external localhost:9092 broker and are not run by any in-repo GitHub Actions job, so the end-to-end Kafka path is not exercised in CI. A maintainer running -Pkafka against a broker before merge would be the real end-to-end check; the JVM toProtobuf and the CH deserialize are kept as literal mirrors of the existing extension-table readers to minimize that risk.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

🤖 Generated with AI

github-actions · 2026-08-20T16:57:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:00:01Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:04:24Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:08:34Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:32:36Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:56:07Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

This PR continues the Substrait 0.98 ReadRel.read_type migration by removing Gluten’s legacy Kafka graft from the vendored Substrait proto and modeling ClickHouse Kafka reads via the official ReadRel.ExtensionTable path, carrying a Gluten-owned gluten.StreamKafka payload in google.protobuf.Any.

Changes:

Remove stream_kafka / ReadRel.StreamKafka from algebra.proto and rely on ReadRel.ExtensionTable + Any type discrimination for Kafka.
Introduce gluten-owned kafka.proto (gluten.StreamKafka) and update JVM producer + ClickHouse native parser accordingly.
Add a shared JVM helper to pack ReadRel.ExtensionTable and adjust build wiring (protobuf dependency + CMake glob).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
gluten-substrait/src/main/scala/org/apache/gluten/utils/SubstraitUtil.scala	Adds helper to pack Gluten payloads into `ReadRel.ExtensionTable.detail`.
gluten-substrait/src/main/scala/org/apache/gluten/utils/SubstraitPlanPrinterUtil.scala	Extends TypeRegistry so `Any`-carried Gluten messages can print/parse as JSON.
gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto	Makes `ReadRel.read_type` match upstream (removes Kafka graft; keeps official fields).
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/StreamKafkaSourceNode.java	Emits split-info payload as `ReadRel.ExtensionTable` containing `gluten.StreamKafka` in `Any`.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java	Replaces boolean Kafka discriminator with an `extension_table.detail` `Any`.
gluten-kafka/src/main/scala/org/apache/gluten/execution/MicroBatchScanExecTransformer.scala	Marks Kafka reads by stamping `gluten.StreamKafka` type into the in-plan `extension_table`.
gluten-kafka/pom.xml	Moves `protobuf-java` to provided scope for compilation/runtime needs.
gluten-core/src/main/resources/org/apache/gluten/proto/kafka.proto	Adds Gluten-owned `StreamKafka` message definition for `Any` payloads.
docs/developers/SubstraitModifications.md	Updates Substrait modification inventory (removes old `StreamKafka` entry).
cpp-ch/local-engine/proto/CMakeLists.txt	Makes proto glob reconfigure-aware (`CONFIGURE_DEPENDS`).
cpp-ch/local-engine/Parser/SerializedPlanParser.cpp	Ensures Kafka reads always consume a split-info entry (asserts extension_table present).
cpp-ch/local-engine/Parser/RelParsers/StreamKafkaRelParser.cpp	Parses split-info as `ReadRel::ExtensionTable` and unpacks `gluten.StreamKafka` from `Any`.
cpp-ch/local-engine/Parser/RelParsers/ReadRelParser.cpp	Detects Kafka reads by `extension_table.detail().Is<gluten::StreamKafka>()`.
backends-clickhouse/src/main/java/org/apache/spark/sql/execution/datasources/clickhouse/ExtensionTableNode.java	Reuses the new helper to build extension tables.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

nielspardon · 2026-08-21T06:02:31Z

  // Definition of which type of scan operation is to be performed
  oneof read_type {
    VirtualTable virtual_table = 5;
    LocalFiles local_files = 6;
    NamedTable named_table = 7;


Fixed the description — that sentence was stale. The code intentionally matches upstream 0.98 exactly, which never allocated field 1000, so nothing is reserved.

nielspardon · 2026-08-21T06:02:29Z

+      // Gluten's own payloads (e.g. StreamKafka) ride in Any fields of the Substrait plan, and
+      // nothing imports kafka.proto, so its messages are not reachable from the plan descriptor.
+      .add(Kafka.getDescriptor.getMessageTypes)


org.apache.gluten.proto.Kafka is the correct generated class. protobuf-java derives the file's outer class from the filename (kafka.proto → Kafka) and only appends OuterClass when a top-level message or enum shares that name — there is no message Kafka, so there is no suffix. With java_multiple_files = true the messages (StreamKafka) are emitted as their own files while Kafka remains the file-descriptor holder, which is exactly what Kafka.getDescriptor.getMessageTypes needs to register StreamKafka in the TypeRegistry. protoc --java_out on this file emits Kafka.java, StreamKafka.java, and StreamKafkaOrBuilder.java, and gluten-substrait compiles against this import. (KafkaProto is the Go / *_pb2 convention, not protobuf-java's.)

github-actions · 2026-08-21T05:50:48Z

Run Gluten Clickhouse CI on x86

… iceberg_table, relocate stream_kafka) Substrait 0.98 added `IcebergTable iceberg_table = 9` to the `ReadRel.read_type` oneof, exactly where Gluten's local `stream_kafka` graft sat. This vendors the 0.98 `iceberg_table` field and `IcebergTable` message verbatim and relocates the `stream_kafka` graft off the collision, as one step of the Substrait v0.23.0 -> 0.98.0 proto rebase (apache#12597). The graft moves to field 1000, following the "Gluten-local fields start at 1000" convention established for WriteRel's bucket_spec (apache#12746); it stays inside the read_type oneof. The enclosing Rel.read oneof tag is unchanged. All accessors are name-based (setStreamKafka/hasStreamKafka), so the field renumber needs no source change, and the new iceberg_table field is unreferenced by any producer or consumer, so no source is touched. A descriptor-level `ReadRelProtoSuite` pins the `read_type` field numbers (and the vendored `IcebergTable` layout), since a renumber round-trips cleanly through the shared schema and would otherwise be invisible to tests. This is the first of three ReadRel slices (read_type / text options / VirtualTable); the other two are separate follow-ups. Part of apache#12597 Generated-by: Claude Code (Claude Opus 4.8)

…Substrait 0.98) Part of apache#12597. Stacked on apache#12832 (the iceberg_table half of the ReadRel read_type migration). Makes ReadRel.read_type verbatim upstream Substrait 0.98 by removing Gluten's last graft on it -- the `bool stream_kafka = 1000` discriminator and the nested `ReadRel.StreamKafka` message. read_type now matches upstream exactly, which never allocated field 1000, so the retired tag is not reserved. Gluten's Kafka streaming read is remodeled onto the official `extension_table = 8` path, the mechanism MergeTree and Range already use. The StreamKafka payload moves to a new Gluten-owned `kafka.proto` (`package gluten`, `org.apache.gluten.proto`), packed into a `google.protobuf.Any` and carried in `ReadRel.ExtensionTable.detail`. The native consumer discriminates a Kafka read by the detail's type_url (`detail().Is<gluten::StreamKafka>()`) -- mirroring the already-merged Velox Iceberg idiom (`enhancement().Is<gluten::IcebergReadExtension>()`). This Any-in-an-official-extension-field pattern is preferred over grafting new fields; `WriteRel.bucket_spec` (field 1000) remains the one legacy graft. Gluten plans are transient and the JAR + native library are generated from one proto source and ship together, so there is no wire-compatibility constraint. Two side effects: the Kafka split-info payload changes from a bare StreamKafka to a ReadRel.ExtensionTable wrapping it, and read_type -- previously set on every ReadRel by the unconditional stream_kafka flag -- is now left unset for non-Kafka scans (no native code reads read_type_case). A JAR and native library must therefore be rebuilt together. ClickHouse-only; Kafka has no Velox path. Generated-by: Claude Code (Claude Opus 4.8)

github-actions · 2026-08-21T06:34:49Z

Run Gluten Clickhouse CI on x86

nielspardon · 2026-08-21T08:31:31Z

Run Gluten Clickhouse CI

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch 4 times, most recently from 550be87 to bedfb96 Compare August 20, 2026 16:55

github-actions Bot added CORE works for Gluten Core CLICKHOUSE DOCS labels Aug 20, 2026

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from bedfb96 to 0f4d98c Compare August 20, 2026 17:46

zhouyuan requested a lite review from Copilot August 20, 2026 19:17

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of zhouyuan August 20, 2026 22:18 View session

nielspardon marked this pull request as ready for review August 21, 2026 05:44

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 0f4d98c to 54cd984 Compare August 21, 2026 05:50

nielspardon added 2 commits August 21, 2026 07:55

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 54cd984 to 3264a7f Compare August 21, 2026 06:32

zhouyuan requested a review from exmy August 21, 2026 09:08

Navigation Menu

Uh oh!

[GLUTEN-12597][CORE] Remodel Kafka read onto ReadRel.ExtensionTable (Substrait 0.98) - #12841

[GLUTEN-12597][CORE] Remodel Kafka read onto ReadRel.ExtensionTable (Substrait 0.98)#12841
nielspardon wants to merge 2 commits into
apache:mainfrom
nielspardon:feat/substrait-0.98-kafka-extension-table

nielspardon commented Aug 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

nielspardon Aug 21, 2026

Uh oh!

nielspardon Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

nielspardon commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nielspardon commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

nielspardon Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

nielspardon Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

nielspardon commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nielspardon commented Aug 20, 2026 •

edited

Loading