Skip to content

[WIP] add bolt backend in gluten#12454

Open
taiyang-li wants to merge 33 commits into
apache:mainfrom
taiyang-li:add_bolt_backend_new
Open

[WIP] add bolt backend in gluten#12454
taiyang-li wants to merge 33 commits into
apache:mainfrom
taiyang-li:add_bolt_backend_new

Conversation

@taiyang-li

@taiyang-li taiyang-li commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Add Bolt backend in Gluten.

This PR depends on the following common patches first:

For more details, please refer to #12456.

How was this patch tested?

WIP stage.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: TraeCli openrouter-3o

@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from 23d492b to 8d197c4 Compare July 6, 2026 09:47
@github-actions github-actions Bot added CORE works for Gluten Core BUILD VELOX DOCS labels Jul 6, 2026
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@taiyang-li taiyang-li changed the title [WIP] try to add bolt backend in gluten [WIP] add bolt backend in gluten Jul 6, 2026
@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from 8d197c4 to 2b6f49a Compare July 6, 2026 11:14
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from 2b6f49a to e04cc1c Compare July 6, 2026 11:21
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from e04cc1c to 9cb20f4 Compare July 6, 2026 11:26
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from f74ab91 to a148383 Compare July 8, 2026 09:19
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

4 similar comments
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@taiyang-li taiyang-li added BOLT and removed VELOX labels Jul 9, 2026
@github-actions github-actions Bot added the VELOX label Jul 10, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

zhangxffff and others added 19 commits July 21, 2026 11:58
* ignore failed uts

* wip

* commit again

* fix some conflicts
* close issue #31

* fix conflicts of MetricsUtil.scala in bolt
* [CORE][BOLT] Dispatch JNI input iterators through Runtime

* [CORE] Export shared JNI state accessors

* [BOLT] Promote JVM-loaded native libraries to global visibility

* [CORE][BOLT] Split Core and Bolt into independent JNI libraries

* [CORE] Remove unused memory manager name plumbing

* [BOLT] Populate operator metrics through the array protocol

* [CORE][BOLT] Reduce native build divergence from community

* [DOC] Record the Core diff-reduction design and snapshot

* Remove doc

* [BOLT] Restore original lazy vector metric semantics

* [CORE] Restore community CMake and JNI entry signatures

* [CORE] Remove JNI input iterator test
* fix one ut

* fix all failed uts

* fix tidy
* fix array.sql in GlutenSQLQueryTestSuite

* fix array.sql in GlutenSQLQueryTestSuite

* fix datetime-parsing-invalid.sql

* fix raise_eror,assert_true and misc-functions.sql

* fix named-function-arguments.sql
* enable shuffle offload in BoltConfig

* enable shuffle offload only when bolt memory manager is enabled

* get metrics from operators for ShuffleReaderNode

* reformat code

* feat(bolt): add gluten flag to control bolt shuffle RowFormat

* format code
* feat(bolt): add gluten flag to control bolt shuffle RowFormat

See merge request: !2012

* fix ut failed caused by shuffle offload
* ignore FullOuter in BroadcastNestLoopJoin

* ignore test explode/posexplode function

* ignore Optimize GetJsonObject(ToJson(NamedStruct))

* remove Check BoltResizeBatches is added on reused shuffle read

* ignore last two uts

* fix style

* fix style
* format cpp files

* fix py and bash

* fix license header

* format cmake
@taiyang-li
taiyang-li force-pushed the add_bolt_backend_new branch from d7a3709 to ef07b54 Compare July 21, 2026 03:59
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

* fix bolt test settings

* fix build failure after rebase upstream

* fix tidy

* fix jvm build error

* fix build issues

* revert makefile
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@taiyang-li

Copy link
Copy Markdown
Contributor Author

Run Gluten Clickhouse CI on x86

@taiyang-li

Copy link
Copy Markdown
Contributor Author

Run Gluten Clickhouse CI on x86

1 similar comment
@zzcclp

zzcclp commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Run Gluten Clickhouse CI on x86

* fix velox uts

* fix bolt ut
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants