[GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size by LuciferYang · Pull Request #12622 · apache/gluten

LuciferYang · 2026-07-24T08:57:37Z

What changes were proposed in this pull request?

GlutenDriverPlugin.setPredefinedConfigs reads the off-heap size in its non-dynamic-sizing branch with the single-arg conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY). checkOffHeapSettings returns early for untracked memory mode (spark.gluten.memory.untracked=true) and skips the requirement that the size be set, so in that mode the key can be absent. SparkConf.get(String) throws NoSuchElementException on a missing key, so driver plugin init crashes instead of proceeding under untracked semantics.

The same single-arg read exists on the Velox driver-start path: VeloxListenerApi.onDriverStart also calls conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY), and it runs after setPredefinedConfigs (via init calling Component.onDriverStart). The dynamic-sizing branch already shields that reader by doing conf.set(SPARK_OFFHEAP_SIZE_KEY, "0"); untracked mode did not, so it was the one unshielded path.

This change normalizes the key in the untracked branch: when spark.memory.offHeap.size is absent, set it to 0, the same way the dynamic-sizing branch does. One change covers both readers. Normal mode is unaffected because checkOffHeapSettings already guarantees the key is set there.

How was this patch tested?

Added GlutenDriverPluginSuite with two tests: untracked mode without an off-heap size no longer throws and yields a 0 off-heap budget, and the normal path reads a configured 512m. The untracked test also asserts spark.memory.offHeap.size is readable afterward, which is what the downstream VeloxListenerApi path needs. The first test fails on the current code (it throws NoSuchElementException) and passes after the fix.

Closes #12621

…ode without off-heap size setPredefinedConfigs read the off-heap size in its non-dynamic branch with the single-arg conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY). checkOffHeapSettings returns early for untracked memory mode and skips the requirement that the size be set, so the key can be absent there. SparkConf.get(String) throws NoSuchElementException on a missing key, so driver plugin init crashed instead of proceeding. VeloxListenerApi.onDriverStart has the same single-arg read and runs after setPredefinedConfigs, so the failure reaches end to end. The dynamic-sizing branch already shields it by setting SPARK_OFFHEAP_SIZE_KEY to "0"; untracked mode did not. Normalize the key to 0 in the untracked branch when it is absent, mirroring the dynamic branch, so one change covers both readers. Normal mode is unaffected since checkOffHeapSettings guarantees the key is set. Add GlutenDriverPluginSuite covering untracked mode without a size and the normal path; the untracked test also asserts the raw key is readable so the downstream VeloxListenerApi path is covered.

github-actions · 2026-07-24T09:01:34Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Fixes a driver initialization crash in gluten-core when running with untracked memory mode enabled and spark.memory.offHeap.size unset, by normalizing the missing Spark off-heap size key to 0 so downstream code paths (including Velox driver-start) can safely read it.

Changes:

Expose GlutenDriverPlugin.setPredefinedConfigs to package scope for direct unit testing.
In non-dynamic-sizing mode, set spark.memory.offHeap.size to "0" when absent (primarily affecting untracked memory mode).
Add GlutenDriverPluginSuite coverage for both the untracked/no-offheap-size case and the normal configured off-heap-size case.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala	Normalizes missing `spark.memory.offHeap.size` to `0` and makes `setPredefinedConfigs` accessible for tests.
gluten-core/src/test/scala/org/apache/gluten/GlutenDriverPluginSuite.scala	Adds unit tests validating untracked mode no longer throws when the off-heap size key is missing, and that normal mode reads configured values.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jackylee-ch · 2026-07-24T13:47:08Z

+        // VeloxListenerApi.onDriverStart, don't hit NoSuchElementException. Normal mode always has
+        // the key set because checkOffHeapSettings enforced it, so this only affects untracked
+        // mode without an explicit off-heap size.
+        if (!conf.contains(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY)) {


we have checked the offHeap size while initing GlutenPlugin, this seems un neccesary? or we have any cases that meet this problem?

It's still needed for untracked mode. You're right that we validate the off-heap size in checkOffHeapSettings, but that path only runs in normal mode: untracked mode returns early at L86-89 (the COLUMNAR_MEMORY_UNTRACKED branch), before the L92-102 size check, so spark.memory.offHeap.size can stay absent.

Repro: spark.gluten.memory.untracked=true with spark.memory.offHeap.size unset and dynamic sizing off → checkOffHeapSettings returns at L89 → the else-branch (main L136) calls the single-arg conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY), which throws NoSuchElementException because SparkConf.get(String) reads the raw settings map and doesn't fall back to the ConfigEntry default. The added GlutenDriverPluginSuite test "setPredefinedConfigs does not throw in untracked mode without an off-heap size" fails without this change.

If it reads clearer, I can add a short comment at the untracked early-return noting the size isn't validated there.

To confirm this reproduces on the real driver-init path and not only in the unit test:

With spark.plugins=org.apache.gluten.GlutenPlugin, spark.gluten.memory.untracked=true, and spark.memory.offHeap.size left unset, constructing a SparkContext runs GlutenDriverPlugin.init -> setPredefinedConfigs and fails with:

java.util.NoSuchElementException: spark.memory.offHeap.size

I checked this locally by wiring a test through new SparkContext(conf) with the real spark.plugins registration. It fails on the unfixed tree with that exception and passes with this change.

There are three places a boot-based test could live, each with a cost:

Direct call to setPredefinedConfigs (what this PR does). Same input state and same throwing line as the real path, runs in gluten-core with no extra setup. It just does not enter through init.

Full SparkContext boot in gluten-core. GlutenDriverPlugin.init calls Component.sorted(), which needs at least one META-INF/gluten-components file on the classpath, and gluten-core ships none (real backends live in the backend modules), so this needs a test-only dummy backend and a registration resource. Component also keeps a JVM-global graph with a one-shot load latch and no reset, and ComponentSuite registers an intentional dependency cycle without cleaning it up, so a boot suite sharing the test JVM hits UnsupportedOperationException: Cycle detected in the component graph. Making it stable needs a testing-only Component reset hook, with ShuffleManagerRegistry.clear() guarded by SparkTestUtil.isTesting as the precedent.

Integration test in a backend module like backends-velox. That module ships a real VeloxBackend component file and has no graph-polluting suite, so a real spark.plugins boot works there without any of the option 2 machinery. The cost is that it goes through VeloxListenerApi.onDriverStart, so it needs the native build and only runs in CI, and the test would sit in a different module than the one-line fix.

I lean toward option 1 for this fix, and option 3 if we want a production-level boot test. Fine either way, tell me which you prefer.

LuciferYang · 2026-07-29T15:03:57Z

Thank you @jackylee-ch

Copilot AI review requested due to automatic review settings July 24, 2026 08:57

github-actions Bot added the CORE works for Gluten Core label Jul 24, 2026

Copilot started reviewing on behalf of LuciferYang July 24, 2026 09:02 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

jackylee-ch reviewed Jul 24, 2026

View reviewed changes

jackylee-ch approved these changes Jul 28, 2026

View reviewed changes

jackylee-ch merged commit 7ae4ea3 into apache:main Jul 29, 2026
54 checks passed

Navigation Menu

Uh oh!

[GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size - #12622

[GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size#12622
jackylee-ch merged 1 commit into
apache:mainfrom
LuciferYang:fix/untracked-offheap-nosuchelement

LuciferYang commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jackylee-ch Jul 24, 2026

Uh oh!

LuciferYang Jul 24, 2026

Uh oh!

LuciferYang Jul 27, 2026

Uh oh!

Uh oh!

LuciferYang commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

LuciferYang commented Jul 24, 2026

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jackylee-ch Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

LuciferYang Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

LuciferYang Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LuciferYang commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants