[ChristopherWirt]

Summary

• Tracking Issue: [JNI] Scans queue a completed Arrow batch per CPU core ahead of Java #11197

A JNI scan hands batches to Java through block_on_stream_thread_safe. That bridge queues one completed batch per CPU core, but each JNI stream has only one consumer. On wide or nested Spark scans those queued Arrow batches add up, so peak memory grows with the host's core count.

This adds a single-consumer bridge that keeps one completed batch ahead of Java, and uses it for JNI scans.

Result: local Spark measurements, medians of macOS peak physical footprint:

Workload	Runs	Peak GiB, before → after	Change
Flat-layout read	3	1.46 → 1.05	−28%
List-layout read	2	3.25 → 2.22	−32%
Ordered Vortex rewrite	2	1.86 → 1.43	−23%

• Batch sizes, compression and decode concurrency don't change. The rewrite's output was identical in size and row count.
• Timing was too noisy on a shared host to claim a speedup or rule out a regression. Only one Spark task with four native workers was measured.

Changes

• vortex-io: CurrentThreadRuntime::block_on_stream_prefetch, returning a PrefetchIterator.
  • It drives the stream in a background task, so native work still overlaps Java consumption.
  • It keeps at most one completed item ahead of the consumer, and skips entering the executor when an item is already waiting.
  • Dropping it cancels the producer. A producer panic is re-raised at end of stream.
• JNI: NativePartition.scanArrow uses it instead of block_on_stream_thread_safe. Other callers, including DuckDB, are unchanged.

Notes for reviewers

• Once a batch is queued, the producer stops pulling the next one until Java takes it. Conversions already in flight keep running, but with a single native worker there is little overlap.
• A native panic during a scan is still re-raised inside the Arrow C stream callback, which aborts the JVM. This PR doesn't change that.

Tests

• vortex-io: the one-item-ahead bound, and propagation of a driver panic.

On the fork (quantcast#78, same change), the vortex-io change is identical. The fork's JNI side goes through a scan_iterator helper from #11127 and has a JNI test for it. develop doesn't have that helper yet, so this branch changes the block_on_stream_thread_safe call directly and leaves out the JNI test. Only git diff --check has been run on the port.

API Changes

• New public CurrentThreadRuntime::block_on_stream_prefetch and PrefetchIterator in vortex-io. Additive only.

AI assistance

Disclosed per the AI policy: AI tools helped write and review this change. Claude Code prepared this upstream port, issue and PR description at my request. I am the responsible contributor.

🤖 Generated with Claude Code

[codspeed]

Merging this PR will degrade performance by 11.06%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

❌ 1 regressed benchmark
✅ 1769 untouched benchmarks
⏩ 47 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	lt_i64_nullable_avx2	9.1 µs	10.3 µs	-11.06%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing quantcast:cwirt/upstream-jni-scan-prefetch (7116d77) with develop (9d2da79)}

Footnotes

1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩