[VL] Make ValueStreamDataSource::next() iterative instead of recursive - #12563
Draft
malinjawi wants to merge 1 commit into
Draft
[VL] Make ValueStreamDataSource::next() iterative instead of recursive#12563malinjawi wants to merge 1 commit into
malinjawi wants to merge 1 commit into
Conversation
malinjawi
force-pushed
the
fix/valuestream-empty-batch
branch
from
July 19, 2026 11:06
b1878ec to
30fda50
Compare
malinjawi
force-pushed
the
fix/valuestream-empty-batch
branch
from
August 10, 2026 11:00
43aebfe to
1799eda
Compare
malinjawi
marked this pull request as ready for review
August 10, 2026 11:01
malinjawi
marked this pull request as draft
August 11, 2026 08:18
ValueStreamDataSource::next() retried by calling itself. Each exhausted iterator, end-of-stream batch, or batch fully eliminated by a dynamic filter consumed a stack frame, so a long run of eliminated batches -- the common case once a selective dynamic filter is pushed into a value stream -- grew the native stack in proportion to the number of skipped batches. The retry logic is now a loop. Every branch keeps its previous semantics: an exhausted stream and a null batch both release the current iterator before retrying, a fully-filtered batch retries on the same iterator, and an empty pending-iterator queue still returns an engaged null RowVectorPtr so TableScan reads it as "current split finished" rather than connector-blocked. Adds a regression test that pushes a dynamic filter eliminating 4096 consecutive batches through a single split.
malinjawi
force-pushed
the
fix/valuestream-empty-batch
branch
from
August 11, 2026 08:25
1799eda to
9a474f1
Compare
malinjawi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
ValueStreamDataSource::next()retried by calling itself. Each exhausted iterator, end-of-stream batch, or batch fully eliminated by a dynamic filter consumed a stack frame, so a long run of eliminated batches — the common case once a selective dynamic filter is pushed into a value stream (#11657) — grew the native stack in proportion to the number of skipped batches.The retry logic is now a loop. This is a pure recursion-to-iteration change; every branch keeps its previous semantics:
RowVectorPtr, soTableScanreads it as "current split finished" rather than connector-blocked.How was this patch tested?
New
manyConsecutiveFullyFilteredBatchescase inValueStreamDynamicFilterTestpushes a dynamic filter that eliminates 4096 consecutive batches within a single split, then asserts the surviving batch after them still comes through. It rides the existingvelox_operators_testtarget.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (reviewed and tested before submission)