Repository navigation
Optimize arrays only with a session; keep constructors lazy - #10793
Open
robert3005 wants to merge 28 commits into
Open
robert3005 wants to merge 28 commits into
robert3005 wants to merge 28 commits into
Conversation
robert3005
added
changelog/break
github-actions
Bot
removed
the
action/bench-sql-extended
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (42)
Powered by Polar Signals Cloud |
Merging this PR will degrade performance by 16.53%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.007x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.999x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.020x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.009x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.041x ➖, 1↑ 2↓)
datafusion / vortex-compact / ns (0.914x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.973x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (1.083x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.964x ➖, 2↑ 2↓)
|
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.990x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.972x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.039x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.022x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 96↑ 105↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.052x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (1.110x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.062x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.025x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.930x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.982x ➖, 1↑ 1↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.009x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.007x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.151x ❌, 0↑ 2↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.987x ➖, 1↑ 1↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
robert3005
force-pushed
the
rk/optimize-with-session
branch
from
October 26, 2026 11:25
be3d4e4 to
0e66881
Compare
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.952x ➖, 2↑ 3↓)
datafusion / vortex-compact / ns (0.984x ➖, 1↑ 3↓)
datafusion / parquet / ns (0.966x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (0.918x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.935x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.979x ➖, 0↑ 2↓)
|
robert3005
force-pushed
the
rk/optimize-with-session
branch
2 times, most recently
from
October 28, 2026 17:44
45e33b5 to
bbfce4d
Compare
robert3005
added
the
action/bench-sql-extended
robert3005
commented
Oct 29, 2026
robert3005
commented
Oct 29, 2026
robert3005
commented
Oct 29, 2026
robert3005
commented
Oct 29, 2026
robert3005
force-pushed
the
rk/optimize-with-session
branch
2 times, most recently
from
October 29, 2026 13:57
530cc95 to
c6471a5
Compare
robert3005
marked this pull request as ready for review
October 29, 2026 15:23
Every encoding's PARENT_RULES set moves into the session's ArrayKernels, registered for each (parent, child) pair the way execute-parent kernels are. Each encoding registers its rules from its initialize function, in the order the set listed them, and the static VTable::reduce_parent hook, ParentRuleSet and ArrayRef::reduce_parent are removed. - ArrayKernels::register_reduce_parent_rule registers a typed ArrayParentReduceRule. Plain ReduceParentFns are stored the same way. - A scalar function array's encoding id is its function's id, so rules that apply to any scalar function, as a parent or as a child, register under AnyScalarFn::id. They are tried after the rules for the exact ids. - The registry records which child ids each parent has rules for. The optimizer probes a child only when a rule is registered for its encoding under that parent, so a chunked array of many primitive chunks no longer evaluates every rule of every chunk. - Optimizing a tree only copies a node's slots once one of its children changes. - ArrayKernels::reduce_parent tries the rules for one child once; the fastlanes slice tests and the CUDA plan builder use it in place of ArrayRef::reduce_parent. The plan builder holds a session with the encodings it walks, since plans are built without one. - vortex-pco gains an initialize function that registers the encoding and its rules. Encoding tests that used a bare session now initialize their crate, so their rules apply. - Optimizer traces label every parent-reduce rule as a session kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
On develop each constructor optimized the node it built, so executing an array never walked the parts of its tree that kernels consume directly. Executing now optimizes the same nodes, at the point execution reaches them, instead of walking the whole input first: - execute_until optimizes the root, and a child when execution focuses it, rather than optimizing the whole tree before it starts. Taking from a chunked array of 2048 chunks no longer visits every chunk. - The pass over an execution's result stops at children that are already optimized, such as arrays a builder optimized as they were appended. - ChildBuilder::append_array optimizes what it is given, so a lazy slice of the elements is kept as the encoding it reduces to. - Rewrites optimize the nodes a rule built by descending only into children that are not optimized, instead of first collecting every node of the rewritten tree to tell new nodes from old ones. - Kernels that optimize with an execution context use the parent-reduce kernels the context captured instead of taking a fresh snapshot of the session's registry for each array. - The chunked-dictionary values pull-up rule is evaluated once per chunked parent instead of once for each of its chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
Pulling a shared dictionary values array above chunked dictionary codes depends on every chunk at once, so it is now one of the chunked array's own reductions instead of a parent-reduce rule each chunk evaluates. Chunked arrays then have no parent-reduce rules registered as parents, so optimizing one no longer visits each of its chunks, which made building a fixed-size list from many tiled fill values slower. A fixed-size list builder appending one repeated list appends the array itself instead of a chunked array of one chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
…ent registration, drop optimize_in - Pull Dict values up through Chunked with a Chunked reduce rule instead of special-casing it in Chunked::reduce. - Register the struct cast parent rule through register_reduce_parent_rule and remove the fn-pointer reduce-parent registration API. - Replace OptimizeInCtx::optimize_in with ArrayOptimizer::optimize on the context's session; the executor keeps optimize_with_kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
Kernels that optimize what they build, such as list element slices and fixed-size-list take, run once per element or chunk. They now optimize against the parent-reduce snapshot the execution context already holds instead of looking up the session's kernels on every call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
Layout readers apply the projection lazily, and apply no longer optimizes, so a scan of a struct returned a pack expression rather than a struct. vx_array_get_field, and the C++ Array::field built on it, require a struct and rejected it. The scan now optimizes each split's array with its session, as does the physical plan's eval stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
ArrayOptimizer::optimize and optimize_recursive take any AsOptimizerKernels source: a VortexSession snapshots its kernels for the call, while an ExecutionCtx reuses the snapshot it took when it was created. Kernels that optimize what they build now call array.optimize(ctx) instead of passing the context's kernel snapshot to optimize_with_kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Claude <noreply@anthropic.com>
Develop now pushes a slice through a filter, and that rule is registered in the session like the others, so optimizing the nested filter reaches the primitive without executing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
Slicing builds a lazy slice, so the test optimizes with a session before it checks how much of the child the pushed-down slice kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
Constructors build lazy arrays, so a parent could reach the optimizer and executor with a child that is still a lazy wrapper, such as a slice of an FSST array under a LIKE. Parent rules and execute-parent kernels keyed on the child's encoding then missed it: the FSST LIKE kernel never fired and the strings were decompressed instead, slowing ClickBench q23 on DuckDB. Optimizing a node now first optimizes the descendants below it that were never optimized, bottom up, as eager constructors used to. Children already optimized are left without visiting their subtrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
The writer stores arrays after optimizing them, so a decoded array and the tree below it need no further rewrites. Marking them lets the optimizer and executor skip them without trying any rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
Slices are now lazy, so written chunks can be a slice of an array rather than the array's own encoding. The dict encoder picks its builder from the dtype, since both builders already execute non-canonical input, and the flat writer optimizes a chunk before serializing it, since lazy nodes have no serialized form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
The write path optimizes chunks before they reach a layout writer, so the flat writer does not need to optimize them again. The dict writer test that feeds raw slices to the strategy now optimizes them as that path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErbGQPZvkCK9ZXH87w Signed-off-by: Robert Kruszewski <github@robertk.io>
- Use one optimized bit. The optimizer marks a node only after its children, so the mark covers the whole tree. Remove the recursive pass and its trace events. - Copy the stats of the input array to the rewritten array. - Derive the child id in register_reduce_parent_rule from the vtable, and add register_scalar_fn_reduce_parent_rule. - Make is_optimized, is_recursively_optimized, has_reduce_parent and DynReduceParentKernel private to the crate. - Fix the mark_decoded_optimized and zstd docs. Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Constructors are lazy, so a slice or cast reaches the CUDA kernels as a Slice or ScalarFn array instead of the encoding they match. - Optimize on entry to execute_cuda and before the ListView rebuild exports its child. - Register the default file encodings in cuda_session so the optimizer has their parent-reduce rules. - Optimize the lazy fixtures in tests that call one executor directly. Signed-off-by: Robert Kruszewski <github@robertk.io>
- Register the default file encodings in the varbin export test session, so a slice of an FSST array reduces to FSST. - Test the device dictionary guard with fixed-size list values. The optimizer now pushes a dictionary of structs into the struct fields, which CUDA decodes, so that fixture no longer reached the guard. Signed-off-by: Robert Kruszewski <github@robertk.io>
Only the optimizer reads `OptimizedState::is_optimized`, so make it private. The doc of `mark_decoded_optimized` said that decoded children stay unmarked, but decoding marks every node it decodes. Only the nodes that a deserializer builds itself stay unmarked. Signed-off-by: Robert Kruszewski <github@robertk.io>
Constructors are lazy now, so arrays that the scan pipeline keeps and uses again were optimized again for each use: - `take_rows` put the rest of a batch back into its inlet as one more lazy slice. Each pack step made the chain deeper, and each later step optimized the full chain again. This made ClickBench q23 on DataFusion about 300x slower. - `TakeSource` and the share plan keep their values in a `Shared` and in the value cache, and each batch optimized the source again. - `ScanSource` sent lazy slices and filters of decoded segments to every later stage. Optimize these arrays once, before they are kept. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/optimize-with-session
branch
from
October 30, 2026 16:30
221a559 to
e9809aa
Compare
robert3005
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
The optimizer now remembers, for one optimize call, the array it rewrote each shared node into. Many parents can hold the same unoptimized node, such as the struct below every field of a projection. Before, each parent rewrote the node and its whole subtree again. Plan v2 now optimizes the rows that take_rows returns and the struct that PackSource builds, so the stages that read them get optimized arrays. The execution context keeps both kernel snapshots in one shared field, and the session publishes a parent-reduce snapshot that the optimizer reads without a reference-count write. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
robert3005
enabled auto-merge (squash)
October 31, 2026 01:22
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The session-less
optimizeis removed andoptimize(session)is the only entry point, so every rewrite sees the session's registered kernels.Builtins such as
castandfill_null, andslice,filter,takeandapply, no longer optimize the node they build. Instead:execute_untilrunsoptimize_recursivewith the context's session before it starts, and again over the array execution produced, so canonical results do not keep lazy children such as a filtered validity;