Repository navigation
Fold aggregates with masked and lane-parallel kernels - #10947
Open
robert3005 wants to merge 7 commits into
Open
robert3005 wants to merge 7 commits into
robert3005 wants to merge 7 commits into
Conversation
Merging this PR will regress 8 benchmarks3 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
|
robert3005
changed the title
With nulls present, the primitive min/max and sum kernels walked the runs of valid values one at a time, so a nullable column with short runs, such as one with half its rows null, reduced thousands of one- or two-element slices. Add a masked reduction lane kernel, IndexedFoldExt::fold_masked, which folds every lane into an accumulator and reads null lanes as an identity of the fold. Lanes are selected against constant bits of each 32-lane half of the validity word, which needs no per-lane variable shift and so vectorizes on baseline x86-64, and halves with no valid lane are skipped. Integer min/max read null lanes as a valid value, and integer sums narrower than 64 bits read them as zero, so both cost the same at any validity density. 64-bit integer sums keep their per-element overflow check, and floats keep their in-order runs. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X1HnweRGxeDWgQwXXM6f5s
Apply fold_masked to the other aggregate kernels that walked runs of valid values or zipped values with validity bits: - Float min/max fold total-order keys, integers that order exactly as total_cmp does, so the reduction is plain integer min/max and its result is bit-for-bit that of the previous total_cmp reduction, all-valid or not. - Decimal min/max read null lanes as a valid value, like integers. - Float NaN counts read null lanes as zero, which is not NaN. - Float sums read null lanes as zero, adding the values in the same order, which leaves the result unchanged. Decimal sums and the grouped sums keep their paths: decimal sums are bound by their checked adds, and grouped sums over short groups pay more to set up the fold per group than the fold saves. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X1HnweRGxeDWgQwXXM6f5s
Float sums, float min/max and 64-bit integer sums did not vectorize: the compiler cannot reorder float adds, float compares or checked adds, so each ran one element at a time, and with nulls each run of valid values ran separately. Add IndexedFoldExt::fold_lanes and fold_lanes_masked. They fold lane i into accumulator i % LANES and merge the accumulators at the end, so one row of lanes is one vector step for any fold. The masked form tests each lane against a constant bit of the validity word, with no branch. An opaque barrier before each row stops the vectorizers from packing one lane of several rows with strided loads. Use the folds in the aggregates: - Float sums use lane folds. NaN skipping runs a second pass only when the first sum is NaN. - Float min/max use float compares, one fold per extremum. A cold pass settles a valid NaN and the sign of a zero extremum. - 64-bit sums add with wrapping and OR together a bound on each value's magnitude. Values past the bound get an exact 128-bit sum, so a sum overflows only if its exact total does not fit. - Narrower integer sums use 16 lanes, except signed values on x86-64 without SSE4.1, which has no vector sign extension. On Graviton4, Sapphire Rapids, Zen 4 and Apple M4 the float sums are 4.6x to 8.8x faster, and up to 30x with scattered nulls. Float max is 2.4x to 8.8x faster, and 64-bit sums are 1.8x to 3.4x faster. Signed-off-by: Robert Kruszewski <github@robertk.io> Claude-Session: https://claude.ai/code/session_018SWbEqgj32dnKrPu3naMzN
robert3005
force-pushed
the
rk/fold-masked
branch
from
October 28, 2026 22:06
a2047c6 to
a3f0e9d
Compare
Format the changed crates with the pinned nightly. Let clippy accept v - v in the float min/max kernel, because it finds NaN and infinite values. Run integer_stats_all_nullable on 16K values instead of 64K. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
changed the title
Decimal values stored in i8 to i64 use the i64 sum kernels, with and without nulls. Wider values, or an i64 sum that overflows, use the checked loop in the result type. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
added
the
action/bench-compress
github-actions
Bot
removed
the
action/bench-compress
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.998x ➖, 1↑ 1↓)
vortex / parquet / ns (0.997x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.997x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Validity::and of two non-nullable bool validities, such as a nullable array masked by another, built a lazy Binary(And) that every reader of the validity evaluated again. Both sides already hold their bits, so AND them word by word into a bool validity up front. Masking a nullable i32 column with scattered nulls and counting its nulls takes 1.36 µs instead of 1.62 µs. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X1HnweRGxeDWgQwXXM6f5s
Validity::and read both validities' bits on the host to AND them, which panics for bits held on a device. Take the word-wise AND only when both validities are host-resident, and keep a lazy conjunction otherwise. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X1HnweRGxeDWgQwXXM6f5s
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most primitive aggregates ran one element at a time: with nulls they walked runs of valid values, and float sums, float min/max and 64-bit sums didn't vectorize because the compiler can't reorder their folds. This PR adds
fold_masked,fold_lanesandfold_lanes_maskedtovortex-compute, which read null lanes as an identity and fold into per-lane accumulators, and uses them in min/max, sum and NaN count. Decimals stored in at most 64 bits are summed with the same vectorizedi64sums. Each reduction is now branch-free, vectorizes on every target and costs the same at any validity density: float sums are 4.6×–8.8× faster (up to 30× with scattered nulls), float max 2.4×–8.8×, 64-bit sums 1.8×–3.4×, andi32min/max with scattered nulls goes from ~45 µs to ~8–9 µs.Validity::andof two bool validities, such as a nullable array masked by another, now ANDs their bits word by word up front instead of building a lazy conjunction that every reader evaluated again; masking a nullable column and counting its nulls goes from 1.62 µs to 1.36 µs.