Repository navigation
perf(arrow): fuse Boolean filter copying - #11918
Open
Rich-T-kid wants to merge 4 commits into
Open
Rich-T-kid wants to merge 4 commits into
Rich-T-kid wants to merge 4 commits into
Conversation
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (2637e5b) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
github-actions
Bot
added
the
arrow-kernels
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (2637e5b) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-2bx3iakq-2158138a-9hhg Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
3 times, most recently
from
October 26, 2026 14:58
154cace to
56c6e8c
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (967d3d0) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (967d3d0) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1tw271hz-870aa374-gtwm Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 26, 2026 15:49
967d3d0 to
f66a20c
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (f66a20c) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
2 times, most recently
from
October 26, 2026 16:11
e9faa76 to
0efb530
Compare
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (f66a20c) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ken38iy-4504e04d-ct9z Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 26, 2026 16:33
0efb530 to
2502bea
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (2502bea) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (2502bea) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1exwvsr2-39029fda-8h8s Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 26, 2026 17:32
2502bea to
a88e90c
Compare
Contributor
Author
created #11921 for this |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 26, 2026 17:44
a88e90c to
05de52d
Compare
Pack selected Boolean values and validity bits directly into the in-progress builders, avoiding temporary filtered arrays. Check filter masks before reading source words so sparse inputs skip empty chunks. Use separate loops for sources with and without actual nulls, and carry compressed fragments across chunk boundaries while preserving lazy validity allocation.
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 26, 2026 18:09
05de52d to
16e65c3
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (16e65c3) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Rich-T-kid
marked this pull request as ready for review
October 26, 2026 18:12
Contributor
Author
|
@mbutrovich if your interested this is ready for review. Should be the final PR related to booleans in the coalescer kernel & it implements the ideas you mentioned here #11732 (comment) 🚀 |
Contributor
Author
|
run benchmark filter_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: filter_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (4d091cd) to 69be95c (merge-base) diff Run configurationrun benchmark filter_kernels
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (e22fa80) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-a7pbla7y-9f7e0324-m8jd Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (4d091cd) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1rom390t-571e000f-ggq5 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: filter_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (4d091cd) to 69be95c (merge-base) diff Run configurationrun benchmark filter_kernels
shards: 1
DetailsPer-runner informationfilter_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1rom390t-571e000f-jrjb Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
mbutrovich
reviewed
Oct 27, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Rich-T-kid. Left an inline comment about a proposed code change, not sure if it's a net win. We'd have to benchmark with the bot.
Comment on lines
+848
to
+861
| fn push(&mut self, bits: u64, count: u32) -> Option<u64> { | ||
| self.current |= bits << self.filled; | ||
| let total = self.filled + mask.count_ones(); | ||
| let total = self.filled + count; | ||
| if total < 64 { | ||
| self.filled = total; | ||
| } else { | ||
| // SAFETY: `count` is the number of set bits in the filter, so | ||
| // at most `count / 64` words are ever completed and the | ||
| // buffer holds `count / 64 + 1` | ||
| unsafe { self.ptr.add(self.idx).write(self.current) }; | ||
| self.idx += 1; | ||
| // `bits >> (64 - filled)`, written so that `filled == 0` | ||
| // shifts everything out | ||
| self.current = (bits >> 1) >> (63 - self.filled); | ||
| self.filled = total - 64; | ||
| return None; | ||
| } | ||
|
|
||
| let complete = self.current; | ||
| // `bits >> (64 - filled)`, written so that `filled == 0` | ||
| // shifts everything out | ||
| self.current = (bits >> 1) >> (63 - self.filled); | ||
| self.filled = total - 64; | ||
| Some(complete) |
Contributor
There was a problem hiding this comment.
Does sharing this loop slow down the regular Boolean filter? The 10% and 80% single_boolean rows are above the 1/16 sparse cutoff (coalesce.rs), so they measure collect_filter_bits_compress, not the fused path (#11921 says the same). On the bot, the 10% rows have been 6% to 8% slower since the Packer refactor (run).
As I read the codegen, returning Option<u64> lets LLVM turn the total < 64 branch into selects, so the carry is computed on every non-zero mask word. With thin LTO on aarch64, the path where no word completes goes from 5 instructions at the merge-base to 12. These are the added ones:
lsr x10, x23, #1 // carry, now on every word
eor w11, w11, #0x3f
lsr x10, x10, x11
cmp w20, #0x3f
csel w11, w11, w20, hi // filled
csel x10, x10, x21, hi // current
cmp w20, #0x40
With -C target-cpu=x86-64-v3, current also moves to the stack, with a load and a store on every non-zero mask word.
What do you think about one packer for both bitmaps? They keep the same number of bits per word, so they can share filled, and completed words can be emitted inside the branch as at the merge-base. This also drops PAIR, the Option and the unwraps:
#[inline(always)]
fn filter_bits_compress_impl<const N: usize>(
buffers: [&BooleanBuffer; N],
predicate: &FilterPredicate,
mut append: impl FnMut([u64; N], usize),
) {
/// Packs the bits extracted from successive words into the low `filled`
/// bits of each of `current`. Every bitmap keeps the same number of bits
/// per word, so they share `filled`.
struct Packer<const N: usize> {
current: [u64; N],
filled: u32,
}
impl<const N: usize> Packer<N> {
#[inline(always)]
fn push(
&mut self,
values: [u64; N],
mask: u64,
append: &mut impl FnMut([u64; N], usize),
) {
let mut bits = [0; N];
for (bits, values) in bits.iter_mut().zip(values) {
*bits = bit_util::compress(values, mask);
}
for (current, bits) in self.current.iter_mut().zip(bits) {
*current |= bits << self.filled;
}
let total = self.filled + mask.count_ones();
if total < 64 {
self.filled = total;
} else {
append(self.current, 64);
for (current, bits) in self.current.iter_mut().zip(bits) {
// `bits >> (64 - filled)`, written so that `filled == 0`
// shifts everything out
*current = (bits >> 1) >> (63 - self.filled);
}
self.filled = total - 64;
}
}
}
let len = predicate.filter.len();
for buffer in buffers {
assert!(buffer.len() >= len);
}
let mask_chunks = predicate.filter.values().bit_chunks();
let chunks = buffers.map(|buffer| BitChunks::new(buffer.values(), buffer.offset(), len));
debug_assert_eq!(predicate.count, predicate.filter.true_count());
let mut packer = Packer { current: [0; N], filled: 0 };
for (index, mask) in mask_chunks.iter().enumerate() {
// Words with no kept bits are skipped before the corresponding values
// are read, so only the mask is touched for them
if mask == 0 {
continue;
}
packer.push(chunks.each_ref().map(|c| c.chunk(index)), mask, &mut append);
}
let mask = mask_chunks.remainder_bits();
if mask != 0 {
packer.push(chunks.each_ref().map(|c| c.remainder_bits()), mask, &mut append);
}
if packer.filled != 0 {
append(packer.current, packer.filled as usize);
}
}filter_bits_compress and filter_bits_compress_pair then pass [buffer] and [first, second]. Two details matter for the generated code. The compress step needs to be a loop, because values.map(...) leaves compress out of line on aarch64. And filter_bits_compress should be #[inline(always)], so the loop lands in collect_filter_bits_compress and idx stays in a register instead of being stored through the closure on every completed word.
Here is the change against the merge-base on an Apple silicon Mac with thin LTO, as the median of three interleaved runs. Each cell gives the default build first, then the build with -C llvm-args=-align-all-functions=7 -C llvm-args=-align-all-nofallthru-blocks=7:
| case | this PR | shared packer |
|---|---|---|
| no nulls, 1% selected | 11% / 14% faster | 25% / 25% faster |
| 10% nulls, 1% selected | 30% / 34% faster | 34% / 34% faster |
| no nulls, 10% selected | 3% faster / 4% slower | 4% / 2% faster |
| 10% nulls, 10% selected | 3% faster / 2% slower | 7% / 2% faster |
| no nulls, 80% selected | 1% / 7% slower | 5% / 5% slower |
| 10% nulls, 80% selected | 3% / 11% slower | 7% / 9% slower |
The 80% rows are still slower with both versions. A profile puts about 80% of that case in the compress loop, and I couldn't find the cause in the listings. Could you look into that case and rerun the bot, which runs on Neoverse-V2?
Could you also add a Boolean data case to filter_kernels, in its own commit first? Today it uses Boolean arrays only as filter masks.
Contributor
Author
There was a problem hiding this comment.
Thanks for digging into this. I tried the shared packer as suggested, including the loop around compress and #[inline(always)] on filter_bits_compress, and added the Boolean filter_kernels cases in a separate commit.
On Neoverse-V2, the shared packer made the regular Boolean path worse: the 80% cases regressed by about 19% without nulls and 22% with nulls, while the 10% cases regressed by about 5-6% (run). I also tried keeping the shared packer but restoring #[inline(never)]; that was slightly worse again, with 21-24% regressions at 80% and 8-11% at 10% (run).
I reverted the shared-packer change. The final-head run is back to roughly flat at 80% (+2% without nulls, flat with nulls), while preserving the sparse gains (run). The 10% cases are still about 7-8% slower, so that part remains unresolved, but the shared packer does not appear to be the fix on Neoverse-V2.
Contributor
Author
|
run benchmark coalesce_kernels |
|
Hi @Rich-T-kid, your benchmark configuration could not be parsed (#11918 (comment)). Error: Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
1 similar comment
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (5bcede6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (5bcede6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (5bcede6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-dz1ftj4m-11a3fe47-twbr Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (5bcede6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-dz1ftj4m-11a3fe47-rzzn Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (b12e8f7) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
github-actions
Bot
added
the
arrow-core
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (b12e8f7) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-719asvrk-4ddf4dad-4r5x Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 29, 2026 16:56
b12e8f7 to
560aec6
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (560aec6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (560aec6) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-19rvsxps-9e352737-8wms Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-boolean-filter-words
branch
from
October 29, 2026 17:11
560aec6 to
f059b1e
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-boolean-filter-words (f059b1e) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
|
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-boolean-filter-words (f059b1e) to 69be95c (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_boolean
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-19rvsxps-9e352737-pg5k Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes
Why
Follow-up to #11732 and #11867.
copy_rows_by_filter_frompreviously ran the Boolean filter kernel, producing temporary value and validity buffers. The coalescer immediately copied those buffers into its builders and discarded them. Now that the builders can append packed words, the filter kernel can write straight into the coalescer output.What changed
The Boolean coalescer now uses the existing word-at-a-time compression kernel. Non-null and all-valid sources append values and advance validity lazily. Nullable sources compress values and validity together in one pass over the filter mask.
Empty filter words are skipped before reading the corresponding source words, which avoids touching most of the source for sparse filters. Sliced bitmaps, a filter shorter than the source, and an empty remainder are handled by the same chunked path.
Note on
filter.rsfilter_bits_compressnow emits packed words through a callback instead of owning its output buffer. The regular Boolean filter still collects those words into aBufferthroughcollect_filter_bits_compress, so its strategy selection and result are unchanged. The paired form lets the coalescer compress values and validity with one mask traversal. These helpers remain crate-private.Example
Take this eight-row input:
The null predicate at row 3 is normalized to
false, so rows 1, 2, 4, and 6 are selected. With row 0 in the least-significant bit:The compression kernel produces:
Those four bits are appended directly to the output builders. The result is:
Larger inputs use the same operation on each 64-row chunk, carrying partial words between chunks.
Results
Test coverage
The tests compare the coalescer output with the Boolean filter kernel for nullable, non-null, and all-valid sources. They cover sliced inputs, null predicate entries, word boundaries, carry between words, a filter shorter than the source, an exact multiple of 64, and optimized and unoptimized predicates. A filter longer than the source is checked separately.