|
| 1 | +<!-- SPDX-License-Identifier: Apache-2.0 --> |
| 2 | +<!-- SPDX-FileCopyrightText: Copyright the Vortex contributors --> |
| 3 | + |
| 4 | +# Bloom skipping-index findings |
| 5 | + |
| 6 | +This implementation answers the core questions in |
| 7 | +[#8901](https://github.com/vortex-data/vortex/issues/8901): a custom aggregate can be persisted in a |
| 8 | +`ZonedLayout`, rebound on a fresh reader, and used by an equality rewrite to skip zones without |
| 9 | +changing scan planning. |
| 10 | + |
| 11 | +Substring search is intentionally out of scope. `LIKE '%substr%'` needs a separate inverted-index |
| 12 | +design rather than overloading the per-zone Bloom interface. |
| 13 | + |
| 14 | +## Result |
| 15 | + |
| 16 | +The Bloom skipping index consists of: |
| 17 | + |
| 18 | +- a serializable per-zone aggregate stored as `Binary`, |
| 19 | +- a scalar membership probe, |
| 20 | +- an equality `StatsRewriteRule` that emits |
| 21 | + `not(bloom_contains(stat(root(), bloom), literal))`, and |
| 22 | +- a `SkipIndex` bundle plus `ZonedLayoutOptions::with_skip_index` for explicit write-side |
| 23 | + declaration. |
| 24 | + |
| 25 | +The roundtrip test writes four zones through the normal `WriteStrategyBuilder` pipeline, opens the |
| 26 | +file with a fresh registered session, and checks both hits and misses. The hit keeps exactly one |
| 27 | +zone. The absent value is deliberately inside every zone's min/max range, but the Bloom proof |
| 28 | +skips all four zones. Indexed results are array-equal to a full scan through a fresh session that |
| 29 | +does not know about the custom index. |
| 30 | + |
| 31 | +Unknown readers remain safe when `allow_unknown()` is enabled: they ignore the unknown aggregate |
| 32 | +and read the data child without pruning. #8904 provides unknown-aggregate ignorability and is now |
| 33 | +on `develop`. #8905 is still open as of 2026-07-23, so its empty-zone-map reader bypass remains on |
| 34 | +this branch as a prerequisite. |
| 35 | + |
| 36 | +## Focused benchmark |
| 37 | + |
| 38 | +Command: |
| 39 | + |
| 40 | +```text |
| 41 | +cargo test --release -p vortex-file --test bloom_skip_index bloom_point_lookup_benchmark -- --ignored --nocapture |
| 42 | +``` |
| 43 | + |
| 44 | +The benchmark uses 2,097,152 interleaved, high-cardinality `i64` values in 256 zones of 8,192 rows. |
| 45 | +Interleaving makes the point lookup fall inside every zone's min/max range, so ordinary range |
| 46 | +statistics cannot skip a zone. Timings are medians of nine fresh-reader scans after two warmups. |
| 47 | +Segment payload counts are measured below the file reader with a counting `SegmentSource`. |
| 48 | + |
| 49 | +| Case | Zones pruned | File size | Median | Segment requests | Unique segments | Segment payload | |
| 50 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| 51 | +| No Bloom | 0/256 | 16,845,924 B | 1.541 ms | 258 | 257 | 16,897,520 B | |
| 52 | +| Bloom | 251/256 | 18,947,380 B | 0.571 ms | 7 | 6 | 2,504,196 B | |
| 53 | + |
| 54 | +The Bloom scan was **2.70x faster** and moved **6.75x less segment payload**. Five zones survived: |
| 55 | +the one true hit and four false positives. The file grew by 2,101,456 bytes, or 12.5%, from one |
| 56 | +8 KiB filter per zone plus layout overhead. |
| 57 | + |
| 58 | +The negative control uses cardinality 16 and queries a value present in every zone: |
| 59 | + |
| 60 | +| Case | Zones pruned | File size | Median | Segment requests | Unique segments | Segment payload | |
| 61 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| 62 | +| No Bloom | 0/256 | 16,843,876 B | 1.807 ms | 513 | 257 | 33,651,004 B | |
| 63 | +| Bloom | 0/256 | 18,945,332 B | 1.832 ms | 513 | 257 | 35,752,396 B | |
| 64 | + |
| 65 | +With nothing to prune, the indexed scan was 1.4% slower and moved 6.2% more segment payload. A |
| 66 | +Bloom index only makes sense when avoided data reads pay for loading and probing the zone table. |
| 67 | + |
| 68 | +## Existing SQL benchmark |
| 69 | + |
| 70 | +The benchmark writer adds a Bloom index only to ClickBench `UserID`, using 8,192-row zones. It |
| 71 | +retains the normal dictionary, compressor, coalescing, and flat-layout stages and writes the |
| 72 | +candidate to a separate `vortex-file-bloom` directory. The candidate job times unchanged |
| 73 | +DataFusion/Parquet alongside indexed DataFusion/Vortex so runner-wide movement is visible. |
| 74 | + |
| 75 | +The earlier mixed-index run already isolated the point-lookup behavior: ClickBench q19 improved |
| 76 | +from 29.23 ms to 21.42 ms, approximately **1.33x faster after its Parquet control**. One scan proved |
| 77 | +12,294 of 12,299 zones absent and kept only five. The full 43-query suite stayed neutral because the |
| 78 | +other queries could not use the equality index. |
| 79 | + |
| 80 | +A Bloom-only controlled rerun is required to replace the mixed-index file-size number and confirm |
| 81 | +the cleaned implementation's wall time. |
| 82 | + |
| 83 | +## Interface recommendation |
| 84 | + |
| 85 | +Keep the aggregate, scalar-function, and rewrite registries composable, but expose one bundle at |
| 86 | +the registration and write-declaration boundary: |
| 87 | + |
| 88 | +```rust |
| 89 | +pub trait SkipIndex: Debug + Send + Sync + 'static { |
| 90 | + fn aggregate_fn(&self, input_dtype: &DType) -> Option<AggregateFnRef>; |
| 91 | + fn register(&self, session: &VortexSession); |
| 92 | +} |
| 93 | + |
| 94 | +impl ZonedLayoutOptions { |
| 95 | + pub fn with_skip_index<I: SkipIndex + ?Sized>( |
| 96 | + self, |
| 97 | + index: &I, |
| 98 | + input_dtype: &DType, |
| 99 | + session: &VortexSession, |
| 100 | + ) -> VortexResult<Self>; |
| 101 | +} |
| 102 | +``` |
| 103 | + |
| 104 | +This keeps the implementation pieces reusable while giving callers one object to register on |
| 105 | +writer and reader sessions and one explicit per-column declaration. The aggregate, probe, and |
| 106 | +rewrite types are private behind `BloomSkipIndex`; users cannot accidentally register only part of |
| 107 | +the index. |
| 108 | + |
| 109 | +`WriteStrategyBuilder::with_field_zoned_options` is the per-column seam. It changes only the zoned |
| 110 | +statistics for a field and preserves the normal physical data pipeline. |
| 111 | + |
| 112 | +## Off-the-shelf filter primitive |
| 113 | + |
| 114 | +Arrow Rust's `parquet::bloom_filter::Sbbf` provides stable split-block Bloom semantics, byte |
| 115 | +serialization, membership checks, NDV/FPP sizing, and folding. It does not replace the |
| 116 | +Vortex-specific aggregate persistence, stats rewrite, probe expression, or writer declaration. |
| 117 | +Using it directly from `vortex-layout` would also add an undesirable Parquet dependency. |
| 118 | + |
| 119 | +For production, put an SBBF-compatible implementation in a small neutral crate, then use it behind |
| 120 | +`BloomSkipIndex`. |
| 121 | + |
| 122 | +## Remaining work |
| 123 | + |
| 124 | +- Aggregate identity includes serialized sizing options. Readers must currently register a rewrite |
| 125 | + with exactly the writer's byte and hash counts; production discovery should be automatic. |
| 126 | +- The hash algorithm, format version, supported dtypes, null semantics, sizing policy, and |
| 127 | + saturation behavior need an explicit compatibility contract. This implementation supports |
| 128 | + `i64` equality only. |
| 129 | +- Unknown aggregate fallback disables the whole zone map rather than retaining known statistics. |
| 130 | + That is safe but leaves pruning performance on the table. |
| 131 | +- Filter sizing and zone length must remain workload-aware and opt-in. The low-cardinality control |
| 132 | + demonstrates the cost when pruning cannot occur. |
| 133 | + |
| 134 | +Recommendation: continue with an opt-in equality Bloom index and the bundled `SkipIndex` interface. |
| 135 | +The correctness proof, synthetic I/O reduction, and ClickBench point lookup justify a |
| 136 | +production-quality follow-up. Design substring search separately as an inverted index. |
0 commit comments