Skip to content

Commit eb70aeb

Browse files
committed
Record skipping-index benchmark findings
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
1 parent 6df9593 commit eb70aeb

1 file changed

Lines changed: 63 additions & 3 deletions

File tree

‎SKIPPING_INDEX_SPIKE.md‎

Lines changed: 63 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,61 @@ its dictionary, compressor, coalescing, or flat-layout stages:
6767

6868
The benchmark artifact is written to a separate `vortex-file-skipping` directory so cached normal
6969
Vortex files cannot contaminate the comparison. The focused draft-PR run covers ClickBench,
70-
ClickBench sorted, FineWeb NVMe, and FineWeb S3 with DataFusion/Vortex. Results will be recorded here
71-
after the dedicated benchmark run completes.
70+
ClickBench sorted, FineWeb NVMe, and FineWeb S3. Each candidate job times unchanged
71+
DataFusion/Parquet alongside indexed DataFusion/Vortex, and the reporting harness subtracts the
72+
Parquet control shift from the attributed Vortex impact. The controlled run is
73+
[29951570194](https://github.com/vortex-data/vortex/actions/runs/29951570194).
74+
75+
| Suite | Vortex PR/base | Parquet control | Attributed Vortex impact | Indexed Vortex size | Verdict |
76+
| --- | ---: | ---: | ---: | ---: | --- |
77+
| ClickBench NVMe | 1.004x | 1.014x | -1.0% | +4.3% | No suite-wide signal |
78+
| ClickBench sorted NVMe | 0.944x | 0.991x | -4.8% | +2.9% | No clear signal; noisy |
79+
| FineWeb NVMe | 1.125x | 1.039x | +8.3% | +3.3% | No clear suite-wide signal; indexed queries regress |
80+
| FineWeb S3 | 1.057x | 0.984x | +7.5% | not captured | No clear signal; environment noisy |
81+
82+
The equality result does materialize in a real benchmark. ClickBench q19 is the indexed
83+
`UserID = 435090932899640449` point lookup. It improves from 29.23 ms to 21.42 ms, a raw 27%
84+
reduction. The matching unchanged Parquet query moves from 28.17 ms to 27.45 ms, so the
85+
control-adjusted Vortex ratio is approximately `0.73 / 0.97 = 0.75`: **about 25% faster, or
86+
1.33x**. The diagnostic mask explains the timing: one scan examines 12,299 zones across 100 files,
87+
proves 12,294 absent, and keeps only five. The entire 43-query ClickBench suite remains neutral
88+
because the other queries cannot use this point-lookup index. The +4.3% file-size cost includes the
89+
`UserID` bloom and two larger trigram indexes, so it is not the isolated equality-index overhead.
90+
The full tables are in the
91+
[ClickBench benchmark comment](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050548603).
92+
93+
The trigram result is negative. FineWeb's one 1,023-zone table produced these masks:
94+
95+
| Predicate | Indexed column | Zones pruned | Average fill |
96+
| --- | --- | ---: | ---: |
97+
| `url LIKE '%google%'` | `url` | 21/1,023 | 35.4% |
98+
| `text LIKE '%Google%'` | `text` | 0/1,023 | 79.9% |
99+
| `text LIKE '% vortex %'` | `text` | 0/1,023 | 79.9% |
100+
| `url LIKE '%espn%'` | `url` | 328/1,023 | 35.4% |
101+
| `url LIKE '%www.espn.go.com%'` | `url` | 625/1,023 | 35.4% |
102+
| `url LIKE '%espn.go.com%'` | `url` | 607/1,023 | 35.4% |
103+
| `file_path LIKE '%/CC-MAIN-2014-%'` | `file_path` | 0/1,023 | 5.6% |
104+
105+
Despite real pruning, every trigram-targeted FineWeb query is slower after its matching Parquet
106+
control adjustment. On NVMe the adjusted costs are approximately +21% for q3, +3% for q5, +10%
107+
for q6, +25% for q7, and +36% for q8. On S3 they are approximately +13%, +28%, +15%, +19%, and
108+
+27%. The 16 KiB text filter is nearly saturated, while the low-fill `file_path` case shows a
109+
different weakness: all of the common three-byte grams occur somewhere in every zone, even though
110+
the complete string does not. Reading and probing the index then adds cost without avoiding enough
111+
data. See the [FineWeb NVMe](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050536937)
112+
and [FineWeb S3](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050543034) reports.
113+
114+
ClickBench q20-q23 and sorted q23 contain the expected `URL`/`Title` substring predicates, but the
115+
trigram probe executes zero times in both job logs. Their expression arrives in a shape not matched
116+
by this rewrite, so any timing movement is not evidence for the index. In particular, sorted q23's
117+
raw 20% improvement cannot be credited to trigram pruning. The
118+
[sorted report](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050576221) is retained
119+
as a useful expression-pushdown gap to investigate.
120+
121+
The SQL harness reports wall time, file size, and the added zone-mask diagnostics, but not
122+
per-query segment requests or payload bytes. Those I/O counters therefore come from the controlled
123+
synthetic benchmark above; the real-suite claim is intentionally limited to wall time, file size,
124+
and zones pruned.
72125

73126
### Off-the-shelf pieces
74127

@@ -118,4 +171,11 @@ data pipeline instead of requiring callers to reconstruct it.
118171
- The hash algorithm, format version, supported dtypes, null semantics, sizing policy, and saturation behavior need an explicit compatibility contract. This spike supports `i64` equality and byte-oriented UTF-8 trigrams only.
119172
- Bloom stats for all zones are regular child-array data. Their layout and caching should be profiled for many indexed columns and remote reads.
120173

121-
Recommendation: pursue the bundle interface and an explicit per-column writer declaration, while retaining the three lower-level registries. The selective equality benchmark is large enough to justify a production-quality experiment, but index selection must remain workload-aware and opt-in. The SQL run must show that real substring selectivity survives filter saturation and pays back the added file bytes before the trigram prototype is considered a win.
174+
Recommendation: pursue the bundle interface and an explicit per-column writer declaration while
175+
retaining the three lower-level registries. A production-quality equality Bloom experiment is
176+
justified for opt-in, high-cardinality point-lookups: it is correct, prunes as expected, and speeds
177+
up ClickBench q19 after a same-run control. Do not pursue this generic fixed-size trigram Bloom as
178+
the production design. First fix rewrite coverage for the actual ClickBench expression shape, then
179+
prototype adaptive sizing/saturation cutoffs and a rarer-token or sequence-aware representation
180+
that can reject zones when common individual trigrams cannot. Any successor should have to beat the
181+
current negative FineWeb results, including index bytes and probe cost, before being enabled.

0 commit comments

Comments
 (0)