You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit eb70aeb
Browse filesBrowse the repository at this point in the historyBrowse files
The trigram result is negative. FineWeb's one 1,023-zone table produced these masks:
94
+
95
+
| Predicate | Indexed column | Zones pruned | Average fill |
96
+
| --- | --- | ---: | ---: |
97
+
|`url LIKE '%google%'`|`url`| 21/1,023 | 35.4% |
98
+
|`text LIKE '%Google%'`|`text`| 0/1,023 | 79.9% |
99
+
|`text LIKE '% vortex %'`|`text`| 0/1,023 | 79.9% |
100
+
|`url LIKE '%espn%'`|`url`| 328/1,023 | 35.4% |
101
+
|`url LIKE '%www.espn.go.com%'`|`url`| 625/1,023 | 35.4% |
102
+
|`url LIKE '%espn.go.com%'`|`url`| 607/1,023 | 35.4% |
103
+
|`file_path LIKE '%/CC-MAIN-2014-%'`|`file_path`| 0/1,023 | 5.6% |
104
+
105
+
Despite real pruning, every trigram-targeted FineWeb query is slower after its matching Parquet
106
+
control adjustment. On NVMe the adjusted costs are approximately +21% for q3, +3% for q5, +10%
107
+
for q6, +25% for q7, and +36% for q8. On S3 they are approximately +13%, +28%, +15%, +19%, and
108
+
+27%. The 16 KiB text filter is nearly saturated, while the low-fill `file_path` case shows a
109
+
different weakness: all of the common three-byte grams occur somewhere in every zone, even though
110
+
the complete string does not. Reading and probing the index then adds cost without avoiding enough
111
+
data. See the [FineWeb NVMe](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050536937)
112
+
and [FineWeb S3](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050543034) reports.
113
+
114
+
ClickBench q20-q23 and sorted q23 contain the expected `URL`/`Title` substring predicates, but the
115
+
trigram probe executes zero times in both job logs. Their expression arrives in a shape not matched
116
+
by this rewrite, so any timing movement is not evidence for the index. In particular, sorted q23's
117
+
raw 20% improvement cannot be credited to trigram pruning. The
118
+
[sorted report](https://github.com/vortex-data/vortex/pull/8907#issuecomment-5050576221) is retained
119
+
as a useful expression-pushdown gap to investigate.
120
+
121
+
The SQL harness reports wall time, file size, and the added zone-mask diagnostics, but not
122
+
per-query segment requests or payload bytes. Those I/O counters therefore come from the controlled
123
+
synthetic benchmark above; the real-suite claim is intentionally limited to wall time, file size,
124
+
and zones pruned.
72
125
73
126
### Off-the-shelf pieces
74
127
@@ -118,4 +171,11 @@ data pipeline instead of requiring callers to reconstruct it.
118
171
- The hash algorithm, format version, supported dtypes, null semantics, sizing policy, and saturation behavior need an explicit compatibility contract. This spike supports `i64` equality and byte-oriented UTF-8 trigrams only.
119
172
- Bloom stats for all zones are regular child-array data. Their layout and caching should be profiled for many indexed columns and remote reads.
120
173
121
-
Recommendation: pursue the bundle interface and an explicit per-column writer declaration, while retaining the three lower-level registries. The selective equality benchmark is large enough to justify a production-quality experiment, but index selection must remain workload-aware and opt-in. The SQL run must show that real substring selectivity survives filter saturation and pays back the added file bytes before the trigram prototype is considered a win.
174
+
Recommendation: pursue the bundle interface and an explicit per-column writer declaration while
175
+
retaining the three lower-level registries. A production-quality equality Bloom experiment is
176
+
justified for opt-in, high-cardinality point-lookups: it is correct, prunes as expected, and speeds
177
+
up ClickBench q19 after a same-run control. Do not pursue this generic fixed-size trigram Bloom as
178
+
the production design. First fix rewrite coverage for the actual ClickBench expression shape, then
179
+
prototype adaptive sizing/saturation cutoffs and a rarer-token or sequence-aware representation
180
+
that can reject zones when common individual trigrams cannot. Any successor should have to beat the
181
+
current negative FineWeb results, including index bytes and probe cost, before being enabled.
0 commit comments