Skip to content

feat: prefix/contains pushdown #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joseph-isaacs wants to merge 45 commits into
base: develop
Choose a base branch
from
Open

feat: prefix/contains pushdown #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
c44ded8
feat(search): compressed-domain prefix/contains DFA matching
claude May 30, 2026
8810d81
feat(search): packed-bitset RowMask + borrowed SearchParts view; C++ …
claude May 30, 2026
8b50851
refactor(search): rename search_for_each -> search_callback; add roof…
claude May 30, 2026
1383a54
perf(search): stateless matchers + KMP fast/slow split + prefix first…
claude May 30, 2026
109e5d5
perf(search): two-pass branchless prefix prefilter; make first-token …
claude May 30, 2026
df0de90
perf(search): AVX2 pass-1 for the prefix first-token prefilter
claude May 30, 2026
1d5777f
bench(search): read Binary/LargeBinary/BinaryView parquet columns
claude May 30, 2026
26ab2bb
bench(search): drop duplicate AsArray import
claude May 30, 2026
c8af16d
bench(search): remove re-introduced duplicate AsArray import
claude May 30, 2026
092fd0e
perf(search): bitmap-merge fast path for prefix search() -> RowMask
claude May 30, 2026
5cb7c12
bench(search): add memchr dev-dependency for the arrow-like contains …
claude May 30, 2026
240825b
perf(search): two-level contains prefilter (scalar)
claude May 30, 2026
9766ff1
perf(search): branchless contains pass-1 so LLVM auto-vectorizes it
claude May 30, 2026
71f984e
fix(search): correct contains pass-1 call site (commit 9766ff1 was bu…
claude May 30, 2026
355e6f4
perf(search): keep contains pass-1 scalar+early-exit (revert bad auto…
claude May 30, 2026
879c8e0
fix(search): clippy clean (commit 355e6f4 failed clippy)
claude May 30, 2026
483ea60
bench(search): arrow baseline packs verdicts with collect_bool (Boole…
claude May 30, 2026
53186a6
perf(search): Teddy-style 2-code chain prefilter for contains
claude May 30, 2026
6dfe987
bench(search): ONPAIR_NEEDLES override to benchmark specific queries
claude May 31, 2026
e410a35
test(search): keep token-level DFA dump tool (KmpAutomaton::dump_dfa …
claude May 31, 2026
aeff95b
test(search): keep inner_probe tool measuring the SIMD-able INNER filter
claude May 31, 2026
262d8ba
perf(search): SIMD INNER range filter for contains (opt-in, ONPAIR_IN…
claude May 31, 2026
547dc40
test(search): keep inner_ranges_dump tool (exact SIMD prefilter ranges)
claude May 31, 2026
35814ab
test(search): tools to probe DFA boundary-state reachability (LPM pru…
claude May 31, 2026
99a323d
perf(search): tighten INNER prefilter to completing+reachable transit…
claude May 31, 2026
625f4e3
bench(search): ONPAIR_BENCH_MAX_ROWS cap for large corpora (FineWeb)
claude May 31, 2026
ec0a3a2
refactor(search): remove dead num_tokens guard in scan_contains
claude May 31, 2026
25fb845
perf(search): 3-layer SIMD→scalar→KMP funnel for contains (opt-in)
claude May 31, 2026
d2521ba
docs(search): handover for compressed-domain LIKE search work
claude May 31, 2026
9ff0d95
refactor(search): slim RowMask to a [u64]+len wrapper
claude May 31, 2026
5a52eb8
fix(search): repair bench after RowMask slim (count_ones removed)
claude May 31, 2026
b4dd9d1
perf(search): drop per-row fallible offset conversion in scan loops
claude May 31, 2026
8f5c260
refactor(search): factor row_codes helper; promote optimization memor…
claude May 31, 2026
abf9857
fix(search): add the row_codes helper (commit 8f5c260 was broken)
claude May 31, 2026
f4ee05f
experiment(search): #1 LPM-aware INNER pruning — DISPROVED (unsound)
claude May 31, 2026
4f40437
experiment(search): #7 bits sweep — bits=16 is search-optimal (no tra…
claude May 31, 2026
9b03118
perf(search): AVX-512BW prefix kernel — ~1.2x over AVX2 (experiment #…
claude May 31, 2026
a0e4a96
experiment(search): #2 packed first_codes index — NOT WORTH IT
claude May 31, 2026
17c67d8
experiment(search): #4 first_two_codes multi-token prefix — DISPROVED
claude May 31, 2026
61626a4
experiment(search): #5 selectivity-adaptive contains — DISPROVED
claude May 31, 2026
c8ee514
experiment(search): #10 TPC-H search + #8 corpus characterization
claude May 31, 2026
5b94fa0
style: cargo fmt (fix CI fmt --check)
claude Jun 1, 2026
5edb52e
Merge remote-tracking branch 'origin/develop' into claude/cpp-dfa-con…
joseph-isaacs Jun 1, 2026
a0a024d
refactor(search): strip experimental scaffolding, tighten public API
claude Jun 3, 2026
5245f78
style(search): fix rustfmt check
joseph-isaacs Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ rand = "0.9.0"
[dev-dependencies]
divan = { package = "codspeed-divan-compat", version = "4.0.4" }
rstest = "0.26.1"
memchr = "2"
arrow-array = "57.1"
arrow-buffer = "57.1"
arrow-schema = "57.1"
parquet = "57.1"
tpchgen = "2.0.2"
Expand All @@ -50,3 +52,7 @@ harness = false
[[bench]]
name = "clickbench"
harness = false

[[bench]]
name = "search"
harness = false
Loading
Loading