Repository navigation
Distributed perf hardening: 3-12x faster fan-out and merge - #33
Merged
Merged
Conversation
…olling, regex agg Performance: - fan_out uses ordered: false (streaming path already did) — 3.6x faster median for distributed filter+agg (99ms → 27ms) - Broadcast threshold scales with worker count (total network stays constant) Correctness: - Merger uses word-boundary regex for aggregate detection instead of String.contains? — prevents COUNT_DISTINCT matching COUNT branch - FLAME spin_up replaces Process.sleep(100) with :pg.get_members polling loop (10ms interval, 5s timeout) Benchmark results (2 local workers, 100K rows): distributed filter+agg: 99ms → 27ms (3.6x) streaming SUM+COUNT: 491ms → 39ms (12.5x) streaming MIN+MAX: 64ms → 16ms (3.9x) broadcast join: 112ms → 32ms (3.5x) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Regex structs contain references that can't be injected into module attributes on all OTP/Elixir versions. Use inline cond with ~r// sigils instead — they're still compiled at compile time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance
ordered: falsein batch fan_out — the streaming path already usedordered: false, but the batch path forced sequential collection. Results are partitioned into success/failure post-hoc, so ordering is irrelevant. This is the biggest win.threshold / n_workers) so total network cost stays constant regardless of cluster size. 2 workers: 128MB each. 10 workers: 25MB each.Correctness
String.contains?with\b-anchored regex module attributes. PreventsCOUNT_DISTINCT(matching theCOUNT(branch.Process.sleep(100)race condition with:pg.get_memberspolling loop (10ms interval, 5s timeout).Benchmark Results (2 local workers, 100K rows)
Full benchmark results in
bench/results/.Test plan
🤖 Generated with Claude Code