[cigrainger]

What

USING KEY graph algorithms, PageRank convergence, HyperLogLog COUNT DISTINCT, weighted shortest paths, community detection.

Changes

USING KEY for graph algorithms (DuckDB 1.4.1)

Shortest paths (local + distributed):

• Rewrote to use USING KEY recursive CTE for unweighted BFS
• 1000x intermediate row reduction on medium graphs (600M → 607K rows)
• Weighted paths fall back to UNION + GROUP BY MIN (USING KEY has a limitation with weighted multi-path convergence in the same iteration)

Connected components (local):

• Single SQL statement with USING KEY replaces Elixir iteration loop
• Eliminates per-iteration IPC, convergence check, and Process.put GC workarounds
• DuckDB handles convergence internally

Weighted shortest paths

New :weight option on shortest_paths/3:

Dux.Graph.shortest_paths(graph, 1, weight: :distance)

Uses Bellman-Ford via recursive CTE with GROUP BY MIN.

PageRank convergence detection

• L1 norm convergence check: exits early when SUM(ABS(new_rank - old_rank)) < tolerance
• :tolerance option (default 1e-6) — set to 0 to disable
• :max_iterations option (default 100, up from fixed 20)
• Backwards compatible: :iterations still works
• Emits [:dux, :graph, :iteration, :stop] telemetry per iteration with convergence status
• Typical graphs converge in 8-15 iterations

HyperLogLog for distributed COUNT DISTINCT

Workers now compute approx_count_distinct(x) (DuckDB's built-in HLL, ~2% error) instead of ARRAY_AGG(DISTINCT x). Constant wire cost — one number per worker per group instead of shipping all distinct values.

Label Propagation community detection

New Dux.Graph.communities/2:

Dux.Graph.communities(graph, max_iterations: 20)

Each vertex votes for the most frequent label among neighbors (including its own to prevent oscillation). Tie-broken by smallest label. Converges when labels stabilize.

STDDEV/VARIANCE formula

Kept current additive decomposition (COUNT, SUM, SUM_OF_SQUARES) with DOUBLE precision. Documented the catastrophic cancellation limitation for extreme values (>1e15 with tiny variance). Full Welford would require architectural changes to the Merger's GROUP BY re-aggregation.

Tests

24 new tests covering:

• USING KEY shortest paths: linear chain, diamond, disconnected, max_depth, cycle
• Weighted shortest paths: weight option, cheaper route, zero-weight edges
• USING KEY connected components: single/two components, isolated vertices, karate club
• PageRank convergence: early exit, tolerance:0 disables, karate club top-ranked, backwards compat
• HLL COUNT DISTINCT: approximate accuracy via distributed workers
• Community detection: two cliques, disconnected, karate club, single vertex
• Adversarial: empty graph, 100 isolated vertices, single vertex PageRank

Test plan

• All existing tests pass (481 total)
• 24 new graph improvement tests pass
• mix check clean (format + compile + test + credo)
• All peer/distributed tests pass

🤖 Generated with Claude Code