[cigrainger]

Summary

New Dux.Remote.PartitionPruner module that skips Parquet files whose Hive partition values don't match pipeline filter predicates. Integrated into the Partitioner before size-balanced assignment.

How it works

Given files like s3://bucket/year=2024/month=01/data.parquet and a filter year = 2024, the pruner:

1. Parses key=value pairs from file path segments
2. Extracts simple equality predicates from {:filter, sql} ops
3. Skips files where partition values don't match

Before: 6 files (year=2023 × 3 months + year=2024 × 3 months) → all 6 distributed
After:  filter(year = 2024) → only 3 files distributed, 3 skipped

What it handles

• Quoted and unquoted column names ("year" = 2024, year = 2024)
• String values (region = 'US')
• AND-connected predicates (year = 2024 AND month = '01')
• Multiple separate filter ops
• Mixed partitioned + non-partitioned files (non-partitioned files pass through safely)

What it doesn't handle (safe — just reads more data)

• OR predicates, IN lists, range comparisons
• Complex expressions, function calls
• Non-Hive partition schemes

Test plan

• 15 unit tests for PartitionPruner (extraction, pruning, edge cases)
• 3 integration tests in coordinator (Hive dirs on disk, single/multi-column, non-partition filter)
• 2 :peer tests (end-to-end pruning with real distributed workers + Hive-partitioned Parquet)
• 596 non-distributed tests pass, Credo clean
• 7 peer tests pass

🤖 Generated with Claude Code