scan_partition_plan computes a single uniform factor from the mean estimated decompressed size of the projected columns across sampled footers, then applies that factor to every file. When file sizes vary across a dataset, the mean is a poor representative. Small files that should be fused into a single task are split instead; large full-sized files and small leftover files from compaction or write job boundaries receive the same factor.
Polars already fetches file sizes from object_store during glob expansion. Each ObjectMeta returned by the listing has a .size field. Polars uses it only to filter out empty objects and then discards it. Neither ScanSources nor FileInfo carry file sizes, so they never reach cudf-polars.
Once per-file sizes are available in the Scan IR, replace the uniform factor in scan_partition_plan with greedy binning: walk files in order, accumulate estimated decompressed size, and emit a task when the cumulative size exceeds target_partition_size. Files larger than target_partition_size are split at row group boundaries as today. For datasets with uniform file sizes this produces the same result as the current approach. When listing sizes are not available (paths given explicitly rather than via glob), fall back to the current mean-based approach.
Related
scan_partition_plancomputes a single uniform factor from the mean estimated decompressed size of the projected columns across sampled footers, then applies that factor to every file. When file sizes vary across a dataset, the mean is a poor representative. Small files that should be fused into a single task are split instead; large full-sized files and small leftover files from compaction or write job boundaries receive the same factor.Polars already fetches file sizes from
object_storeduring glob expansion. EachObjectMetareturned by the listing has a.sizefield. Polars uses it only to filter out empty objects and then discards it. NeitherScanSourcesnorFileInfocarry file sizes, so they never reach cudf-polars.Once per-file sizes are available in the
ScanIR, replace the uniform factor inscan_partition_planwith greedy binning: walk files in order, accumulate estimated decompressed size, and emit a task when the cumulative size exceedstarget_partition_size. Files larger thantarget_partition_sizeare split at row group boundaries as today. For datasets with uniform file sizes this produces the same result as the current approach. When listing sizes are not available (paths given explicitly rather than via glob), fall back to the current mean-based approach.Related
Scannode in cudf-polars #22734 file size is in the sameObjectMetastruct and could be surfaced alongside etag/last-modified