[PERF]: Reuse sampled parquet footers from statistics collection

[Matt711]

Statistics collection fetches footers for up to max_footer_samples (default 3) files, but only
stores them in ParquetSourceInfo.cached_parquet_info when all files were sampled:

cudf/python/cudf_polars/cudf_polars/streaming/io.py

Lines 1153 to 1166 in d87da31

	@functools.cache
	def _build_parquet_source(
	paths: tuple[str, ...],
	needed_cols: frozenset[str],
	schema: tuple[tuple[str, DataType], ...],
	max_footer_samples: int,
	max_row_group_samples: int,
	) -> ParquetSourceInfo:
	"""Return cached, fully-computed Parquet datasource information."""
	return ParquetSourceInfo.from_paths(
	paths, needed_cols, schema, max_footer_samples, max_row_group_samples
	)

So prefetch_parquet_file_metadata_for_ir re-fetches those same footers:

cudf/python/cudf_polars/cudf_polars/dsl/utils/io.py

Lines 145 to 157 in d87da31

	cached_parquet_info: dict[str, CachedParquetInfo] = {}
	if stats is not None:
	for node, datasource_info in stats.scan_stats.items():
	if (
	isinstance(node, Scan)
	and node.typ == "parquet"
	and isinstance(datasource_info, ParquetSourceInfo)
	and datasource_info.cached_parquet_info is not None
	):
	for info in datasource_info.cached_parquet_info:
	cached_parquet_info[info.path] = info
	missing_paths = all_paths - set(cached_parquet_info.keys())

Related

• [PERF]: Cache parquet metadata at the session level #23526 session-level CachedParquetInfo cache (across queries; the cache here is within a query)
• [PERF]: Use object metadata from polars Scan node in cudf-polars #22734 object metadata from polars (would eliminate the HEAD per file)