You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Prune nested leaves in the projection fallback for files without field ids #3480
Is your feature request related to a problem or challenge?
For a Parquet file with no field ids and no name mapping, get_arrow_projection_mask_fallback projects each requested top-level column whole, so the reader decodes every leaf under it.
After #3437 (fix for #3436), fields inside a struct of such a file have no id and read as null, as in iceberg-java. Their leaves are still decoded and then discarded by RecordBatchTransformer. For wide or deeply nested struct columns this is wasted I/O and decoding.
Primitive top-level columns, and the element of a list and the key and value of a map, which match by position and keep their values.
One leaf per struct. A struct keeps the file's validity, which Parquet stores in the definition levels of its leaves, so at least one leaf must still be read. Picking a cheap leaf (for example the narrowest primitive) would keep the cost low.
The read result should not change; only the projection mask does.
Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg community
Activity