Skip to content

Prune nested leaves in the projection fallback for files without field ids #3480

Open

Description

@zhangfengcdt

Is your feature request related to a problem or challenge?

For a Parquet file with no field ids and no name mapping, get_arrow_projection_mask_fallback projects each requested top-level column whole, so the reader decodes every leaf under it.

After #3437 (fix for #3436), fields inside a struct of such a file have no id and read as null, as in iceberg-java. Their leaves are still decoded and then discarded by RecordBatchTransformer. For wide or deeply nested struct columns this is wasted I/O and decoding.

Raised in review of #3437.

Describe the solution you'd like

Project only the leaves the result needs:

  • Primitive top-level columns, and the element of a list and the key and value of a map, which match by position and keep their values.
  • One leaf per struct. A struct keeps the file's validity, which Parquet stores in the definition levels of its leaves, so at least one leaf must still be read. Picking a cheap leaf (for example the narrowest primitive) would keep the cost low.

The read result should not change; only the projection mask does.

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg community

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions