Skip to content

Allow skipping directory checks in read_parquet #408

@hombit

Description

@hombit

Feature request

Since we switched to fsspec.parquet.open_parquet_file for our parquet reading code, we added extra round-trips for remote filesystems to check if the destination is a directory. We should allow users to specify if the destination is a directory or a file, so we can skip this check.

I propose to add read_parquet(..., is_dir: bool | None = None), which will perform that check, the same way it is implemented in HATS:
astronomy-commons/hats#601

This would help to resolve this HATS issue:
astronomy-commons/hats#578

Before submitting
Please check the following:

  • I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
  • I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
  • If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions