Skip to content

Fail zarr format specification requirements if format cannot be inferred#31

Merged
leifdenby merged 3 commits into
mlcast-community:mainfrom
leifdenby:docs/clarify-need-for-storage-options-during-zarr-format-validation
Mar 16, 2026
Merged

Fail zarr format specification requirements if format cannot be inferred#31
leifdenby merged 3 commits into
mlcast-community:mainfrom
leifdenby:docs/clarify-need-for-storage-options-during-zarr-format-validation

Conversation

@leifdenby

@leifdenby leifdenby commented Mar 8, 2026

Copy link
Copy Markdown
Member

Describe your changes

There are a few things about the current zarr format requirement checks that are brittle:

  • If we can't open the underlying zarr store to infer the version we default to 2. I think it would be better for the requirement to fail here. It doesn't make sense to have something as a requirement if we don't verify if the requirement is satisfied or not
  • we want the validation function to be able to take a xr.Dataset to apply validation too, but unfortunately xarray does not set the storage_options arguments anywhere on the xr.Dataset it returns when xr.open_zarr() / xr.open_dataset() is called to open the dataset. These storage_options are needed when we want to access the underlying store. As a hack the cli sets the storage-options on ds.encoding['storage_options']. With this PR we now warn the caller if storage_options isn't defined on ds.encoding if the zarr format requirement checks fail to find out either what the zarr version is or whether there is consolidated metadata.

The motivation here is two-fold: 1) ensure that we are actually checking for the things the spec requires and 2) nudge the user to supply storage_options on the provided xr.Dataset.

No change of dependencies required.

Issue Link

Partially resolves #2

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality, e.g. adding a new dataset)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected, e.g. removing or moving a dataset in the catalog)
  • 📖 Documentation (Addition or improvements to documentation)
  • 🔨 Maintenance (Fixing issues with CI/CD and changes to upstream dependencies)

Checklist before requesting a review

  • My branch is up-to-date with the target branch - if not update your fork with the changes from the target branch (use pull with --rebase option if possible).
  • I have performed a self-review of my code
  • For any new/modified functions/classes I have added docstrings that clearly describe its purpose, expected inputs and returned values
  • I have placed in-line comments to clarify the intent of any hard-to-understand passages of my code
  • I have updated the documentation to cover introduced code changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have given the PR a name that clearly describes the change, written in imperative form (context).
  • I have requested a reviewer and an assignee (assignee is responsible for merging)

Checklist for reviewers

Each PR comes with its own improvements and flaws. The reviewer should check the following:

  • the code is readable
  • the code is well tested
  • the code is documented (including return types and parameters)
  • the code is easy to maintain

Author checklist after completed review

  • I have added a line to the CHANGELOG describing this change, in a section
    reflecting type of change (add section where missing):
    • added: when you have added new functionality
    • changed: when default behaviour of the code has been changed
    • fixes: when your contribution fixes a bug

Checklist for assignee

  • PR is up to date with the base branch
  • the tests pass
  • author has added an entry to the changelog (and designated the change as added, changed or fixed)
  • Once the PR is ready to be merged, squash commits and merge the PR.

add warning to report when validating remote datasets and storage_options are missing in dataset attributes
@leifdenby leifdenby requested a review from franchg March 8, 2026 21:11
@leifdenby leifdenby added this to the v0.3.0 milestone Mar 8, 2026

@franchg franchg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@leifdenby leifdenby merged commit be31c9d into mlcast-community:main Mar 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Required work before first tagged feature complete release

2 participants