NodeTypeValidationError when opening with zarr.open_zarr but xarray.open_zarr suceeds

[rbavery]

I'm trying to investigate/learn how dask chunks get translated to Zarr v3 shards (or chunks?). So I'm working with an example where I save a Dask array to a zarr and then try to open it with zarr-python to inspect the shard size. From this comment it sounds like dask chunks get translated to shards: pydata/xarray#9938 (reply in thread)

import xarray as xr
import dask.array as da
import numpy as np
import zarr

ar = da.random.random((4096, 4096), chunks=(64, 64))
dset = xr.DataArray(ar, coords={"x": range(4096), "y": range(4096)}).to_dataset(
    name="variables"
)
encoding = {"variables": {"shards": (2048, 2048)}}
dset["variables"].encoding = encoding
dset.to_zarr("test.zarr", zarr_format=3, mode="w")
rt = xr.open_zarr("test.zarr") # has a chunk size same as dask chunk size, not the shard size

import zarr
zarr.open_array(store="test.zarr") #fails

The error is

zarr.open_array(store="test.zarr")

---------------------------------------------------------------------------
NodeTypeValidationError                   Traceback (most recent call last)
Cell In[23], line 2
      1 import zarr
----> 2 zarr.open_array(store="test.zarr")

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/api/synchronous.py:1289, in open_array(store, zarr_version, path, storage_options, **kwargs)
   1259 def open_array(
   1260     store: StoreLike | None = None,
   1261     *,
   (...)   1265     **kwargs: Any,
   1266 ) -> Array:
   1267     """Open an array using file-mode-like semantics.
   1268 
   1269     Parameters
   (...)   1286         The opened array.
   1287     """
   1288     return Array(
-> 1289         sync(
   1290             async_api.open_array(
   1291                 store=store,
   1292                 zarr_version=zarr_version,
   1293                 path=path,
   1294                 storage_options=storage_options,
   1295                 **kwargs,
   1296             )
   1297         )
   1298     )

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/sync.py:163, in sync(coro, loop, timeout)
    160 return_result = next(iter(finished)).result()
    162 if isinstance(return_result, BaseException):
--> 163     raise return_result
    164 else:
    165     return return_result

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/sync.py:119, in _runner(coro)
    114 """
    115 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
    116 exception, the exception will be returned.
    117 """
    118 try:
--> 119     return await coro
    120 except Exception as ex:
    121     return ex

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:1261, in open_array(store, zarr_version, zarr_format, path, storage_options, **kwargs)
   1258     _warn_write_empty_chunks_kwarg()
   1260 try:
-> 1261     return await AsyncArray.open(store_path, zarr_format=zarr_format)
   1262 except FileNotFoundError:
   1263     if not store_path.read_only and mode in _CREATE_MODES:

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/array.py:898, in AsyncArray.open(cls, store, zarr_format)
    875 """
    876 Async method to open an existing Zarr array from a given store.
    877 
   (...)    895 <AsyncArray memory://... shape=(100, 100) dtype=int32>
    896 """
    897 store_path = await make_store_path(store)
--> 898 metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
    899 # TODO: remove this cast when we have better type hints
    900 _metadata_dict = cast(ArrayV3MetadataDict, metadata_dict)

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/array.py:210, in get_array_metadata(store_path, zarr_format)
    207     assert zarr_json_bytes is not None
    208     metadata_dict = json.loads(zarr_json_bytes.to_bytes())
--> 210     parse_node_type_array(metadata_dict.get("node_type"))
    212 return metadata_dict

File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/metadata/v3.py:64, in parse_node_type_array(data)
     62 if data == "array":
     63     return "array"
---> 64 raise NodeTypeValidationError("node_type", "array", data)

NodeTypeValidationError: Invalid value for 'node_type'. Expected 'array'. Got 'group'.

any pointers on how to open this test.zarr with zarr-python? am I missing something about how I should save or open the zarr store?

[rbavery]

I was using open_array instead of open woopsidaisy

[ljstrnadiii]

Looks like sharding is not used here and must be declared another way.

import zarr
x = zarr.open_group("test.zarr")
np.array(x['variables'])
x['variables'].shards -> None