-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
What happened?
When writing a dataset to a zarr store, the time encoding is correctly stored and retrieved.
However, when appending to an existing store, it is not possible to set the encoding, and the current encoding is not applied to the appended ata
What did you expect to happen?
I expected the encoding to be consistent between the first write and the subsequent ones.
Minimal Complete Verifiable Example
import tempfile
from datetime import datetime, timedelta
import numpy as np
import xarray as xr
import zarr
with tempfile.TemporaryDirectory() as temp_dir:
storage = temp_dir
# Test parameters
base_time = datetime(year=1, month=1, day=1)
time_dtype = "datetime64[ms]"
time_unit = "milliseconds since 1970-01-01T00:00:00"
time_encoding = {
"units": time_unit,
"dtype": time_dtype,
}
print(f"Base time: {base_time}")
print(f"Time encoding: {time_encoding}")
# Write first timestep
print("\n--- Writing first timestep ---")
sim_time1 = base_time + timedelta(minutes=1)
time_coord1 = np.array([np.datetime64(sim_time1, "ms")], dtype=time_dtype)
data1 = xr.DataArray(
data=np.array([[[1.0, 2.0], [3.0, 4.0]]]),
coords={"time": time_coord1, "y": [0, 1], "x": [0, 1]},
dims=["time", "y", "x"],
name="test_var"
)
ds1 = xr.Dataset({"test_var": data1})
ds1.to_zarr(
storage,
encoding={"time": time_encoding}, # Encoding provided here
mode="w",
)
print(f"Written: {sim_time1}")
# Write second timestep using append_dim (this is where the bug occurs)
print("\n--- Writing second timestep with append_dim ---")
sim_time2 = base_time + timedelta(minutes=2)
time_coord2 = np.array([np.datetime64(sim_time2, "ms")], dtype=time_dtype)
data2 = xr.DataArray(
data=np.array([[[5.0, 6.0], [7.0, 8.0]]]),
coords={"time": time_coord2, "y": [0, 1], "x": [0, 1]},
dims=["time", "y", "x"],
name="test_var"
)
ds2 = xr.Dataset({"test_var": data2})
ds2.to_zarr(
storage,
append_dim="time",
mode="a",
# NOTE: Cannot pass encoding={"time": time_encoding} here!
# xarray raises: "variable 'time' already exists, but encoding was provided"
)
print(f"Written: {sim_time2}")
# Read back and demonstrate the bug
print("\n--- Reading back data ---")
ds_read = xr.open_zarr(storage)
print(f"Time coordinate values: {ds_read['time'].values}")
print(f"Time dtype: {ds_read['time'].dtype}")
# Expected vs actual
expected_times = [
np.datetime64(sim_time1, "ms"),
np.datetime64(sim_time2, "ms")
]
actual_times = ds_read['time'].values
print(f"\nExpected: {expected_times}")
print(f"Actual: {actual_times}")
# Check if bug is present
assert np.array_equal(expected_times, actual_times)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
Base time: 0001-01-01 00:00:00
Time encoding: {'units': 'milliseconds since 1970-01-01T00:00:00', 'dtype': 'datetime64[ms]'}
--- Writing first timestep ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
warnings.warn(
Written: 0001-01-01 00:01:00
--- Writing second timestep with append_dim ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
warnings.warn(
Written: 0001-01-01 00:02:00
--- Reading back data ---
Time coordinate values: ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Time dtype: datetime64[ms]
Expected: [np.datetime64('0001-01-01T00:01:00.000'), np.datetime64('0001-01-01T00:02:00.000')]
Actual: ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Traceback (most recent call last):
File "/home/laurent/software/itzi/xarray_zarr_time.py", line 86, in <module>
assert np.array_equal(expected_times, actual_times)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
commit: None
python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.14.0-27-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development
xarray: 2025.7.1
pandas: 2.3.1
numpy: 2.3.2
scipy: 1.16.1
netCDF4: 1.7.2
pydap: 3.5.5
h5netcdf: 1.6.4
h5py: 3.14.0
zarr: 3.1.1
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: None
conda: None
pytest: 8.4.1
mypy: None
IPython: None
sphinx: None