Skip to content

to_zarr() does not maintain time encoding when appending to an existing store #10639

@lrntct

Description

@lrntct

What happened?

When writing a dataset to a zarr store, the time encoding is correctly stored and retrieved.
However, when appending to an existing store, it is not possible to set the encoding, and the current encoding is not applied to the appended ata

What did you expect to happen?

I expected the encoding to be consistent between the first write and the subsequent ones.

Minimal Complete Verifiable Example

import tempfile
from datetime import datetime, timedelta
import numpy as np
import xarray as xr
import zarr

with tempfile.TemporaryDirectory() as temp_dir:
    storage = temp_dir
    # Test parameters
    base_time = datetime(year=1, month=1, day=1)
    time_dtype = "datetime64[ms]"
    time_unit = "milliseconds since 1970-01-01T00:00:00"

    time_encoding = {
        "units": time_unit,
        "dtype": time_dtype,
    }

    print(f"Base time: {base_time}")
    print(f"Time encoding: {time_encoding}")

    # Write first timestep
    print("\n--- Writing first timestep ---")
    sim_time1 = base_time + timedelta(minutes=1)
    time_coord1 = np.array([np.datetime64(sim_time1, "ms")], dtype=time_dtype)

    data1 = xr.DataArray(
        data=np.array([[[1.0, 2.0], [3.0, 4.0]]]),
        coords={"time": time_coord1, "y": [0, 1], "x": [0, 1]},
        dims=["time", "y", "x"],
        name="test_var"
    )

    ds1 = xr.Dataset({"test_var": data1})

    ds1.to_zarr(
        storage,
        encoding={"time": time_encoding},  # Encoding provided here
        mode="w",
    )

    print(f"Written: {sim_time1}")

    # Write second timestep using append_dim (this is where the bug occurs)
    print("\n--- Writing second timestep with append_dim ---")
    sim_time2 = base_time + timedelta(minutes=2)
    time_coord2 = np.array([np.datetime64(sim_time2, "ms")], dtype=time_dtype)

    data2 = xr.DataArray(
        data=np.array([[[5.0, 6.0], [7.0, 8.0]]]),
        coords={"time": time_coord2, "y": [0, 1], "x": [0, 1]},
        dims=["time", "y", "x"],
        name="test_var"
    )

    ds2 = xr.Dataset({"test_var": data2})

    ds2.to_zarr(
        storage,
        append_dim="time",
        mode="a",
        # NOTE: Cannot pass encoding={"time": time_encoding} here!
        # xarray raises: "variable 'time' already exists, but encoding was provided"
    )

    print(f"Written: {sim_time2}")

    # Read back and demonstrate the bug
    print("\n--- Reading back data ---")
    ds_read = xr.open_zarr(storage)

    print(f"Time coordinate values: {ds_read['time'].values}")
    print(f"Time dtype: {ds_read['time'].dtype}")

    # Expected vs actual
    expected_times = [
        np.datetime64(sim_time1, "ms"),
        np.datetime64(sim_time2, "ms")
    ]
    actual_times = ds_read['time'].values

    print(f"\nExpected: {expected_times}")
    print(f"Actual:   {actual_times}")

    # Check if bug is present
    assert np.array_equal(expected_times, actual_times)

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Base time: 0001-01-01 00:00:00
Time encoding: {'units': 'milliseconds since 1970-01-01T00:00:00', 'dtype': 'datetime64[ms]'}

--- Writing first timestep ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Written: 0001-01-01 00:01:00

--- Writing second timestep with append_dim ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Written: 0001-01-01 00:02:00

--- Reading back data ---
Time coordinate values: ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Time dtype: datetime64[ms]

Expected: [np.datetime64('0001-01-01T00:01:00.000'), np.datetime64('0001-01-01T00:02:00.000')]
Actual:   ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Traceback (most recent call last):
  File "/home/laurent/software/itzi/xarray_zarr_time.py", line 86, in <module>
    assert np.array_equal(expected_times, actual_times)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.14.0-27-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.7.1
pandas: 2.3.1
numpy: 2.3.2
scipy: 1.16.1
netCDF4: 1.7.2
pydap: 3.5.5
h5netcdf: 1.6.4
h5py: 3.14.0
zarr: 3.1.1
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: None
conda: None
pytest: 8.4.1
mypy: None
IPython: None
sphinx: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugneeds triageIssue that has not been reviewed by xarray team member

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions