Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving and re-opening a v2 array does not report (or save) correct memory order #2950

Open
dstansby opened this issue Apr 3, 2025 · 7 comments · May be fixed by #2951
Open

Saving and re-opening a v2 array does not report (or save) correct memory order #2950

dstansby opened this issue Apr 3, 2025 · 7 comments · May be fixed by #2951
Labels
bug Potential issues with the zarr-python library

Comments

@dstansby
Copy link
Contributor

dstansby commented Apr 3, 2025

import zarr
import zarr.storage

store = zarr.storage.MemoryStore()
arr = zarr.create((100,), store=store, path="a", zarr_format=2, order="F")
print(arr.order)
# "F"

arr = zarr.open(store, path="a")
print(arr.order)
# "C"

I would expect the memory order to be persisted and the same when I re-open the array.

@dstansby dstansby added the bug Potential issues with the zarr-python library label Apr 3, 2025
@d-v-b
Copy link
Contributor

d-v-b commented Apr 3, 2025

iirc this is because we made order part of the runtime configuration, due to order not being part of the zarr v3 metadata model. I'm pretty sure a global config is the wrong location for this parameter -- I think we need a per-array namespace for runtime configuration like this.

@dstansby
Copy link
Contributor Author

dstansby commented Apr 3, 2025

It's part of the v2 metadata though, so shouldn't this be persisted and read in with the array itself (for zarr format 2)?

@d-v-b
Copy link
Contributor

d-v-b commented Apr 3, 2025

if you inspect the array.metadata.order, I think you should see that it has the expected F value. The issue is that when zarr opens an array, it defers to the global config when setting the order of the in-memory array associated with that zarr array, whether the array is v2 or v3. The problem is our use of this global config.

@dstansby
Copy link
Contributor Author

dstansby commented Apr 3, 2025

if you inspect the array.metadata.order, I think you should see that it has the expected F value.

Nope, arr.metadata.order still gives the wrong order.

@d-v-b
Copy link
Contributor

d-v-b commented Apr 3, 2025

:( that's bad! we definitely need to fix this!

@d-v-b
Copy link
Contributor

d-v-b commented Apr 3, 2025

i can get to this in the next few days unless someone does it first

@dstansby dstansby linked a pull request Apr 3, 2025 that will close this issue
6 tasks
@d-v-b
Copy link
Contributor

d-v-b commented Apr 3, 2025

if you inspect the array.metadata.order, I think you should see that it has the expected F value. The issue is that when zarr opens an array, it defers to the global config when setting the order of the in-memory array associated with that zarr array, whether the array is v2 or v3. The problem is our use of this global config.

this explanation is actually very wrong -- we are already using the array-specific config. the underlying issue, addressed in #2951, is a simple bug in our implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Potential issues with the zarr-python library
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants