document dtype extension #3157
document dtype extension #3157
Conversation
github-actions
bot
added
the
needs release notes
There was a problem hiding this comment.
Nice, this is a great improvment. I left some comments and suggested improvements. The other thing I'd wish for is to the format the lines to to 80 characters long. In general I like 100 line length, but when rendered on the docs page you as it currently stands you have to horizontally scroll to read the example code.
examples/custom_dtype.py
Outdated
|
|
||
| class Int2(ZDType[int2_dtype_cls, int2_scalar_cls]): | ||
| """ | ||
| This class provides a Zarr compatibility layer around the int2 data type and the int2 |
There was a problem hiding this comment.
Is there a nice link explaining the difference between these? I think I've inferred it but would be nice to make it explicit.
There was a problem hiding this comment.
no I don't actually think there is a nice link that explains the data type / scalar type difference. The numpy docs should explain this, but they don't. I can add something to our docs.
examples/custom_dtype.py
Outdated
|
|
||
| def to_json_scalar(self, data: object, *, zarr_format: ZarrFormat) -> int: | ||
| """Convert a python object to a scalar.""" | ||
| return int(data) |
There was a problem hiding this comment.
Can this be more specific to the example? e.g. explain something to the effect of "needs to be int to be compatible with json." and mention int2 somewhere.
| Zarr Python defines a collection of Zarr data types. This collection, called a "data type registry", | ||
| is essentially a dict where the keys are strings (a canonical name for each data type), and the values are | ||
| the data type classes themselves. Dynamic data type resolution entails iterating over these data | ||
| type classes, invoking a special class constructor defined on each one, and returning a concrete |
There was a problem hiding this comment.
Maybe mention the name of method, or link to on ZDType.
github-actions
bot
removed
the
needs release notes
|
|
||
| Data types in Zarr version 2 | ||
| ----------------------------- | ||
|
|
||
| Version 2 of the Zarr format defined its data types relative to | ||
| `NumPy's data types <https://numpy.org/doc/2.1/reference/arrays.dtypes.html#data-type-objects-dtype>`_, | ||
| and added a few non-NumPy data types as well. Thus the JSON identifier for a NumPy-compatible data | ||
| and added a few non-NumPy data types as well. With one exception, the Zarr V2 JSON identifier for a data |
There was a problem hiding this comment.
| and added a few non-NumPy data types as well. With one exception, the Zarr V2 JSON identifier for a data | |
| and added a few non-NumPy data types as well. With one exception (`<the exception>`), the Zarr V2 JSON identifier for a data |
otherwise confusing with the mention of two special cases below
This PR adds a working example of custom dtype creation and registration. because it's a lot of code, I put this in a new top-level directory called
examples, which contains the executable python filedtype_example.py. This file uses PEP-723 metadata to declare aml_dtypesdependency, and it uses a local zarr install, which means it can be tested properly against local changes.I also expanded the current dtype docs in the user guide to include content about the data type resolution process.
TODO:
docs/user-guide/*.rstchanges/