Set `unit` as optional for Groups extending `TimeSeries`

[luiztauffer]

I’m creating a couple of NWBGroupSpec classes that extend TimeSeries. For those classes, the data has no physical unit, and I’d like to make that attribute not required, or at least set a default value for it. I’ve tried to overwrite the data Dataset, see here, giving it an optional unit attribute, but I keep getting this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 6
----> 6 latent_space_series = LatentSpaceSeries(
      7     name="LatentSpaceSeries",
      8     data=latent_data,
      9     rate=10.,
     10 )

File ~/anaconda3/envs/vame-desktop/lib/python3.12/site-packages/hdmf/utils.py:667, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
--> 667     pargs = _check_args(args, kwargs)
    668     return func(args[0], **pargs)

File ~/anaconda3/envs/vame-desktop/lib/python3.12/site-packages/hdmf/utils.py:660, in docval.<locals>.dec.<locals>._check_args(args, kwargs)
    658     if parse_err:
    659         msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
--> 660         raise ExceptionType(msg)
    662 return parsed['args']

TypeError: CustomClassGenerator.set_init.<locals>.__init__: missing argument 'unit'

Code to reproduce here.

How can solve this?

I can get rid of this error defining an attribute for the Group like this:

latent_space_series = NWBGroupSpec(
        neurodata_type_def="LatentSpaceSeries",
        neurodata_type_inc="TimeSeries",
        ...
        attributes=[
            NWBAttributeSpec(
                name="unit",
                doc="The physical unit of the data.",
                dtype="text",
                required=False,
                default_value="n/a",
            ),
        ],
    )

but, is this correct? According to the TimeSeries schema, unit is an attribute of the dataset, not of the group.

[luiztauffer]

Update

Setting unit as an attribute of the data NWBDatasetSpec does not work, the error described above keeps happening. Nothing changes if I use value or default_value in this case, or if I set required to True or False, the error above keeps happening.

latent_space_series = NWBGroupSpec(
    neurodata_type_def="LatentSpaceSeries",
    neurodata_type_inc="TimeSeries",
    doc="An extension of TimeSeries to include VAME latent space data.",
    quantity="?",
    datasets=[
        NWBDatasetSpec(
            name="data",
            dtype="int32",
            shape=(None, None),
            doc="Latent-space vectors over time.",
            attributes=[
                NWBAttributeSpec(
                    name="unit",
                    doc="No physical unit applies.",
                    dtype="text",
                    required=True,
                    value="n/a",
                ),
            ],
        )
    ],
)

This is the only spec definition that allows me to instantiate LatentSpaceSeries without passing a value for unit:

latent_space_series = NWBGroupSpec(
    neurodata_type_def="LatentSpaceSeries",
    neurodata_type_inc="TimeSeries",
    ...
    attributes=[
        NWBAttributeSpec(
            name="unit",
            doc="No physical unit applies.",
            dtype="text",
            default_value="n/a",
        ),
    ],
)

weirdly, if I change default_value for value in this case, the error returns.

[rly]

To answer your initial question, removing a required field ("unit") is not allowed as that would break inheritance. You can set a fixed or default value, like in

nwb-schema/core/nwb.misc.yaml

Line 70 in a8127bb

value: n/a

or you can make LatentSpaceSeries inherit from NWBDataInterface instead.

I see your update -- that may be a bug in HDMF where the mapper that maps data.unit is not being applied correctly to the auto-generated class. Let me investigate today and get back to you.

[luiztauffer]

@rly I'm using hdmf==3.14.6, if that helps. I'll try upgrading it to 4 and test again

update: same behavior happens with hdmf==4.0.0

[rly]

I can reproduce this issue on my end with the latest hdmf and pynwb. I will create an issue on HDMF to resolve this. In the meantime, you can do the following workaround:

from pynwb import load_namespaces, get_class, TimeSeries
from hdmf.utils import get_docval, docval

load_namespaces("nwb_schema_630/ndx-test.namespace.yaml")  # change me

AutoLatentSpaceSeries = get_class("LatentSpaceSeries", "ndx-test")

ts_docval = list(get_docval(TimeSeries.__init__))

# Remove the "unit" argument from the TimeSeries constructor docval
for arg in ts_docval.copy():
    if arg["name"] == "unit":
        ts_docval.remove(arg)
        break

class LatentSpaceSeries(AutoLatentSpaceSeries):

    @docval(*ts_docval)
    def __init__(self, **kwargs):
        # Set the unit to "n/a" in the constructor, as specified in the spec
        kwargs["unit"] = "n/a"
        super().__init__(**kwargs)

obj = LatentSpaceSeries(
    name="test",
    description="test",
    data=[1, 2, 3],
    timestamps=[1., 2., 3.],
)

from pynwb import NWBHDF5IO
from pynwb.testing.mock.file import mock_NWBFile

nwb = mock_NWBFile()
nwb.add_acquisition(obj)

with NWBHDF5IO("test.nwb", "w") as io:
    io.write(nwb)

with NWBHDF5IO("test.nwb", "r") as io:
    read_nwb = io.read()
    read_obj = read_nwb.get_acquisition("test")
    print(read_obj)
    print(read_obj.unit)

[luiztauffer]

thanks @rly , I'll try that workaround and let you know

[luiztauffer]

I did a small modification of your suggestion, in order to preserve the arguments of LatentSpaceSeries, MotifSeries and CommunitySeries in their specific docvals:

AutoLatentSpaceSeries = get_class("LatentSpaceSeries", "ndx-vame")
AutoMotifSeries = get_class("MotifSeries", "ndx-vame")
AutoCommunitySeries = get_class("CommunitySeries", "ndx-vame")


ls_docval = list(get_docval(AutoLatentSpaceSeries.__init__))
for arg in ls_docval.copy():
    if arg["name"] == "unit":
        ls_docval.remove(arg)
        break
class LatentSpaceSeries(AutoLatentSpaceSeries):
    @docval(*ls_docval)
    def __init__(self, **kwargs):
        # Set the unit to "n/a" in the constructor, as specified in the parent's spec
        kwargs["unit"] = "n/a"
        super().__init__(**kwargs)


ms_docval = list(get_docval(AutoMotifSeries.__init__))
for arg in ms_docval.copy():
    if arg["name"] == "unit":
        ms_docval.remove(arg)
        break
class MotifSeries(AutoMotifSeries):
    @docval(*ms_docval)
    def __init__(self, **kwargs):
        # Set the unit to "n/a" in the constructor, as specified in the parent's spec
        kwargs["unit"] = "n/a"
        super().__init__(**kwargs)


cs_docval = list(get_docval(AutoCommunitySeries.__init__))
for arg in cs_docval.copy():
    if arg["name"] == "unit":
        cs_docval.remove(arg)
        break
class CommunitySeries(AutoCommunitySeries):
    @docval(*cs_docval)
    def __init__(self, **kwargs):
        # Set the unit to "n/a" in the constructor, as specified in the parent's spec
        kwargs["unit"] = "n/a"
        super().__init__(**kwargs)

[rly]

Looks good.

Note that custom API classes like what we are creating here are not shipped with the data. If you share code like

obj = LatentSpaceSeries(
    name="test",
    description="test",
    data=[1, 2, 3],
    timestamps=[1., 2., 3.],
)

Then the Python extension will need to be installed for this code to work. If the extension is loaded from an NWB file without the Python extension loaded first, then pynwb will automatically create classes for each of the extension types using get_class and will not modify the constructor arguments. I think it will be rare that someone tries to create new data objects using a spec loaded from an NWB file and a Python script that assumes the Python extension is installed, but I just want to note this minor gotcha of using custom API classes.

Can we close this issue? I have opened hdmf-dev/hdmf#1278 in HDMF for the underlying issue with get_class.

[luiztauffer]

@rly I understand the issue, but I agree, it should be rare if someone tries this, and there's a simple solution of installing the extension in that case. I managed to open the NWB file without problems in an environment without the extension, so I guess that's the best we can do for now. I'm closing this, thanks for the support!