Repository navigation
Do not recurse when getting the metadata of an NWB file in a BIDS dataset - #1958
Conversation
…aset For an NWB file in a BIDS dataset, get_metadata() took the BIDS metadata from NWBBIDSAsset.get_metadata(), which also extracts the NWB metadata by calling nwb2asset() and thus get_metadata() again, and so on until RecursionError: ~100 levels, each running the BIDS validator. The result was still correct, as each level falls back on errors, but it took about a minute per file (and timed out CI on Windows). Take only the BIDS metadata, with BIDSAsset.get_metadata(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1958 +/- ##
==========================================
+ Coverage 90.77% 90.82% +0.05%
==========================================
Files 92 92
Lines 14130 14159 +29
==========================================
+ Hits 12826 12860 +34
+ Misses 1304 1299 -5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@yarikoptic this one was discovered back in the bids-dandisets days but fixing was delayed until closer to 'eventual' BEP032 integration Might as well do it now though, with all the other validation improvements going on |
… asset Getting the metadata of an NWB file in a BIDS dataset recursed because of a cycle between two methods that each merge BIDS and NWB metadata: `NWBBIDSAsset.get_metadata()` extracts the NWB metadata with `dandi.metadata.nwb.get_metadata()`, which itself got the BIDS metadata of the file with `get_metadata()` of its `NWBBIDSAsset`. Calling `BIDSAsset.get_metadata()` unbound only patched this one call site. Instead, give BIDS assets a method returning only their BIDS metadata, which no subclass overrides to read the content, and use it wherever only the BIDS metadata is wanted: when extracting the metadata of an NWB file, and in `NWBBIDSAsset.get_metadata()` and `ZarrBIDSAsset.get_metadata()` (no change in behavior). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
yarikoptic
left a comment
There was a problem hiding this comment.
I traced how this became a recursion and checked the fix locally. LGTM; the notes below are non-blocking.
How the loop formed
- #1076 (Aug 2022):
NWBBIDSAsset.get_metadata()mergesBIDSAsset.get_metadata()withNWBAsset.get_metadata()→nwb2asset()→metadata.get_metadata(). That function knew nothing about BIDS yet, so there was no loop. - #1164 (Dec 2022, for
dandi ls) added a BIDS branch tometadata.get_metadata()that calls the overridabledf.get_metadata(). At first it sat in theelse:for non-NWB files, which was safe. 11c18f5 then put it before the NWB branch (if bids: … elif nwb:). For an NWB file in BIDS,dfis anNWBBIDSAsset, which calls back intometadata.get_metadata(): the loop. - #1183 (Jan 2023) changed
eliftoifand addedieeg_epilepsyNWBandtest_bids_nwb_metadata_integration. The test passed becauseNWBAsset.get_metadata()catches theRecursionErrorat the innermost level (except Exceptionwithignore_errors=True), after which every outer level completes normally. - Every level builds a new
BIDSDatasetDescriptionAssetand validates the whole dataset (bidsschematools for the metadata, deno for the version). That is where the ~60 s and the Windows timeouts come from.
On the "band-aid" question
It isn't one. A band-aid would guard against recursion depth; this PR instead removes the overridable call that closed the loop. get_bids_metadata() is defined once on BIDSAsset and never reads file content, so this call can no longer recurse, whatever subclass df is.
There is a more general option, possibly simpler: for NWB files, the BIDS branch in nwb.get_metadata() contributes nothing. Every key it can set (age, genotype, identifier, sex, species, strain) is then overwritten by _get_pynwb_metadata(), with None if the NWB file lacks it. The only key that survives is bids_version, which is always None (#1961). Skipping the branch for NWB files would make metadata.nwb stop depending on files.bids for NWB files, and would also remove two full-dataset validations per NWB file. Filed as #1962; fine as a follow-up.
Verified locally (Python 3.13)
- The PR's new tests against master's code fail with 68 nested
nwb2asset()calls (53 s and 57 s).test_bids_nwb_metadata_integrationstill passes there, but takes 77 s and logsRecursionErrorwarnings frombases.py. - On this branch all 3 pass in about 1 s. The spy does catch the calls, because
bases.pyimportsnwb2assetinside the function. - NWB values still take priority over BIDS values, both in the flat dict and in the
NWBBIDSAssetmerge.
Nits (optional)
- The
get_bids_metadata()docstring mentions "files such as participants.tsv". Here bidsschematools only provides path entities (subject_id,session_id);participants.tsvis not read. get_bids_metadata()returns the object cached in_asset_metadataand modifies it in place (digest,path;ZarrBIDSAssetalso setsencodingFormat). This is pre-existing and harmless today. Now that it is a public method meant to be reused, copying it first (.model_copy(deep=True)) would be safer.- A short comment on why the test uses
get_metadata.__wrapped__would help: it bypasses the cache, so a cached result can't makecall_count == 0pass trivially. The two new tests could also share the copytree/path setup.
Generated by Claude Code
Hah, nice to see the story laid out |
Problem
For an NWB file in a BIDS dataset,
dandi.metadata.nwb.get_metadata()took the BIDS part of the metadata fromdf.get_metadata().dfis anNWBBIDSAsset, and itsget_metadata()also extracts the NWB metadata, viaNWBAsset.get_metadata()→nwb2asset()→get_metadata(). That calls back into the same BIDS branch, so the cycle repeats untilRecursionError.The result was still correct, because
NWBAsset.get_metadata()falls back to default metadata on errors at each level. But the BIDS validator ran at every one of roughly 100 levels. Onieeg_epilepsyNWBfrom bids-examples, oneget_metadata()call made 98 nestednwb2asset()calls and took about 60 s. That is what madetest_bids_nwb_metadata_integrationtime out on Windows CI.Fix
Take only the BIDS metadata, via
bids.BIDSAsset.get_metadata(df, ...), which was the intent of that branch. The NWB metadata is extracted by the rest ofget_metadata()itself.Measured on the same file:
nwb2asset()callssubject_id/sex01/U01/U🤖 Generated with Claude Code
https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr