Skip to content

Do not recurse when getting the metadata of an NWB file in a BIDS dataset - #1958

Merged
CodyCBakerPhD merged 2 commits into
masterfrom
claude/fix-bids-nwb-metadata-recursion
Oct 13, 2026
Merged

CodyCBakerPhD merged 2 commits into
masterfrom
claude/fix-bids-nwb-metadata-recursion

Conversation

@CodyCBakerPhD

@CodyCBakerPhD CodyCBakerPhD commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

For an NWB file in a BIDS dataset, dandi.metadata.nwb.get_metadata() took the BIDS part of the metadata from df.get_metadata(). df is an NWBBIDSAsset, and its get_metadata() also extracts the NWB metadata, via NWBAsset.get_metadata() → nwb2asset() → get_metadata(). That calls back into the same BIDS branch, so the cycle repeats until RecursionError.

The result was still correct, because NWBAsset.get_metadata() falls back to default metadata on errors at each level. But the BIDS validator ran at every one of roughly 100 levels. On ieeg_epilepsyNWB from bids-examples, one get_metadata() call made 98 nested nwb2asset() calls and took about 60 s. That is what made test_bids_nwb_metadata_integration time out on Windows CI.

Fix

Take only the BIDS metadata, via bids.BIDSAsset.get_metadata(df, ...), which was the intent of that branch. The NWB metadata is extracted by the rest of get_metadata() itself.

Measured on the same file:

nwb2asset() calls time subject_id / sex
before 98 (nested) 60.9 s 01 / U
after 0 0.7 s 01 / U

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

…aset

For an NWB file in a BIDS dataset, get_metadata() took the BIDS metadata
from NWBBIDSAsset.get_metadata(), which also extracts the NWB metadata by
calling nwb2asset() and thus get_metadata() again, and so on until
RecursionError: ~100 levels, each running the BIDS validator.  The result
was still correct, as each level falls back on errors, but it took about
a minute per file (and timed out CI on Windows).  Take only the BIDS
metadata, with BIDSAsset.get_metadata().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.82%. Comparing base (ad8f64d) to head (3ee7aba).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1958      +/-   ##
==========================================
+ Coverage   90.77%   90.82%   +0.05%     
==========================================
  Files          92       92              
  Lines       14130    14159      +29     
==========================================
+ Hits        12826    12860      +34     
+ Misses       1304     1299       -5     
Flag Coverage Δ
unittests 90.82% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CodyCBakerPhD CodyCBakerPhD self-assigned this Oct 12, 2026
@CodyCBakerPhD CodyCBakerPhD added cmd-validate bug Something isn't working patch Increment the patch version when merged labels Oct 12, 2026
@CodyCBakerPhD
CodyCBakerPhD marked this pull request as ready for review October 12, 2026 04:55
@CodyCBakerPhD

Copy link
Copy Markdown
Contributor Author

@yarikoptic this one was discovered back in the bids-dandisets days but fixing was delayed until closer to 'eventual' BEP032 integration

Might as well do it now though, with all the other validation improvements going on

Comment thread dandi/metadata/nwb.py Resolved
… asset

Getting the metadata of an NWB file in a BIDS dataset recursed because
of a cycle between two methods that each merge BIDS and NWB metadata:
`NWBBIDSAsset.get_metadata()` extracts the NWB metadata with
`dandi.metadata.nwb.get_metadata()`, which itself got the BIDS metadata
of the file with `get_metadata()` of its `NWBBIDSAsset`.  Calling
`BIDSAsset.get_metadata()` unbound only patched this one call site.

Instead, give BIDS assets a method returning only their BIDS metadata,
which no subclass overrides to read the content, and use it wherever
only the BIDS metadata is wanted: when extracting the metadata of an
NWB file, and in `NWBBIDSAsset.get_metadata()` and
`ZarrBIDSAsset.get_metadata()` (no change in behavior).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

@yarikoptic yarikoptic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I traced how this became a recursion and checked the fix locally. LGTM; the notes below are non-blocking.

How the loop formed

  • #1076 (Aug 2022): NWBBIDSAsset.get_metadata() merges BIDSAsset.get_metadata() with NWBAsset.get_metadata() → nwb2asset() → metadata.get_metadata(). That function knew nothing about BIDS yet, so there was no loop.
  • #1164 (Dec 2022, for dandi ls) added a BIDS branch to metadata.get_metadata() that calls the overridable df.get_metadata(). At first it sat in the else: for non-NWB files, which was safe. 11c18f5 then put it before the NWB branch (if bids: … elif nwb:). For an NWB file in BIDS, df is an NWBBIDSAsset, which calls back into metadata.get_metadata(): the loop.
  • #1183 (Jan 2023) changed elif to if and added ieeg_epilepsyNWB and test_bids_nwb_metadata_integration. The test passed because NWBAsset.get_metadata() catches the RecursionError at the innermost level (except Exception with ignore_errors=True), after which every outer level completes normally.
  • Every level builds a new BIDSDatasetDescriptionAsset and validates the whole dataset (bidsschematools for the metadata, deno for the version). That is where the ~60 s and the Windows timeouts come from.

On the "band-aid" question

It isn't one. A band-aid would guard against recursion depth; this PR instead removes the overridable call that closed the loop. get_bids_metadata() is defined once on BIDSAsset and never reads file content, so this call can no longer recurse, whatever subclass df is.

There is a more general option, possibly simpler: for NWB files, the BIDS branch in nwb.get_metadata() contributes nothing. Every key it can set (age, genotype, identifier, sex, species, strain) is then overwritten by _get_pynwb_metadata(), with None if the NWB file lacks it. The only key that survives is bids_version, which is always None (#1961). Skipping the branch for NWB files would make metadata.nwb stop depending on files.bids for NWB files, and would also remove two full-dataset validations per NWB file. Filed as #1962; fine as a follow-up.

Verified locally (Python 3.13)

  • The PR's new tests against master's code fail with 68 nested nwb2asset() calls (53 s and 57 s). test_bids_nwb_metadata_integration still passes there, but takes 77 s and logs RecursionError warnings from bases.py.
  • On this branch all 3 pass in about 1 s. The spy does catch the calls, because bases.py imports nwb2asset inside the function.
  • NWB values still take priority over BIDS values, both in the flat dict and in the NWBBIDSAsset merge.

Nits (optional)

  • The get_bids_metadata() docstring mentions "files such as participants.tsv". Here bidsschematools only provides path entities (subject_id, session_id); participants.tsv is not read.
  • get_bids_metadata() returns the object cached in _asset_metadata and modifies it in place (digest, path; ZarrBIDSAsset also sets encodingFormat). This is pre-existing and harmless today. Now that it is a public method meant to be reused, copying it first (.model_copy(deep=True)) would be safer.
  • A short comment on why the test uses get_metadata.__wrapped__ would help: it bypasses the cache, so a cached result can't make call_count == 0 pass trivially. The two new tests could also share the copytree/path setup.

Generated by Claude Code

@CodyCBakerPhD

Copy link
Copy Markdown
Contributor Author

#1183 (Jan 2023) changed elif to if and added ieeg_epilepsyNWB and test_bids_nwb_metadata_integration. The test passed because NWBAsset.get_metadata() catches the RecursionError at the innermost level (except Exception with ignore_errors=True), after which every outer level completes normally.

Hah, nice to see the story laid out

@CodyCBakerPhD
CodyCBakerPhD merged commit 0d5f96e into master Oct 13, 2026
44 checks passed
@CodyCBakerPhD
CodyCBakerPhD deleted the claude/fix-bids-nwb-metadata-recursion branch October 13, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cmd-validate patch Increment the patch version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants