Populate per-asset dataStandard for BIDS, HED, NWB, and extensions

[yarikoptic]

Populates the per-asset dataStandard introduced in dandi/dandi-schema#371 so the archive can tell which standards, versions and extensions each asset (and thus dandiset) uses:

• NWB: version from the file, ndx-* extensions (latest cached version of each) from the HDF5 specifications group (get_nwb_extensions()).
• BIDS: BIDSVersion from dataset_description.json, declared on every asset of the BIDS dataset.
• HED: HEDVersion, on dataset_description.json; library schemas ([prefix:]library_version, e.g. sc:score_1.0.0) become extensions.
• OME/NGFF: for .ome.zarr assets.

All of it is gated on _SCHEMA_BAREASSET_HAS_DATASTANDARD, so it is a no-op with released dandischema (< 0.15.0). The last commit (TEMP) installs dandischema from the dandi-schema PR branch and is to be replaced with a version bound once that is released.

Details: review responses, changes, testing

Review responses

• Previous description's TODO about a CI run against the schema PR: covered by the TEMP commit pinning enh-hed (dandi-schema's own CI runs dandi-cli master against its branch).
• @bendichter: "line 542 of bases appears to have similar logic" -- the append-if-missing logic now lives in a single _add_data_standard() in files/bases.py, used by NWB, BIDS and Zarr assets; NWB extraction moved into _get_nwb_data_standard().
• HED stored inside NWB: captured as the ndx-hed extension for now (as seen on 001636); anything more specific waits until the representation settles.
• CodeQL "cyclic import" alerts on files/bids.py: bids.py imported from .bases already on master, so this is not new.

Changes in this round (rebased onto current master)

• Rebased; resolved conflicts in .gitignore, CLAUDE.md (now defers to DEVELOPMENT.md; the import-placement guidance went there), pyproject.toml, tests.
• Expect the feature in dandischema 0.15.0 (guard and TODOs said 0.12.2).
• get_nwb_extensions(): after rebase Version was no longer imported in pynwb_utils, and the broad except Exception silently dropped every extension. Now imports it, catches only InvalidVersion, and falls back to the lexicographically last version instead of dropping the extension.
• HED parsing followed "sc:1.0.0", which is not the spec'd format: entries are [prefix:]version or [prefix:]library_version; the prefix is a dataset-local alias, so extensions are now named by library (score, not sc), and prefix-less partnered libraries (lang_1.1.0) work.
• No crash on non-string BIDSVersion, or on a dataset_description.json that is not a JSON object; NWB files whose version/extensions cannot be read still get a (bare) NWB standard with ignore_errors=True.
• OME/NGFF also declared for non-BIDS .ome.zarr assets (detected by the .ome.zarr ending, so dots in the name are fine).
• BIDS is declared on all assets of a BIDS dataset (it is the field's "standard(s) applicable to this asset"), and NWB files in BIDS datasets keep it: their merged metadata used to let the NWB dataStandard override the BIDS one. dataset_description.json is read once, as UTF-8.
• Unit test that _maybe_downgrade_metadata() drops a populated per-asset dataStandard for 0.7.0/0.8.0 servers (needs the downgrade in Add per-asset dataStandard, HED standard, and extensions to StandardsType dandi-schema#371).
• TEMP note in pyproject.toml: when restoring the version bound after the dandischema release, set the version checked by the _SCHEMA_BAREASSET_HAS_DATASTANDARD guard (assumed 0.15.0) to that release's.

Testing

• Full test suite incl. docker-compose integration tests (dandischema from the PR branch, server with dandischema 0.12.1): 1229 passed, 1 skipped. With released dandischema 0.14.0 the feature is a no-op and its tests skip. mypy clean against both dandischema versions; flake8, black, isort clean.
• End-to-end against a local dandi-archive (this repo's docker-compose fixture), uploading generated NWB files using ndx-hed/ndx-pose, a .ome.zarr, and bids-examples eeg_ds003645s_hed_library:

	server w/ dandischema 0.12.1 (schema 0.7.0, as production)	server w/ dandi/dandi-schema#371 (schema 0.9.0)
asset validation	all Valid (3 NWB/Zarr, 32 BIDS)	all Valid
asset schemaVersion	downgraded by the client to 0.7.0	0.9.0
asset dataStandard	dropped by the downgrade	NWB 2.11.0 + ndx-hed 0.2.0 (+ ndx-pose 0.4.0); OME/NGFF; BIDS 1.11.1 on every BIDS asset; HED 8.4.0 + score 1.0.0, testlib 1.0.2 on dataset_description.json
assetsSummary.dataStandard	OME/NGFF, NWB; BIDS (path heuristics, as before)	OME/NGFF, NWB 2.11.0 + {ndx-hed, ndx-pose}; BIDS 1.11.1, HED 8.4.0 + {score, testlib} -- one entry per standard

(Images from mirror.gcr.io; the 0.9.0 server is the stock dandiarchive/dandiarchive-api with the dandischema wheel from the schema PR branch installed, as dandi-schema's tools/api-with-schema.Dockerfile does. The summary is aggregated only from Valid assets when the draft becomes pending, so the test touched the dandiset metadata after validation to get it re-aggregated.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01P3VJkUsu17gjmUEf4ZY1RW

[codecov]

Codecov Report

❌ Patch coverage is 97.74775% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.96%. Comparing base (ad8f64d) to head (d05dcca).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/files/bids.py	94.33%	3 Missing ⚠️
dandi/files/bases.py	92.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1811      +/-   ##
==========================================
+ Coverage   90.78%   90.96%   +0.18%     
==========================================
  Files          92       92              
  Lines       14130    14382     +252     
==========================================
+ Hits        12828    13083     +255     
+ Misses       1302     1299       -3     

Flag	Coverage Δ
unittests	90.96% <97.74%> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bendichter]

line 542 of bases appears to have similar logic

[bendichter]

@yarikoptic lgtm! Let me just check on some of my own data

[bendichter]

Added NWB spec version extraction in 279c297.

get_nwb_version() already existed in pynwb_utils.py but wasn't being used when constructing the StandardsType. Now the version is populated from the HDF5 nwb_version attribute.

Tested against dandiset 001636 (346 NWB files, macaque electrophysiology):

Before:

Neurodata Without Borders (NWB) (version=None)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

After:

Neurodata Without Borders (NWB) (version=2.9.0)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

[bendichter]

ok, I made a minor correction. LGTM now!

Best general approach: break the cycle by removing one side of the top-level import relationship. In this file, that means avoiding eager import from .bases at module import time, and instead doing lazy/local imports only where needed. This preserves behavior while preventing the cycle during initialization.

Single best fix in this snippet:

• In dandi/files/bids.py, keep only type-only imports of GenericAsset, LocalFileAsset, and NWBAsset under TYPE_CHECKING.
• Replace runtime inheritance/import dependency on LocalFileAsset by using postponed annotations (already enabled) plus a runtime alias guarded by local import logic, or (more simply in this file) keep class bases as-is but defer the import block from .bases into function/class scopes where _add_data_standard and _SCHEMA_BAREASSET_HAS_DATASTANDARD are used.
• Because only shown lines are available, the concrete edit we can safely apply is to remove eager runtime import of .bases symbols and move them to type-checking context at top level; then add local imports at usage points in shown regions if any. In the provided snippet, usage points for helper constants/functions are outside shown region, so we can only prepare safe type-only separation here.

To break the cycle without changing functionality, the best fix is to defer the import from .bases until runtime at the point of use, instead of importing it at module load time.

In this file (dandi/files/zarr.py), the problematic top-level line is:

• from .bases import LocalDirectoryAsset, _add_data_standard (line 58 in the snippet)

A robust cycle-breaking pattern is:

1. Remove LocalDirectoryAsset and _add_data_standard from top-level imports.
2. Keep static typing intact by importing LocalDirectoryAsset only under TYPE_CHECKING.
3. Add small helper functions that lazily import these symbols from .bases.
4. Replace direct use sites:
   • use a string annotation ("LocalDirectoryAsset") or TYPE_CHECKING import for type hints
   • call helper(s) where _add_data_standard is needed.

This preserves behavior while avoiding import-time cycles.

Remove the unused symbol from the import statement in dandi/files/zarr.py without changing runtime behavior.

Best fix:

• In dandi/files/zarr.py, edit the import on line 58.
• Change from .bases import LocalDirectoryAsset, _add_data_standard to from .bases import _add_data_standard.
• No other code changes, methods, or new imports are needed.

This preserves existing functionality and keeps only the actually used import.