Add per-asset dataStandard, HED standard, and extensions to StandardsType by yarikoptic · Pull Request #371 · dandi/dandi-schema

yarikoptic · 2026-02-24T22:11:01Z

Adds per-asset dataStandard (BareAsset.dataStandard: List[StandardsType]), version and extensions (e.g. NWB ndx-*, HED library schemas) on StandardsType, and a hed_standard constant, so dandi-cli can declare which standards (and versions/extensions) each asset uses and the server can aggregate them into assetsSummary. Schema version is bumped to 0.9.0 (0.8.0 is already published).

aggregate_assets_summary() collects per-asset dataStandard, merging entries per standard and version (extensions united); the path/encoding heuristics remain for assets from older clients, and their bare entries are dropped when a versioned entry for the same standard exists, so e.g. NWB is not listed twice (the web UI shows only names).
migrate() downgrades below 0.9.0 by dropping per-asset dataStandard (new LOSSY_DOWNGRADES: it is extracted automatically, nothing user-provided is lost) and stripping version/extensions from assetsSummary.dataStandard. This matters since dandi-cli now downgrades all outgoing metadata, assets included, to the server's schema (production is at 0.7.0).

Companion: dandi/dandi-cli#1811 (tested together end-to-end, see below).

Details: review responses, design notes, testing

Review responses

readOnly on BareAsset.dataStandard: dropped in favor of nskey: dandi, since it is populated by the client via the API (@candleindark, @bendichter; @satra later noted it is asset-, not dandiset-level, metadata).
nskey added to extensions; description no longer says "in this dataset"; forward-ref quotes removed.
Upper-casing *_standard constants: left for a separate PR, as agreed.

Changes since the previous review round (rebased onto current master)

Rebased onto master (consolidated BareAsset/Asset, downgrade framework from Add ability to downgrade schema all the way to 0.6.10 #342, schema 0.8.0).
migrate() docstring updated: it also downgrades Asset metadata.
Bump DANDI_SCHEMA_VERSION 0.8.0 → 0.9.0; add 0.8.0 to ALLOWED_INPUT_SCHEMAS and ALLOWED_TARGET_SCHEMAS.
Downgrade support as above. Without it, dandi-cli talking to a 0.7.0/0.8.0 server would fail to downgrade any NWB/BIDS asset and send 0.9.0 metadata the server cannot validate.
Aggregation merging/dedup as above. Previously every dandiset with new-client uploads would have gotten both a bare and a versioned NWB/BIDS entry.
Replaced "remove heuristics after 2026-12-01" with a TODO to remove once all assets on the server carry dataStandard -- the heuristics also serve already-uploaded assets.
CLAUDE.md: kept master's version and added architecture notes and a schema change checklist (the earlier "comprehensive" version had gone stale after consolidation).
Verified RRID:SCR_014074 is HED's primary RRID.

Design note: dropping vs keeping `dataStandard` on downgrade

Since DandiBaseModel ignores extra fields, keeping dataStandard in metadata relabeled as 0.7.0 would also validate on old servers and preserve it for later aggregation, but it would make records claim a schema version they do not conform to. I went with explicit removal; the metadata can be re-sent (e.g. dandi upload --existing force) once the server is upgraded.

Known follow-up for dandi-archive

The web UI lists assetsSummary.dataStandard entries by name only. A dandiset mixing e.g. NWB 2.7.0 and 2.8.0 files gets one entry per version, which would show as two identical "Neurodata Without Borders (NWB)" rows until the UI also shows version (and possibly extensions).

Testing

tox-equivalent locally: 265 passed, 19 skipped (no network); flake8/black/isort/codespell clean; mypy clean.
End-to-end with dandi-cli#1811 against a local dandi-archive (docker compose from dandi-cli's test fixtures), uploading generated NWB files using ndx-hed/ndx-pose, a .ome.zarr, and bids-examples eeg_ds003645s_hed_library (HEDVersion: ["8.4.0", "sc:score_1.0.0", "test:testlib_1.0.2"]):

	server w/ dandischema 0.12.1 (schema 0.7.0, as production)	server w/ this PR (schema 0.9.0)
asset validation	all `Valid` (3 NWB/Zarr, 32 BIDS)	all `Valid`
asset `schemaVersion`	downgraded by the client to 0.7.0	0.9.0
asset `dataStandard`	dropped by the downgrade	NWB 2.11.0 + `ndx-hed` 0.2.0 (+ `ndx-pose` 0.4.0); OME/NGFF; BIDS 1.11.1 on every BIDS asset; HED 8.4.0 + `score` 1.0.0, `testlib` 1.0.2 on `dataset_description.json`
`assetsSummary.dataStandard`	OME/NGFF, NWB; BIDS (path heuristics, as before)	OME/NGFF, NWB 2.11.0 + {`ndx-hed`, `ndx-pose`}; BIDS 1.11.1, HED 8.4.0 + {`score`, `testlib`} -- one entry per standard

(Images from mirror.gcr.io; the 0.9.0 server is the stock dandiarchive/dandiarchive-api with the dandischema wheel from this branch installed, as tools/api-with-schema.Dockerfile does. The summary is aggregated only from Valid assets when the draft becomes pending, so the test touched the dandiset metadata after validation to get it re-aggregated.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01P3VJkUsu17gjmUEf4ZY1RW

codecov · 2026-02-24T22:58:45Z

Codecov Report

❌ Patch coverage is 52.17391% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.15%. Comparing base (3e8ddd7) to head (f454667).

Files with missing lines	Patch %	Lines
dandischema/metadata.py	0.00%	37 Missing ⚠️
dandischema/models.py	0.00%	5 Missing ⚠️
dandischema/consts.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #371      +/-   ##
==========================================
+ Coverage   47.96%   48.15%   +0.19%     
==========================================
  Files          19       19              
  Lines        2427     2515      +88     
==========================================
+ Hits         1164     1211      +47     
- Misses       1263     1304      +41

Flag	Coverage Δ
unittests	`48.15% <52.17%> (+0.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

candleindark · 2026-03-03T19:07:02Z

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


readOnly should set to true only in the case that the dataStandard field is assigned and managed solely by the server per https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-validation-00#rfc.section.9.4. Is it the case that the dataStandard field is intended to be assigned and managed solely by the server?

I would guess the answer is no -- that it is intended to be extracted from the source files by the DANDI CLI running client-side.

INTERESTING! We also have readOnly on above "TODO" attributes which I guess nobody ever assigned to any asset. Although we are indeed to populate it from client (or on server) but since we have other similar wasGeneratedBy also editable, I see no point in making this readOnly, and I guess it is really our own thing, hence

Suggested change

json_schema_extra={"readOnly": True},

json_schema_extra={"nskey": DANDI_NSKEY},

right?

Good point — this is populated client-side by dandi-cli, not server-managed, so readOnly is incorrect. Replaced with "nskey": DANDI_NSKEY in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

bendichter · 2026-03-03T19:46:42Z

@yarikoptic this looks great! It will be exciting to be able to query the archive for NWB version and extension usage for data going forward. I agree with @candleindark that this probably should not be readOnly. Other than that, looks good. It would be great to also see a draft of dandi-cli that extracts this information. I'd prefer to have a draft of that before merging this, in case we are able to catch any edge cases.

yarikoptic · 2026-03-03T19:50:46Z

It would be great to also see a draft of dandi-cli that extracts this information

It is there waiting for you

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi-cli#1811

candleindark

Left some minor suggestions for change

candleindark · 2026-03-03T19:13:13Z

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


Shouldn't we have an "nskey" key in the json_schema_extra dict? If dataStandard is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

Agreed — added "nskey": DANDI_NSKEY in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

it should still have readOnly as this is a metadata element that should not be editable using the UI or API. it is also filled in on the server side.

IIRC that readOnly only for web ui entry -- we are the ones to provide metadata records via API while uploading assets. So if we forbid that in API, nothing would fill that in AFAIK.

webUI does not ATM care about any asset level metadata entry, so all that readOnly is of no effect and somewhat inconsistent ATM anyways so I would not split hairs over it if someone pushes 'readOnly' there

i'm pretty positive asset summary is filled by the server validation.

sorry didn't realize this is asset level. i was thinking dandiset level. however, there are a set of fields that the server unsets when uploaded.

candleindark · 2026-03-03T19:58:17Z

+        description="Version of the standard used.",
+        json_schema_extra={"nskey": "schema"},
+    )
+    extensions: Optional[List["StandardsType"]] = Field(


Suggested change

extensions: Optional[List["StandardsType"]] = Field(

extensions: Optional[List[StandardsType]] = Field(

With from __future__ import annotations at the beginning of the file, you don't need the type to be wrapped in a string.

Right — removed the quotes since from __future__ import annotations handles it. Fixed in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

candleindark · 2026-03-03T20:07:28Z

+    extensions: Optional[List["StandardsType"]] = Field(
+        None,
+        description="Extensions to the standard used in this dataset "
+        "(e.g. NWB extensions like ndx-*, HED library schemas).",


Shouldn't we set "nskey" key in the json_schema_extra dict? If extensions is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

Agreed — added "nskey": DANDI_NSKEY to extensions in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

candleindark · 2026-03-03T20:22:25Z

    identifier="DOI:10.25504/FAIRsharing.9af712",
 ).model_dump(mode="json", exclude_none=True)

+hed_standard = StandardsType(


Defining hed_standard in lower case is consistent with the established pattern, as in ome_ngff_standard. However, all the standard variables, nwb_standard, bids_standard, ome_ngff_standard, and hed_standard should be specified in upper case laters since they function as constants.

I think that should be done in a separate PR though.

Agreed — keeping lowercase here for consistency with the existing pattern, and the rename to UPPER_CASE can be done in a separate PR as you suggest.

bendichter · 2026-03-03T20:27:59Z

I approve the dandi dli PR. Once we respond to @candleindark 's comments, this is good to go from my end

@candleindark

…ield Address @candleindark's review comments on PR #371: - BareAsset.dataStandard: replace readOnly=True with nskey=DANDI_NSKEY (field is populated client-side by dandi-cli, not server-managed) - StandardsType.extensions: add nskey=DANDI_NSKEY, fix description ("in this dataset" -> removed, since standard applies per-asset), and remove unnecessary string quotes on forward ref (from __future__ import annotations handles it) Co-Authored-By: Claude Code 2.1.104 / Claude Opus 4.6 <noreply@anthropic.com>

yarikoptic · 2026-04-13T18:49:12Z

@@ -880,6 +897,11 @@ class StandardsType(BaseType):
    identifier="DOI:10.25504/FAIRsharing.9af712",
 ).model_dump(mode="json", exclude_none=True)


@kabilar please add additional constructs for anything LINC related -- like those for bigtiff etc, and then here

https://github.com/dandi/dandi-cli/pull/1811/changes#diff-70a4bb759311189af6f4468d78254d863319fc4660ebe437060b6634c5a7b724R395

as to use them in dandi-cli

…s on StandardsType - Bump DANDI_SCHEMA_VERSION to 0.8.0, add 0.7.0 to ALLOWED_INPUT_SCHEMAS - Add `dataStandard: Optional[List[StandardsType]]` to BareAsset for per-asset standard declarations (NWB, BIDS, HED, OME/NGFF) - Add `version: Optional[str]` to StandardsType - Add `extensions: Optional[List["StandardsType"]]` (self-referencing) to StandardsType for NWB ndx-* extensions, HED library schemas, etc. - Add `hed_standard` constant (RRID:SCR_014074) - Collect per-asset dataStandard in aggregate_assets_summary(), with deprecated path/encoding heuristic fallbacks (remove after 2026-12-01) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…klist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@candleindark

…ield Address @candleindark's review comments on PR #371: - BareAsset.dataStandard: replace readOnly=True with nskey=DANDI_NSKEY (field is populated client-side by dandi-cli, not server-managed) - StandardsType.extensions: add nskey=DANDI_NSKEY, fix description ("in this dataset" -> removed, since standard applies per-asset), and remove unnecessary string quotes on forward ref (from __future__ import annotations handles it) Co-Authored-By: Claude Code 2.1.104 / Claude Opus 4.6 <noreply@anthropic.com>

- Bump DANDI_SCHEMA_VERSION to 0.9.0 since 0.8.0 is already released without dataStandard on assets and version/extensions on StandardsType; allow 0.8.0 as an input and downgrade target. - migrate(): when downgrading below 0.9.0, drop per-asset dataStandard (LOSSY_DOWNGRADES: it is extracted automatically, so nothing user-provided is lost) and strip version/extensions from assetsSummary.dataStandard. dandi-cli applies migrate() to outgoing asset metadata for older servers (the production one is at 0.7.0), and refusing to downgrade would leave such assets with a schemaVersion the server cannot validate. - aggregate_assets_summary(): merge entries per standard and version, uniting extensions, and drop the bare entries added by heuristics for older clients' assets when a versioned entry exists. Otherwise every dandiset with new-client uploads would list e.g. NWB twice (the web UI shows only names). - Reword the "remove after 2026-12-01" note: heuristics are needed for as long as there are assets without dataStandard on the server. - Tests marked ai_generated; imports moved to module top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3VJkUsu17gjmUEf4ZY1RW

yarikoptic mentioned this pull request Feb 24, 2026

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi/dandi-cli#1811

Open

yarikoptic commented Mar 3, 2026

View reviewed changes

Comment thread dandischema/metadata.py Outdated

yarikoptic requested review from bendichter, candleindark and satra March 3, 2026 16:31

yarikoptic force-pushed the enh-hed branch from be40b26 to 571978e Compare March 3, 2026 16:33

candleindark reviewed Mar 3, 2026

View reviewed changes

yarikoptic force-pushed the enh-hed branch from 571978e to 721bd11 Compare April 13, 2026 18:31

yarikoptic commented Apr 13, 2026

View reviewed changes

yarikoptic mentioned this pull request Oct 13, 2026

deno: Report/record bids validator + hed validator in the bids-validator output bids-standard/bids-validator#10

Open

yarikoptic and others added 4 commits October 13, 2026 20:30

docs: extend CLAUDE.md with architecture notes and schema change chec…

68be356

…klist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yarikoptic force-pushed the enh-hed branch from 702f464 to f454667 Compare October 13, 2026 21:35

yarikoptic added minor Increment the minor version when merged release Create a release when this pr is merged labels Oct 13, 2026 — with Claude

	json_schema_extra={"readOnly": True},
	json_schema_extra={"nskey": DANDI_NSKEY},

	extensions: Optional[List["StandardsType"]] = Field(
	extensions: Optional[List[StandardsType]] = Field(

		@@ -880,6 +897,11 @@ class StandardsType(BaseType):
		identifier="DOI:10.25504/FAIRsharing.9af712",
		).model_dump(mode="json", exclude_none=True)

Conversation

yarikoptic commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review responses

Changes since the previous review round (rebased onto current master)

Design note: dropping vs keeping dataStandard on downgrade

Known follow-up for dandi-archive

Testing

Uh oh!

codecov Bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

yarikoptic commented Mar 3, 2026

Uh oh!

candleindark left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yarikoptic commented Feb 24, 2026 •

edited

Loading

Design note: dropping vs keeping `dataStandard` on downgrade

codecov Bot commented Feb 24, 2026 •

edited

Loading