Make name required and non-empty for Contributor and BaseType

[yarikoptic-gitmate]

Closes #442. Related: dandi/dandi-archive#2936, dandi/dandi-archive#2942, dandi/dandi-archive#2944, #389, #405.

Problem

Contributor.name (and so Organization.name) and BaseType.name (and so
Anatomy, Disorder, GenericType, SpeciesType, StandardsType, …) were
Optional. This let items with no human-readable label, e.g.
{"schemaKey": "Organization"} or {"schemaKey": "GenericType"}, pass
validation, and the landing page rendered them as blank cards.

These classes were also inconsistent with each other for the LinkML
migration. Person tightened name from optional to required in a subclass,
which LinkML only accepts as a required refinement in slot_usage
(#405, pydantic2linkml#66). With name required on Contributor, Person
only adds its "Last, First" pattern.

Changes

• Contributor.name: str (min_length=1). Organization and Person inherit it. Person.name was already required and keeps its pattern.
• BaseType.name: str (min_length=1, max_length=150), covering every subject-matter / type class.
• min_length=1 rejects "" but not whitespace-only names. The audit found no empty or blank names anywhere, so no pattern was added.
• Resource.name and RelatedParticipant.name stay optional. See the audit below.
• DANDI_SCHEMA_VERSION 0.8.0 → 0.8.1, because 0.8.0 is already released as schema-0.8.0. 0.8.0 is kept as an allowed input and as a migration target; downgrading is lossless.
• Tests: TestContributor.test_name_required (Contributor, Organization, Person) and TestBaseTypeName (every BaseType subclass). Existing Contributor tests now pass a name.
• tools/audit_nameless_items.py: an audit of the public S3 manifests (see below). It validates every Dandiset and asset record with the installed dandischema and, given --baseline (the output of a run with the preceding dandischema), reports which records turn from valid to invalid. Nested items with a missing or blank name, or with nothing but schemaKey, are listed as pointers.

Impact on existing metadata (audit of all public S3 manifests, 2026-09-29)

tools/audit_nameless_items.py was run over every dandisets/*/<version>/dandiset.jsonld
(629 published versions, 963 drafts present on S3) and over the assets.jsonld of 628
published versions (248,614 assets; one file over 100 MB was skipped).

Class	Dandiset level (published)	Asset level
Contributor / Organization in contributor	0	0
Organization elsewhere	1 (000631, wasGeneratedBy[].wasAssociatedWith[])	0
BaseType subclasses	2 (001262 Anatomy, 001341 GenericType)	0
Resource (not changed here)	24	61,271 (30 Dandisets, from NWB related_publications)
Empty-string name (any class)	0	0

Rerun on 2026-10-06 with validation-based verdicts (master vs this PR, --include-draft --assets --max-assets-mb 0):
852,353 asset records were validated (346,906 published, 505,447 in drafts on S3), including the 148 MB
manifest of 000571/0.260515.0242 skipped above. No asset record turns invalid. The only records that
do are the Dandiset records below (published) and the drafts of 000631, 001262, 001341 and 001755.
279 drafts have no manifest on S3 (embargoed or not yet written) and were not covered.

Published versions that validate with 0.8.0 and would not validate with 0.8.1:

Version	Location	Current value	Suggested retroactive name
000631/0.231012.1508	wasGeneratedBy[0].wasAssociatedWith[0] (Project "Targeted Neuromodulation by Nanosecond Pulsed Electric Fields", id NIH 1R21EY034258)	{"schemaKey": "Organization", "roleName": [], "contactPoint": [], "includeInCitation": false}	"National Institutes of Health": the same award number is on the NIH Funder contributor. Removing the empty item is an alternative.
001262/0.241205.0959	about[1]	{"schemaKey": "Anatomy", "identifier": "UBERON:0004727"}	"cochlear nerve" (UBERON label)
001341/0.260414.1316	about[0]	{"schemaKey": "GenericType"}	nothing to derive; remove the empty item

The drafts of 000631, 001262 and 001341 contain the same items. The 001755
draft has about[0] = {"schemaKey": "GenericType", "identifier": "pubmed:39491780"},
which should become a relatedResource or get a name.

Published versions keep their original schemaVersion and are not changed by
this PR. The table is for the archive admins, if they want to patch them.

Considered and not done here: a generic "nothing but schemaKey" rule

The audit script also flags nested records of any class whose other fields
are all absent, empty or at their defaults (empty_record). Across all
manifests this adds only one class beyond those already covered by the name
requirement: {"schemaKey": "ContactPoint"}. It appears in
ethicsApproval[].contactPoint, access[].contactPoint and
Organization.contactPoint[]: 93 published versions in 59 Dandisets, and 79
drafts. New ones keep appearing every year from 2021 to 2026, so tooling
(likely the metadata editor) produces them, not users. No asset-level record
is empty.

A generic invalidating validator was left out for these reasons:

• It would block publishing for about 80 drafts because of objects the users never filled in.
• It can't be expressed in JSON Schema, and would be another model_validator that LinkML can't translate (Handle LinkML migration issue of pydantic2linkml: Unable to translate the logic contained in the after validation function #391).

Better options: drop such records, which loses no information (in the
archive on save, or with a dandischema helper), and fix whatever emits them.
That is filed as dandi/dandi-archive#2944. Our published JSON Schema
contributes: TransitionalGenerateJsonSchema drops null from Optional
fields, so contactPoint: null and {} are both rejected, and
{"schemaKey": "ContactPoint"} is the smallest valid value. Advisory
(non-fatal) reporting of such records is proposed separately as
dandischema.lint.

Follow-ups (not in this PR)

• dandi-archive: bump the dandischema pin. The metadata editor will then mark name as required, and drafts with nameless items will show validation errors and cannot be published until fixed.
• dandi-cli: extract_sex (sex given as an IRI) and extract_species (NCBITaxon IRI whose label lookup fails) build nameless SexType/SpeciesType. Fixed, together with the dandischema < 0.15.0 pin bump, in Give SexType/SpeciesType built from IRIs a name; allow dandischema 0.15.x dandi-cli#1949, which should be merged before this is released.
• dandi-archive: a nameless item in asset metadata makes aggregate_assets_summary fail during publishing and leaves the draft in PUBLISHING (A failure in the publish task leaves the draft stuck in PUBLISHING dandi-archive#2966). The revalidate command can't re-validate drafts/assets already marked VALID after a dandischema upgrade (revalidate management command can't re-validate assets or draft versions already marked VALID dandi-archive#2965, Do re-validate dandisets upon dandischema upgrade dandi-archive#2024).
• LinkML branch: regenerate. Contributor gets name required: true, and Person's slot_usage.name drops the required refinement.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NdN9xPzPo1AzsnEMuU8UQs

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.32%. Comparing base (3e8ddd7) to head (45994e1).

Files with missing lines	Patch %	Lines
dandischema/consts.py	0.00%	2 Missing ⚠️
dandischema/models.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #444      +/-   ##
==========================================
+ Coverage   47.96%   48.32%   +0.36%     
==========================================
  Files          19       19              
  Lines        2427     2444      +17     
==========================================
+ Hits         1164     1181      +17     
  Misses       1263     1263              

Flag	Coverage Δ
unittests	48.32% <83.33%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[candleindark]

The schema change itself looks good to me. The notes below are about downstream code that interacts with it. I also left some inline comments on the audit script.

1. dandi-cli constructs a nameless SexType

extract_sex constructs a SexType with name being None when the NWB subject.sex value is a URL, which is no longer valid with this PR.

Details

No dandi-cli test exercises this branch, which is why test-dandi-cli is green here. dandi-cli also pins dandischema < 0.15.0, so that pin needs a bump if this is released as 0.15.0.

2. Draft assets already marked VALID are not re-validated

After dandi-archive bumps its dandischema pin, draft assets validated under the old schema keep their VALID status. I'd suggest that dandi/dandi-archive#2024 (re-validating drafts upon a dandischema upgrade) cover draft assets too. For this PR: which Dandiset's assets.jsonld was skipped by the audit, and could it be scanned, together with the draft assets?

Details

Asset validation only runs for PENDING assets, so nothing re-validates an asset that is already VALID. The audit found no nameless items in published asset metadata, except that one assets.jsonld over 100 MB was skipped. Draft assets were not scanned at all, so we don't know whether any draft asset currently marked VALID would fail under 0.8.1. For non-embargoed Dandisets, the draft assets.jsonld manifests are on S3, so --include-draft --assets should be able to answer that.

3. aggregate_assets_summary can fail during publishing and leave the draft in PUBLISHING

publish_dandiset_task runs _publish_dandiset, which calls aggregate_assets_summary over all assets of the new version, none of which are re-validated at that point. A single nameless approach, measurementTechnique, species, or dataStandard item among them makes it raise, and the draft is left in PUBLISHING. Given the audit, I don't expect this PR to hit it (pending point 2), but the same applies to any future tightening of models whose instances are copied from asset metadata into AssetsSummary.

Why, and a reproduction

• The assets of the new version are the published assets carried over from earlier versions plus the draft's assets. aggregate_assets_summary copies the items above from their stored metadata and builds a models.AssetsSummary with the current models.
• publish_dandiset sets PUBLISHING (via _lock_dandiset_for_publishing) in its own transaction and queues the task with transaction.on_commit(...), so the task is queued only after the PUBLISHING status is committed. If the task fails, its own transaction is rolled back, but the committed PUBLISHING status is not.
• The task has no error handling, so no error is shown to the user, and retrying returns "already being published" until the draft is modified.

I verified this with a test in dandi-archive (at 909cbe9b), both with its current dandischema==0.12.1 (using an ApproachType name over 150 characters, which fails at the same place) and with this PR's head (using a nameless ApproachType). For an unpublished asset marked VALID as well as for a carried-over published asset, publishing raises a ValidationError for AssetsSummary, no new version is created, the draft stays Publishing, a retry raises DandisetAlreadyPublishingError, and a metadata edit resets the draft to Pending.

Reproduction test (copy into dandiapi/api/tests/)

"""
Reproduce: a failure in `aggregate_assets_summary` during publishing leaves the draft in
`PUBLISHING` (dandi/dandi-schema#444 review).

Copy into `dandi-archive/dandiapi/api/tests/` and run with pytest. The trigger adapts to the
installed dandischema: a nameless `ApproachType` if `name` is required (dandi-schema#444), else a
151-character name, which violates the existing `max_length=150` and fails at the same place
(`models.AssetsSummary(**stats)`).
"""

from __future__ import annotations

from dandischema import models as schema_models
import pydantic
import pytest

from dandiapi.api.models import Asset, Version
from dandiapi.api.services.publish import publish_dandiset
from dandiapi.api.services.publish.exceptions import DandisetAlreadyPublishingError
from dandiapi.api.tests.factories import DraftVersionFactory, UserFactory

if schema_models.ApproachType.model_fields['name'].is_required():
    BAD_APPROACH = {'schemaKey': 'ApproachType'}
else:
    BAD_APPROACH = {'schemaKey': 'ApproachType', 'name': 'x' * 151}


@pytest.mark.django_db
@pytest.mark.parametrize('asset_kind', ['draft_marked_valid', 'published'])
def test_failed_summary_leaves_draft_publishing(
    api_client,
    draft_asset_factory,
    published_asset_factory,
    django_capture_on_commit_callbacks,
    asset_kind,
):
    user = UserFactory.create()
    draft_version: Version = DraftVersionFactory.create(
        status=Version.Status.VALID, dandiset__owners=[user]
    )
    dandiset = draft_version.dandiset

    # An asset that was marked VALID earlier (or was published earlier) and is not re-validated
    if asset_kind == 'published':
        asset: Asset = published_asset_factory()
    else:
        asset = draft_asset_factory(status=Asset.Status.VALID)
    Asset.objects.filter(id=asset.id).update(metadata={**asset.metadata, 'approach': [BAD_APPROACH]})
    draft_version.assets.add(asset)
    assert draft_version.publishable

    # Publish through the service used by the API: lock, then run the task on commit (eager)
    with (
        pytest.raises(pydantic.ValidationError) as exc_info,
        django_capture_on_commit_callbacks(execute=True),
    ):
        publish_dandiset(user=user, dandiset=dandiset)
    assert exc_info.value.title == 'AssetsSummary'
    print(f'\n[{asset_kind}] task raised: {exc_info.value!r}')

    # The publish transaction was rolled back ...
    assert dandiset.versions.count() == 1
    asset.refresh_from_db()
    assert asset.published == (asset_kind == 'published')
    # ... but the draft is left in PUBLISHING
    draft_version.refresh_from_db()
    assert draft_version.status == Version.Status.PUBLISHING
    print(f'[{asset_kind}] draft status after failure: {draft_version.status}')

    # Retrying is refused
    with pytest.raises(DandisetAlreadyPublishingError):
        publish_dandiset(user=user, dandiset=dandiset)

    # Any metadata edit resets the status to PENDING (no PUBLISHING guard)
    api_client.force_authenticate(user=user)
    resp = api_client.put(
        f'/api/dandisets/{dandiset.identifier}/versions/draft/',
        {'metadata': {**draft_version.metadata, 'description': 'edited'}, 'name': 'edited'},
        format='json',
    )
    assert resp.status_code == 200, resp.data
    draft_version.refresh_from_db()
    assert draft_version.status == Version.Status.PENDING
    print(f'[{asset_kind}] draft status after a metadata edit: {draft_version.status}')

To run it with this PR's dandischema: uv run --extra development --group test --with "dandischema @ git+https://github.com/dandi/dandi-schema@5cd2a79b4b902c2482901015debdd28f20578827" pytest dandiapi/api/tests/test_pr444_publish_stuck.py -s

[yarikoptic-gitmate]

Thanks @candleindark. The inline comments are addressed in 45994e1. On the review points:

2. Draft assets and the skipped manifest. I reran the reworked script (verdict now decided by validation, see the inline replies) over every manifest on S3. It ran twice, once with master as the baseline and once with this PR's head, with --include-draft --assets --max-assets-mb 0:

	versions	assets validated	BREAKS
published	634	346,906	3: the Dandiset records of 000631/0.231012.1508, 001262/0.241205.0959, 001341/0.260414.1316
drafts on S3	969	505,447	4: the Dandiset records of 000631, 001262, 001341, 001755

• No asset record, published or draft, turns invalid. All seven breaks are the Dandiset-level items already listed in the description.
• The manifest skipped earlier was 000571/0.260515.0242/assets.jsonld (148 MB, 68,210 assets). It is now included and unchanged; the 000571 draft (140,137 assets) is also unchanged.
• Caveat: 279 drafts have no manifest on S3 (embargoed or not yet written). Those, and any drafts edited after the S3 manifests were written, are not covered.

1. dandi-cli extract_sex. Confirmed. There is a second spot of the same kind: extract_species builds a nameless SpeciesType for an NCBITaxon IRI that is not in species_map when the label lookup fails (ConnectionError) or finds no label. A dandi-cli change is prepared for both:

• In both places the name falls back to the IRI itself, and the two PATO IRIs that extract_sex produces map back to "Male"/"Female".
• As a side fix, the IRI is no longer lowercased (PATO_0000384 currently becomes pato_0000384).
• dandi-cli's metadata tests pass with it against this PR's head. It will be proposed to dandi-cli, together with the dandischema < 0.15.0 pin bump, before this is released.

3. aggregate_assets_summary leaving the draft in PUBLISHING. Agreed. That's a dandi-archive robustness bug which any tightening of models copied into AssetsSummary can trigger, independent of this PR (and per point 2, this PR's data doesn't hit it). It deserves its own dandi-archive issue, together with re-validating draft assets on a dandischema upgrade (dandi/dandi-archive#2024).

---

Generated by Claude Code