Skip to content

Commit 0344543

Browse files
committed
Docs: trim the DataLad validation section per review
- Drop the recipe for looping over the whole `dandisets` superdataset. - Note that streaming Zarr subdatasets is a follow-up for when NWB Zarr support has matured across the ecosystem. - Explain that suppressing content-dependent BIDS checks loses nothing for NWB datasets: the sidecar files are in git and the rest is in the names. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
1 parent 10658f8 commit 0344543

1 file changed

Lines changed: 8 additions & 14 deletions

File tree

‎docs/source/cmdline/validate.rst‎

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -87,17 +87,7 @@ a Dandiset::
8787
Each streamed file yields an ``INFO``-level ``DANDI.FILE_CONTENT_STREAMED``
8888
result naming the URL its content was read from; a file that cannot be streamed
8989
(not an annexed file, or no URL registered for it) yields a
90-
``DANDI.FILE_CONTENT_MISSING`` error instead. To do this for every Dandiset,
91-
install the subdatasets of the superdataset without fetching any content and
92-
validate each one (the superdataset itself is not a Dandiset)::
93-
94-
datalad clone https://github.com/dandisets/dandisets
95-
cd dandisets
96-
datalad get -n -r . # or: git submodule update --init
97-
for ds in 0*/; do
98-
dandi validate --missing-file-content=stream --min-severity=INFO \
99-
--format=json_lines --output="records/${ds%/}.jsonl" "$ds"
100-
done
90+
``DANDI.FILE_CONTENT_MISSING`` error instead.
10191

10292
Notes:
10393

@@ -109,11 +99,15 @@ Notes:
10999
of data streamed for a file depends on its content; it is nevertheless
110100
usually a small fraction of the file.
111101
- Zarr assets are stored as separate subdatasets (https://github.com/dandizarrs)
112-
and are not streamed: an uninstalled Zarr subdataset is an empty directory
113-
that is not validated at all.
102+
and are not streamed yet: an uninstalled Zarr subdataset is an empty directory
103+
that is not validated at all. Streaming them is a follow-up for when NWB Zarr
104+
support has matured across the ecosystem.
114105
- The BIDS validator cannot stream content, so BIDS errors that require reading
115106
a file (e.g., unreadable NIfTI headers) are suppressed for annexed files under
116-
the ``stream`` and ``only-non-data`` policies.
107+
the ``stream`` and ``only-non-data`` policies. For NWB datasets, the primary
108+
use case, nothing is lost: everything the BIDS validator needs is either
109+
present (the non-annexed sidecar files, which are kept in git) or encoded in
110+
the file and folder names of the annexed files themselves.
117111

118112
Alternatively, `datalad-fuse`_ can mount a DataLad dataset as a file system
119113
that fetches content transparently on read, in which case plain ``dandi

0 commit comments

Comments
 (0)