You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5b292aa
Browse filesBrowse the repository at this point in the historyBrowse files
Add --missing-file-content=stream to validate DataLad Dandisets by streaming annexed content
`dandi validate` can now validate a DataLad/git-annex Dandiset (such as the
clones at https://github.com/dandisets) whose content has not been fetched:
with `--missing-file-content=stream`, the content of each annexed file is
streamed on demand (via fsspec) from the URLs registered for it in git-annex,
preferring direct S3 URLs over DANDI API download URLs, so that pynwb and
nwbinspector run without downloading the (possibly terabytes of) data.
- New `dandi.support.annex` module: parses git-annex keys from the broken
symlinks, reads URL logs from the `git-annex` branch (local or
remote-tracking) using only `git`, and provides an `AnnexReadableFile`
`Readable` that streams from the first URL that can be opened.
- `LocalFileAsset.content_source` lets an asset read its content from a
`Readable` instead of `filepath`; `pynwb_utils.validate()` accepts a
`readable`, and `NWBAsset` runs nwbinspector on the streamed file via
`inspect_nwbfile_object()`.
- Each streamed file yields an INFO `DANDI.FILE_CONTENT_STREAMED` result
naming the URL; a file that cannot be streamed yields
`DANDI.FILE_CONTENT_MISSING`. BIDS content-dependent errors are
suppressed for annexed files under `stream` as under `only-non-data`.
- `dandi validate` now accepts a broken symlink directly as a path argument
(`click.Path(exists=True)` used to reject it).
- Docs: describe validating DataLad Dandisets, including every Dandiset of
the `dandisets` superdataset, and the datalad-fuse alternative.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
0 commit comments