Skip to content

Enable dandi validate to operate on Datalad datasets through streaming - #1933

Draft
CodyCBakerPhD wants to merge 27 commits into
claude/memoize-readable-fingerprintfrom
claude/brave-hamilton-ast647
Draft

CodyCBakerPhD wants to merge 27 commits into
claude/memoize-readable-fingerprintfrom
claude/brave-hamilton-ast647

Conversation

@CodyCBakerPhD

@CodyCBakerPhD CodyCBakerPhD commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Towards the goal of having a source of internally trusted Truth regarding validation records on a Dandiset, it is necessary to be able to run the validation tool through streamed file contents.

This requires a bit of reworking here on the CLI.

Current strategy is, for simplicity, fsspec on a typical DataLad dataset with special remotes on S3.

AI Summary

Summary

dandi validate can now validate a DataLad / git-annex Dandiset (such as the clones at https://github.com/dandisets) whose content has not been fetched. With the new --missing-file-content=stream policy, the content of each annexed file is streamed on demand (via fsspec) from the URLs registered for it in git-annex, so pynwb and nwbinspector run on the file without downloading the (possibly terabytes of) data. Only git is needed to read the git-annex metadata (neither git-annex nor DataLad has to be installed), plus fsspec[http] (pip install "dandi[extras]") for the streaming.

The motivation is generating validation records for every Dandiset:

$ git clone https://github.com/dandisets/000029
$ dandi validate --missing-file-content=stream --min-severity=INFO -f json_lines -o 000029.jsonl 000029

The docs (docs/source/cmdline/validate.rst) gain a "Validating DataLad Dandisets" section, including how to loop over the subdatasets of the dandisets superdataset, and mention the datalad-fuse alternative (which needs FUSE and so does not work in many containers/CI environments; this PR works anywhere git and HTTPS are available).

What changed

  • dandi/support/annex.py (new): parses git-annex keys from the broken symlinks (SHA256E-s<size>--<hash>.nwb), locates a key's URL log in the git-annex branch (local or remote-tracking, e.g. origin/git-annex after a plain git clone) via git cat-file, and provides AnnexReadableFile, a Readable that streams from the first URL that can be opened (direct S3 URLs are preferred over api.dandiarchive.org/.../download/ URLs, which redirect on every request). Uses fsspec's blockcache so h5py's random access only fetches the blocks it touches.
  • LocalFileAsset.content_source: an optional Readable an asset reads its content from instead of filepath. pynwb_utils.validate() takes a readable= (results not cached in that case), and NWBAsset runs nwbinspector on the streamed file through inspect_nwbfile_object(), replicating inspect_nwbfile()'s error reporting so the result IDs are identical to local validation.
  • validate() core: under stream, each streamable file yields an INFO DANDI.FILE_CONTENT_STREAMED result naming the URL; a broken symlink that cannot be streamed (not annexed / no URL registered) yields DANDI.FILE_CONTENT_MISSING. BIDS content-dependent errors are suppressed for annexed files under stream as they are under only-non-data. A clear error is raised up front if fsspec/aiohttp are missing.
  • CLI: stream added to --missing-file-content; dandi validate now also accepts a broken symlink directly as a path argument (click.Path(exists=True) used to reject it with "does not exist").
  • Tests (all marked ai_generated): key/URL-log parsing, AnnexRepo on a repo whose git-annex branch is created with git plumbing (both local and origin/ refs), AnnexReadableFile over file:// URLs and over a local range-capable HTTP server (h5py opens the file), and end-to-end validate()/CLI runs asserting that streamed results equal those of validating the same NWB file locally.

Verification

  • Real data: dandi validate --missing-file-content=stream on a git clone of dandisets/000029 (6 NWB files, 18 KB–18 MB) streams everything from S3 in ~10 s and produces 77 records (the same pynwb/nwbinspector findings as for local files); the same for 000029 checked out as a git submodule (.git file) and for a single 2.8 MB file of 000035. A 61 GB file of 000003 opens with one HEAD and a single 4 MiB range request.

Limitations / notes

  • Zarr assets are not streamed (in the dandisets repos they are separate uninstalled subdatasets, i.e. empty directories that are not validated at all).
    • TODO [follow-up 1]: this will be implemented in follow-ups that facilitate NWB-Zarr integration across the ecosystem.
    • TODO [follow-up 2]: OME-Zarr validation will also be added in a follow-up.
  • The BIDS validator cannot stream (though it our use cases, ought not need to), so BIDS errors needing file content are suppressed for annexed files.
    • TODO [follow-up 3]: assess the need for BIDS content validation on existing Dandisets to determine if this is needed. NWB-BIDS integration will NOT require this and always runs with nifti headers disabled.
  • Some nwbinspector checks read data arrays (e.g. timestamps), so the amount streamed per file depends on its content.
  • The clone must include the git-annex branch (no --single-branch).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA


Generated by Claude Code

…reaming annexed content

`dandi validate` can now validate a DataLad/git-annex Dandiset (such as the
clones at https://github.com/dandisets) whose content has not been fetched:
with `--missing-file-content=stream`, the content of each annexed file is
streamed on demand (via fsspec) from the URLs registered for it in git-annex,
preferring direct S3 URLs over DANDI API download URLs, so that pynwb and
nwbinspector run without downloading the (possibly terabytes of) data.

- New `dandi.support.annex` module: parses git-annex keys from the broken
  symlinks, reads URL logs from the `git-annex` branch (local or
  remote-tracking) using only `git`, and provides an `AnnexReadableFile`
  `Readable` that streams from the first URL that can be opened.
- `LocalFileAsset.content_source` lets an asset read its content from a
  `Readable` instead of `filepath`; `pynwb_utils.validate()` accepts a
  `readable`, and `NWBAsset` runs nwbinspector on the streamed file via
  `inspect_nwbfile_object()`.
- Each streamed file yields an INFO `DANDI.FILE_CONTENT_STREAMED` result
  naming the URL; a file that cannot be streamed yields
  `DANDI.FILE_CONTENT_MISSING`.  BIDS content-dependent errors are
  suppressed for annexed files under `stream` as under `only-non-data`.
- `dandi validate` now accepts a broken symlink directly as a path argument
  (`click.Path(exists=True)` used to reject it).
- Docs: describe validating DataLad Dandisets, including every Dandiset of
  the `dandisets` superdataset, and the datalad-fuse alternative.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.12500% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.78%. Comparing base (43cf4fc) to head (d21b552).
⚠️ Report is 1 commits behind head on claude/memoize-readable-fingerprint.

Files with missing lines Patch % Lines
dandi/files/bases.py 82.14% 5 Missing ⚠️
dandi/pynwb_utils.py 70.00% 3 Missing ⚠️
dandi/validate/_types.py 0.00% 2 Missing ⚠️
dandi/validate/_core.py 97.22% 1 Missing ⚠️
Additional details and impacted files
@@                           Coverage Diff                           @@
##           claude/memoize-readable-fingerprint    #1933      +/-   ##
=======================================================================
+ Coverage                                78.63%   78.78%   +0.15%     
=======================================================================
  Files                                       94       94              
  Lines                                    14595    14727     +132     
=======================================================================
+ Hits                                     11477    11603     +126     
- Misses                                    3118     3124       +6     
Flag Coverage Δ
unittests 78.78% <93.12%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CodyCBakerPhD CodyCBakerPhD added the minor Increment the minor version when merged label Sep 27, 2026 — with Claude
@CodyCBakerPhD CodyCBakerPhD self-assigned this Sep 27, 2026
@CodyCBakerPhD CodyCBakerPhD added cmd-validate enhancement New feature or request labels Sep 27, 2026
@CodyCBakerPhD CodyCBakerPhD changed the title Add --missing-file-content=stream to validate DataLad Dandisets by streaming annexed content Enable dandi validate to operate on Datalad datasets through streaming Sep 27, 2026
Comment thread docs/source/cmdline/validate.rst Outdated
Comment thread docs/source/cmdline/validate.rst Outdated
Comment thread docs/source/cmdline/validate.rst Outdated
fsspec renamed its LRU block cache type from "block" to "blockcache" in
2023, and the declared minimum (2022.11.0) only knows the old name, so
`AnnexReadableFile.open()` failed with `KeyError: 'blockcache'` in the
lowest-deps CI job.  Pick whichever name the installed fsspec registers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
- Drop the recipe for looping over the whole `dandisets` superdataset.
- Note that streaming Zarr subdatasets is a follow-up for when NWB Zarr
  support has matured across the ecosystem.
- Explain that suppressing content-dependent BIDS checks loses nothing for
  NWB datasets: the sidecar files are in git and the rest is in the names.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Comment thread docs/source/cmdline/validate.rst Outdated
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
@CodyCBakerPhD
CodyCBakerPhD changed the base branch from master to claude/docs-cli-options-sync September 27, 2026 17:45
@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1935 September 27, 2026 17:53
`dandi validate` already handles annexed files whose content has not been
fetched (the broken symlinks of a DataLad dataset) according to
`--missing-file-content`: `error`, `skip`, or `only-non-data`.  Those policies
were applied only when such a file was reached through its directory,
though: given directly on the command line, the file never got past argument
parsing, because `click.Path(exists=True)` follows the link and rejected it
with "does not exist".  Check the path with `lexists()` instead, so that

    dandi validate --missing-file-content=skip sub-01/sub-01.nwb

works the same as validating `sub-01/`, while paths that truly do not exist
are still rejected with the same error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Git merged the synced option reference and this branch's additions without
conflict but with the `--missing-file-content`, `--format`, and `--output`
entries and the link targets appearing twice, which Sphinx rejects.  Keep the
synced entries, adding the `stream` policy to `--missing-file-content`, and
the "Validating DataLad Dandisets" section.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Comment thread docs/source/cmdline/validate.rst Outdated
@CodyCBakerPhD
CodyCBakerPhD removed this pull request from stack #1935 September 27, 2026 18:04
@CodyCBakerPhD
CodyCBakerPhD changed the base branch from claude/docs-cli-options-sync to claude/validate-broken-symlink-path September 27, 2026 18:05
@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1938 September 27, 2026 18:05
…hamilton-ast647

The PR is now stacked on the broken-symlink fix, which carries its own copy of
`ExistingPath` and its test; take that branch's docstring and extended test
and keep this branch's streaming test alongside.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
The declared floor is pynwb >= 3.1.0 (and nwbinspector >= 0.7.0 requires
pynwb >= 3.1 anyway), which makes several version guards unreachable:

- validate(): only the `pynwb.validate(path=...)` branch (pynwb >= 3.0) can
  run; the `paths=[...]` tuple-returning form (2.2 to 2.x) and the
  `io=` fallback for even older releases are gone.
- copy_nwb_file(): `cache_spec` has been accepted by `export()` since
  pynwb 2.8.2, so pass it unconditionally.
- _get_external_images(): `ExternalImage` exists in every supported
  pynwb, so import it at module level instead of guarding an ImportError.

The now-unused `packaging.version.Version` import is removed as well.
The NWB *schema* version checks (e.g. the < 2.1.0 error filter) are about
the file being validated, not about pynwb, and are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
@CodyCBakerPhD
CodyCBakerPhD changed the base branch from claude/validate-broken-symlink-path to claude/pynwb-utils-drop-legacy-pynwb September 27, 2026 19:28
@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1940 September 27, 2026 19:29
…-hamilton-ast647

Resolves the conflict in dandi/pynwb_utils.py: the streaming (`readable`)
branch keeps validating through `pynwb.validate(io=...)`, while the local
path now always uses `pynwb.validate(path=...)`; the pre-3.0 pynwb
branches and the tuple normalization are gone along with the base.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
fscacher's memoize_path derives its cache key from a stat of the file at
the given path, so it never caches a call made on a `Readable` (there is
nothing to stat).  Some readables can nevertheless vouch for their content
better than mtime/inode ever could, e.g. with a content digest such as a
git-annex key: results computed from one such resource are valid for any
other with the same fingerprint.

- `Readable.get_fingerprint()`: new optional hook returning a content
  fingerprint string, `None` (the default) meaning "unknown, do not cache".
- `pynwb_utils.memoize_source(cache, tokens)`: a decorator that behaves
  exactly like `cache.memoize_path` for path arguments and, for a
  `Readable` with a fingerprint, caches under (file name, fingerprint,
  tokens, other arguments) via `cache.memoize` instead.  The tokens are
  passed explicitly since plain `memoize` does not add them by itself.
- Applied to `get_metadata`, `get_neurodata_types`, and
  `nwb_has_external_links`, the memoized functions that already accept a
  `Readable`.  No behavior change for paths or for readables without a
  fingerprint (still none of them at this point).

Since the decorator preserves the signature, mypy now sees through
`get_metadata`, whose return annotation claimed `dict | None` although it
never returns `None`; fixed to `dict[str, Any]`.  isort also reordered a
pair of local imports in `RemoteReadableAsset.open()` while at it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647

Brings in `Readable.get_fingerprint()` and `pynwb_utils.memoize_source`.
The only conflict was the `typing` import line of the test fixtures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
`AnnexReadableFile.get_fingerprint()` returns the file's git-annex key,
which is a digest of the content (plus its size), and `pynwb_utils.validate`
now goes through `memoize_source` for its `readable=` as well, with the
content source as the first argument of the memoized `_validate_cached`.
The metadata side (`get_metadata` & co.) picked the caching up already by
accepting a `Readable`.

So re-running `dandi validate --missing-file-content=stream` on a clone
skips the pynwb validation and the metadata extraction of every file whose
key has not changed, instead of streaming it again.  nwbinspector's checks
are not cached (they are not for local files either).  Documented in the
notes of the "Validating DataLad Dandisets Remotely" section.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
`PersistentCache.memoize(exclude_kwargs=...)` only exists from fscacher 0.4
on; the declared floor is 0.3.0, where the argument is called `ignore`, so
the lowest-deps job failed at import time.  Pick the name from the
signature.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647

Picks up the fscacher < 0.4 compatibility fix for `memoize_source`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
joblib 1.3 (the declared floor) mistakes positional arguments beyond the
fingerprint ones for the keyword-only `_source` parameter of the memoized
function ("Keyword-only parameter '_source' was passed as positional
parameter"), which broke `validate(path, readable=...)` under lowest-deps.
Bind the decorated function's other arguments by name instead (defaults
included), which also makes the cache key independent of how they were
passed; the decorated function must not take *args as a consequence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647

Picks up the joblib 1.3 compatibility fix for `memoize_source`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
@CodyCBakerPhD
CodyCBakerPhD removed this pull request from stack #1940 September 27, 2026 20:17
@CodyCBakerPhD
CodyCBakerPhD changed the base branch from claude/pynwb-utils-drop-legacy-pynwb to claude/memoize-readable-fingerprint September 27, 2026 20:17
@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1942 September 27, 2026 20:17
The adapter only existed because `_validate` kept its pre-caching parameter
order; with the content source as its first parameter, `memoize_source` can
decorate `_validate` directly.  Same arguments and values reach the memoized
function, so cache keys are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…uments

Instead of re-deriving a `readable` from the type of `source`, read whatever
the source is through `open_readable()` and validate via `pynwb.validate(io=)`
for local files too; since pynwb 3.0 that is the same code path as
`validate(path=)`, cached namespaces included (`_get_pynwb_metadata` already
reads local files this way).  Document both arguments on the function.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
ba2f2b3 routed local files through open_readable() + h5py's file-object
driver + pynwb.validate(io=...), the same way streamed content is
validated.  The lowest-deps CI job started failing on that commit while
the same tests pass locally under the same dependency floors, so go back
to pynwb.validate(path=...) for local files and keep the io route for
Readables only.  The explicit (source, path) arguments stay.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
@CodyCBakerPhD

Copy link
Copy Markdown
Contributor Author

Try using datalad-fuse (with fsspec-adapter), NOT via FUSE itself

Test against; datalad/datalad-fuse#131

…hamilton-ast647

The base was rebased onto master (with #1937 and #1939 squash-merged)
and now uses fscacher >= 0.5.0's memoize_path(custom_fingerprint=...).
Resolve accordingly:

- Drop memoize_source and its tests: get_metadata, get_neurodata_types,
  nwb_has_external_links and _validate now use
  memoize_path(custom_fingerprint=readable_fingerprint).
- Drop ExistingPath in favor of LinkAwarePath(lexists=True) from #1937.
- Keep the streaming support (_validate reading from a Readable) and
  its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
CodyCBakerPhD pushed a commit that referenced this pull request Oct 6, 2026
…ally

Moved over from #1933 without its integration into `dandi validate`:
dandi.support.annex reads the key of an annexed file from its (possibly
broken) symlink and the URLs registered for it from the git-annex
branch, using only git, and AnnexReadableFile streams the content from
those URLs with fsspec.  Its get_fingerprint() is the key, so results
of the functions memoized with readable_fingerprint (metadata, ...) are
cached under it and shared by files with the same key and name.

Tests cover key and URL-log parsing, the git-annex branch lookup,
streaming from file:// and HTTP(S) URLs, and caching by key, using
fixtures that fake a DataLad Dandiset with plain git.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
claude added 3 commits October 6, 2026 19:23
…hamilton-ast647

AnnexReadableFile and its tests moved to the base; take its versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
fscacher is untyped, so since _validate is decorated with fscacher's
memoize_path (instead of the typed memoize_source) mypy sees it as
returning Any.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…hamilton-ast647

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cmd-validate enhancement New feature or request minor Increment the minor version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants