Repository navigation
Enable dandi validate to operate on Datalad datasets through streaming - #1933
Draft
CodyCBakerPhD wants to merge 27 commits into
Draft
CodyCBakerPhD wants to merge 27 commits into
CodyCBakerPhD wants to merge 27 commits into
Conversation
…reaming annexed content `dandi validate` can now validate a DataLad/git-annex Dandiset (such as the clones at https://github.com/dandisets) whose content has not been fetched: with `--missing-file-content=stream`, the content of each annexed file is streamed on demand (via fsspec) from the URLs registered for it in git-annex, preferring direct S3 URLs over DANDI API download URLs, so that pynwb and nwbinspector run without downloading the (possibly terabytes of) data. - New `dandi.support.annex` module: parses git-annex keys from the broken symlinks, reads URL logs from the `git-annex` branch (local or remote-tracking) using only `git`, and provides an `AnnexReadableFile` `Readable` that streams from the first URL that can be opened. - `LocalFileAsset.content_source` lets an asset read its content from a `Readable` instead of `filepath`; `pynwb_utils.validate()` accepts a `readable`, and `NWBAsset` runs nwbinspector on the streamed file via `inspect_nwbfile_object()`. - Each streamed file yields an INFO `DANDI.FILE_CONTENT_STREAMED` result naming the URL; a file that cannot be streamed yields `DANDI.FILE_CONTENT_MISSING`. BIDS content-dependent errors are suppressed for annexed files under `stream` as under `only-non-data`. - `dandi validate` now accepts a broken symlink directly as a path argument (`click.Path(exists=True)` used to reject it). - Docs: describe validating DataLad Dandisets, including every Dandiset of the `dandisets` superdataset, and the datalad-fuse alternative. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## claude/memoize-readable-fingerprint #1933 +/- ##
=======================================================================
+ Coverage 78.63% 78.78% +0.15%
=======================================================================
Files 94 94
Lines 14595 14727 +132
=======================================================================
+ Hits 11477 11603 +126
- Misses 3118 3124 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dandi validate to operate on Datalad datasets through streaming
CodyCBakerPhD
commented
Sep 27, 2026
CodyCBakerPhD
commented
Sep 27, 2026
CodyCBakerPhD
commented
Sep 27, 2026
fsspec renamed its LRU block cache type from "block" to "blockcache" in 2023, and the declared minimum (2022.11.0) only knows the old name, so `AnnexReadableFile.open()` failed with `KeyError: 'blockcache'` in the lowest-deps CI job. Pick whichever name the installed fsspec registers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
- Drop the recipe for looping over the whole `dandisets` superdataset. - Note that streaming Zarr subdatasets is a follow-up for when NWB Zarr support has matured across the ecosystem. - Explain that suppressing content-dependent BIDS checks loses nothing for NWB datasets: the sidecar files are in git and the rest is in the names. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
commented
Sep 27, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
changed the base branch from
master
to
claude/docs-cli-options-sync
September 27, 2026 17:45
CodyCBakerPhD
added this pull request to stack #1935
September 27, 2026 17:53
`dandi validate` already handles annexed files whose content has not been
fetched (the broken symlinks of a DataLad dataset) according to
`--missing-file-content`: `error`, `skip`, or `only-non-data`. Those policies
were applied only when such a file was reached through its directory,
though: given directly on the command line, the file never got past argument
parsing, because `click.Path(exists=True)` follows the link and rejected it
with "does not exist". Check the path with `lexists()` instead, so that
dandi validate --missing-file-content=skip sub-01/sub-01.nwb
works the same as validating `sub-01/`, while paths that truly do not exist
are still rejected with the same error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Git merged the synced option reference and this branch's additions without conflict but with the `--missing-file-content`, `--format`, and `--output` entries and the link targets appearing twice, which Sphinx rejects. Keep the synced entries, adding the `stream` policy to `--missing-file-content`, and the "Validating DataLad Dandisets" section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
commented
Sep 27, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
removed this pull request from stack #1935
September 27, 2026 18:04
CodyCBakerPhD
changed the base branch from
claude/docs-cli-options-sync
to
claude/validate-broken-symlink-path
September 27, 2026 18:05
CodyCBakerPhD
added this pull request to stack #1938
September 27, 2026 18:05
…hamilton-ast647 The PR is now stacked on the broken-symlink fix, which carries its own copy of `ExistingPath` and its test; take that branch's docstring and extended test and keep this branch's streaming test alongside. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
The declared floor is pynwb >= 3.1.0 (and nwbinspector >= 0.7.0 requires pynwb >= 3.1 anyway), which makes several version guards unreachable: - validate(): only the `pynwb.validate(path=...)` branch (pynwb >= 3.0) can run; the `paths=[...]` tuple-returning form (2.2 to 2.x) and the `io=` fallback for even older releases are gone. - copy_nwb_file(): `cache_spec` has been accepted by `export()` since pynwb 2.8.2, so pass it unconditionally. - _get_external_images(): `ExternalImage` exists in every supported pynwb, so import it at module level instead of guarding an ImportError. The now-unused `packaging.version.Version` import is removed as well. The NWB *schema* version checks (e.g. the < 2.1.0 error filter) are about the file being validated, not about pynwb, and are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
changed the base branch from
claude/validate-broken-symlink-path
to
claude/pynwb-utils-drop-legacy-pynwb
September 27, 2026 19:28
CodyCBakerPhD
added this pull request to stack #1940
September 27, 2026 19:29
…-hamilton-ast647 Resolves the conflict in dandi/pynwb_utils.py: the streaming (`readable`) branch keeps validating through `pynwb.validate(io=...)`, while the local path now always uses `pynwb.validate(path=...)`; the pre-3.0 pynwb branches and the tuple normalization are gone along with the base. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
fscacher's memoize_path derives its cache key from a stat of the file at the given path, so it never caches a call made on a `Readable` (there is nothing to stat). Some readables can nevertheless vouch for their content better than mtime/inode ever could, e.g. with a content digest such as a git-annex key: results computed from one such resource are valid for any other with the same fingerprint. - `Readable.get_fingerprint()`: new optional hook returning a content fingerprint string, `None` (the default) meaning "unknown, do not cache". - `pynwb_utils.memoize_source(cache, tokens)`: a decorator that behaves exactly like `cache.memoize_path` for path arguments and, for a `Readable` with a fingerprint, caches under (file name, fingerprint, tokens, other arguments) via `cache.memoize` instead. The tokens are passed explicitly since plain `memoize` does not add them by itself. - Applied to `get_metadata`, `get_neurodata_types`, and `nwb_has_external_links`, the memoized functions that already accept a `Readable`. No behavior change for paths or for readables without a fingerprint (still none of them at this point). Since the decorator preserves the signature, mypy now sees through `get_metadata`, whose return annotation claimed `dict | None` although it never returns `None`; fixed to `dict[str, Any]`. isort also reordered a pair of local imports in `RemoteReadableAsset.open()` while at it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647 Brings in `Readable.get_fingerprint()` and `pynwb_utils.memoize_source`. The only conflict was the `typing` import line of the test fixtures. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
`AnnexReadableFile.get_fingerprint()` returns the file's git-annex key, which is a digest of the content (plus its size), and `pynwb_utils.validate` now goes through `memoize_source` for its `readable=` as well, with the content source as the first argument of the memoized `_validate_cached`. The metadata side (`get_metadata` & co.) picked the caching up already by accepting a `Readable`. So re-running `dandi validate --missing-file-content=stream` on a clone skips the pynwb validation and the metadata extraction of every file whose key has not changed, instead of streaming it again. nwbinspector's checks are not cached (they are not for local files either). Documented in the notes of the "Validating DataLad Dandisets Remotely" section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
`PersistentCache.memoize(exclude_kwargs=...)` only exists from fscacher 0.4 on; the declared floor is 0.3.0, where the argument is called `ignore`, so the lowest-deps job failed at import time. Pick the name from the signature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647 Picks up the fscacher < 0.4 compatibility fix for `memoize_source`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
joblib 1.3 (the declared floor) mistakes positional arguments beyond the
fingerprint ones for the keyword-only `_source` parameter of the memoized
function ("Keyword-only parameter '_source' was passed as positional
parameter"), which broke `validate(path, readable=...)` under lowest-deps.
Bind the decorated function's other arguments by name instead (defaults
included), which also makes the cache key independent of how they were
passed; the decorated function must not take *args as a consequence.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…hamilton-ast647 Picks up the joblib 1.3 compatibility fix for `memoize_source`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
CodyCBakerPhD
removed this pull request from stack #1940
September 27, 2026 20:17
CodyCBakerPhD
changed the base branch from
claude/pynwb-utils-drop-legacy-pynwb
to
claude/memoize-readable-fingerprint
September 27, 2026 20:17
CodyCBakerPhD
added this pull request to stack #1942
September 27, 2026 20:17
The adapter only existed because `_validate` kept its pre-caching parameter order; with the content source as its first parameter, `memoize_source` can decorate `_validate` directly. Same arguments and values reach the memoized function, so cache keys are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
…uments Instead of re-deriving a `readable` from the type of `source`, read whatever the source is through `open_readable()` and validate via `pynwb.validate(io=)` for local files too; since pynwb 3.0 that is the same code path as `validate(path=)`, cached namespaces included (`_get_pynwb_metadata` already reads local files this way). Document both arguments on the function. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
ba2f2b3 routed local files through open_readable() + h5py's file-object driver + pynwb.validate(io=...), the same way streamed content is validated. The lowest-deps CI job started failing on that commit while the same tests pass locally under the same dependency floors, so go back to pynwb.validate(path=...) for local files and keep the io route for Readables only. The explicit (source, path) arguments stay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Contributor
Author
|
Try using datalad-fuse (with fsspec-adapter), NOT via FUSE itself Test against; datalad/datalad-fuse#131 |
…hamilton-ast647 The base was rebased onto master (with #1937 and #1939 squash-merged) and now uses fscacher >= 0.5.0's memoize_path(custom_fingerprint=...). Resolve accordingly: - Drop memoize_source and its tests: get_metadata, get_neurodata_types, nwb_has_external_links and _validate now use memoize_path(custom_fingerprint=readable_fingerprint). - Drop ExistingPath in favor of LinkAwarePath(lexists=True) from #1937. - Keep the streaming support (_validate reading from a Readable) and its test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
CodyCBakerPhD
pushed a commit
that referenced
this pull request
Oct 6, 2026
…ally Moved over from #1933 without its integration into `dandi validate`: dandi.support.annex reads the key of an annexed file from its (possibly broken) symlink and the URLs registered for it from the git-annex branch, using only git, and AnnexReadableFile streams the content from those URLs with fsspec. Its get_fingerprint() is the key, so results of the functions memoized with readable_fingerprint (metadata, ...) are cached under it and shared by files with the same key and name. Tests cover key and URL-log parsing, the git-annex branch lookup, streaming from file:// and HTTP(S) URLs, and caching by key, using fixtures that fake a DataLad Dandiset with plain git. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…hamilton-ast647 AnnexReadableFile and its tests moved to the base; take its versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
fscacher is untyped, so since _validate is decorated with fscacher's memoize_path (instead of the typed memoize_source) mypy sees it as returning Any. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…hamilton-ast647 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Towards the goal of having a source of internally trusted Truth regarding validation records on a Dandiset, it is necessary to be able to run the validation tool through streamed file contents.
This requires a bit of reworking here on the CLI.
Current strategy is, for simplicity, fsspec on a typical DataLad dataset with special remotes on S3.
AI Summary
Summary
dandi validatecan now validate a DataLad / git-annex Dandiset (such as the clones at https://github.com/dandisets) whose content has not been fetched. With the new--missing-file-content=streampolicy, the content of each annexed file is streamed on demand (via fsspec) from the URLs registered for it in git-annex, so pynwb and nwbinspector run on the file without downloading the (possibly terabytes of) data. Onlygitis needed to read the git-annex metadata (neither git-annex nor DataLad has to be installed), plusfsspec[http](pip install "dandi[extras]") for the streaming.The motivation is generating validation records for every Dandiset:
The docs (
docs/source/cmdline/validate.rst) gain a "Validating DataLad Dandisets" section, including how to loop over the subdatasets of thedandisetssuperdataset, and mention the datalad-fuse alternative (which needs FUSE and so does not work in many containers/CI environments; this PR works anywheregitand HTTPS are available).What changed
dandi/support/annex.py(new): parses git-annex keys from the broken symlinks (SHA256E-s<size>--<hash>.nwb), locates a key's URL log in thegit-annexbranch (local or remote-tracking, e.g.origin/git-annexafter a plaingit clone) viagit cat-file, and providesAnnexReadableFile, aReadablethat streams from the first URL that can be opened (direct S3 URLs are preferred overapi.dandiarchive.org/.../download/URLs, which redirect on every request). Uses fsspec'sblockcacheso h5py's random access only fetches the blocks it touches.LocalFileAsset.content_source: an optionalReadablean asset reads its content from instead offilepath.pynwb_utils.validate()takes areadable=(results not cached in that case), andNWBAssetruns nwbinspector on the streamed file throughinspect_nwbfile_object(), replicatinginspect_nwbfile()'s error reporting so the result IDs are identical to local validation.validate()core: understream, each streamable file yields an INFODANDI.FILE_CONTENT_STREAMEDresult naming the URL; a broken symlink that cannot be streamed (not annexed / no URL registered) yieldsDANDI.FILE_CONTENT_MISSING. BIDS content-dependent errors are suppressed for annexed files understreamas they are underonly-non-data. A clear error is raised up front if fsspec/aiohttp are missing.streamadded to--missing-file-content;dandi validatenow also accepts a broken symlink directly as a path argument (click.Path(exists=True)used to reject it with "does not exist").ai_generated): key/URL-log parsing,AnnexRepoon a repo whosegit-annexbranch is created with git plumbing (both local andorigin/refs),AnnexReadableFileoverfile://URLs and over a local range-capable HTTP server (h5py opens the file), and end-to-endvalidate()/CLI runs asserting that streamed results equal those of validating the same NWB file locally.Verification
dandi validate --missing-file-content=streamon agit cloneofdandisets/000029(6 NWB files, 18 KB–18 MB) streams everything from S3 in ~10 s and produces 77 records (the same pynwb/nwbinspector findings as for local files); the same for000029checked out as a git submodule (.gitfile) and for a single 2.8 MB file of000035. A 61 GB file of000003opens with one HEAD and a single 4 MiB range request.Limitations / notes
dandisetsrepos they are separate uninstalled subdatasets, i.e. empty directories that are not validated at all).git-annexbranch (no--single-branch).🤖 Generated with Claude Code
https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA
Generated by Claude Code