validate: stream missing annexed content with datalad-fuse

[CodyCBakerPhD]

Stacked on #1955. This is an alternative to #1933: the same validate integration, built on the datalad-fuse readable from #1955 instead of the git-only reader.

Changes

• New stream policy for --missing-file-content:
  • Annexed files whose content isn't present locally are opened through AnnexedReadableFile (datalad-fuse, no FUSE mount).
  • pynwb and nwbinspector then run on that streamed content.
  • Each streamed file yields an INFO result, DANDI.FILE_CONTENT_STREAMED, naming the first URL git-annex reported for its content.
  • A file that can't be streamed yields a DANDI.FILE_CONTENT_MISSING error.
• Pipeline support, carried over from Enable dandi validate to operate on Datalad datasets through streaming #1933:
  • LocalFileAsset.content_source lets an asset read its content from a Readable.
  • pynwb_utils.validate(..., readable=...) validates content from a Readable; its results are cached under the annex key via annex_fingerprint from Read missing annexed content with datalad-fuse, cached by annex key #1955.
  • nwbinspector inspects the streamed file object.
  • BIDS errors that need file content are suppressed for streamed files, as under only-non-data.
• Requirements: the policy requires datalad-fuse (pip install "dandi[datalad]") and git-annex, and raises an error if either is missing.
• Docs: "Validating DataLad Dandisets Remotely" in the validate docs explains the policy, using datalad clone.
• Tests: the stream tests from Enable dandi validate to operate on Datalad datasets through streaming #1933 are rewritten to use a real git-annex repository and the local HTTP range server from Read missing annexed content with datalad-fuse, cached by annex key #1955. They cover:
  • the results match a local copy of the file;
  • pynwb results are cached by key;
  • the unstreamable and unreadable-URL cases;
  • the missing-requirements error.

Compared to #1941 + #1933

905 added lines against the current stack's 1493; excluding tests, 406 against 642. Everything besides the content reader is the same as #1933.

The trade-offs:

• Dependencies: streaming needs git-annex and DataLad. The current stack needs only git and fsspec.
• Setup: git-annex must be initialized in the clone, which datalad clone does.
• Test coverage: the streaming tests run only in the dedicated datalad CI job, while the current stack's run on every platform.
• URL order: the git-only reader prefers direct S3 URLs over DANDI API download URLs, while this uses the order git-annex reports.
• Real data: neither stack has been tried yet against a real dandisets/ clone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.80952% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.93%. Comparing base (7964633) to head (0be1c28).

Files with missing lines	Patch %	Lines
dandi/validate/_core.py	94.59%	2 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                        @@
##           claude/datalad-fuse-readable    #1956      +/-   ##
================================================================
+ Coverage                         90.86%   90.93%   +0.07%     
================================================================
  Files                                94       94              
  Lines                             14381    14520     +139     
================================================================
+ Hits                              13067    13204     +137     
- Misses                             1314     1316       +2     

Flag	Coverage Δ
unittests	90.93% <98.80%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.