Skip to content

Read missing annexed content with datalad-fuse, cached by annex key - #1955

Draft
CodyCBakerPhD wants to merge 7 commits into
masterfrom
claude/datalad-fuse-readable
Draft

Read missing annexed content with datalad-fuse, cached by annex key#1955
CodyCBakerPhD wants to merge 7 commits into
masterfrom
claude/datalad-fuse-readable

Conversation

@CodyCBakerPhD

@CodyCBakerPhD CodyCBakerPhD commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This PR and the one stacked on it are an alternative to #1941 and #1933. They build on datalad-fuse 0.5.1, the current PyPI release, instead of a git-only reader of the git-annex branch. The goal is the same: dandi validate on a DataLad clone of a Dandiset, streaming file content without downloading it.

Changes

  • dandi.support.datalad_fuse.AnnexedReadableFile is a Readable for a locked annexed file whose content isn't present locally.
    • It streams the content through datalad-fuse's FsspecAdapter (no FUSE mount, caching=False). The adapter asks git-annex where the content is (whereis) and reads it over HTTP(S) with fsspec.
    • get_annexed_readable(path) returns None in any of these cases:
      • datalad-fuse isn't installed;
      • the path isn't a broken symlink;
      • git-annex isn't initialized in the repository (otherwise DataLad would initialize it);
      • the key records no size;
      • git-annex knows no URL for the content.
  • Caching by annex key: get_metadata, get_neurodata_types and nwb_has_external_links use the new pynwb_utils.annex_fingerprint as their custom_fingerprint.
    • It wraps fscacher 0.5.0's annex_key_fingerprint, which keys on the path paired with the annex key.
    • Locked annexed files are cached by key, whether they are streamed or present locally. Only content-hash backends count, so WORM and URL keys fall back to stat().
    • Unlike Memoize readable fingerprints #1941, this needs no fingerprint hook on Readable.
  • Dependencies:
    • New datalad extra: datalad-fuse >= 0.5.1, which also requires git-annex.
    • fscacher >= 0.5.0.
  • Tests:
    • make_git_annex_dandiset() builds a real git-annex repository (content added, URL registered, then dropped).
    • A range_http_server fixture serves files with range requests.
    • New skipif.no_git_annex condition.
    • A new CI job (mode: datalad, ubuntu, Python 3.12) installs git-annex and .[datalad] from PyPI. The other jobs skip these tests.

Compared to #1941

410 added lines against #1941's 990. It drops AnnexKey, AnnexRepo, the URL-log parsing, the fake-annex fixtures and the Readable.get_fingerprint hook. In exchange, streaming requires git-annex and DataLad.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

Generated by Claude Code

@CodyCBakerPhD CodyCBakerPhD mentioned this pull request Oct 6, 2026
Draft
@CodyCBakerPhD CodyCBakerPhD self-assigned this Oct 6, 2026
@codecov

codecov Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.90617% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.95%. Comparing base (ad8f64d) to head (2d461dd).

Files with missing lines Patch % Lines
dandi/support/datalad_fuse.py 91.13% 14 Missing ⚠️
dandi/tests/fixtures.py 93.24% 5 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1955      +/-   ##
==========================================
+ Coverage   90.77%   90.95%   +0.18%     
==========================================
  Files          92       94       +2     
  Lines       14130    14500     +370     
==========================================
+ Hits        12826    13188     +362     
- Misses       1304     1312       +8
Flag Coverage Δ
unittests 90.95% <94.90%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1959 October 12, 2026 21:30
@CodyCBakerPhD CodyCBakerPhD added cmd-validate enhancement New feature or request minor Increment the minor version when merged labels Oct 12, 2026
@CodyCBakerPhD
CodyCBakerPhD force-pushed the claude/datalad-fuse-readable branch from 7964633 to 716d0d7 Compare October 12, 2026 21:32
claude added 7 commits October 15, 2026 13:58
@claude
Read missing annexed content with datalad-fuse, cached by annex key
e7a0cea 
Add `dandi.support.datalad_fuse.AnnexedReadableFile`, a `Readable` for a
locked annexed file whose content is not present locally, streamed with
the adapter of datalad-fuse (no FUSE mount), which asks git-annex where
the content is.  It is available with the new `datalad` extra.

Cache NWB metadata of locked annexed files, streamed or local, under
their git-annex key paired with their path, with fscacher 0.5.0's
`annex_key_fingerprint`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Move annex_fingerprint next to AnnexedReadableFile; trim field comments
0eb52d4 
annex_fingerprint is not specific to NWB files, and living in
dandi.support.datalad_fuse it needs no deferred import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Note URL order and proxy behavior of datalad-fuse
ace870b 
As documented in datalad/datalad-fuse#138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Import fscacher lazily in dandi.support.datalad_fuse
09666db 
fscacher imports joblib, which imports numpy, so importing it with the
module made the CLI (via dandi.validate in the stacked PR) import numpy,
failing test_no_heavy_imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Disable automatic GC while a streamed annexed file is open
c1f2bab 
h5py holds its global lock while reading from a Python file object, and
the read waits for another thread to fetch the data over HTTP (fsspec's
I/O thread, or an HTTP server in the same process as in the tests).  A
garbage collection in that thread could then run the finalizer of an
unreferenced h5py-backed object (e.g., hdmf's HDF5IO.__del__), which
waits for the same lock: a deadlock, as seen in CI in
test_validate_stream, ended by pytest-timeout with a segfault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Read streamed annexed content in small cached blocks
636589f 
datalad-fuse opens the files with fsspec's defaults: 5 MiB of read-ahead,
discarded whenever a read jumps elsewhere.  h5py reads many small pieces
scattered over an NWB file, so each open downloaded about as much as the
whole file (64 MB for the neurodata types of a 16 MB file in the new
test), and dandi opens each file several times during validation.  Read
in cached blocks of 256 KiB instead (1.3 MB for the same file).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@claude
Fetch consecutive missing blocks of streamed content in one request
608f3ab 
fsspec's "blockcache" fetches each block of a read with its own request,
so a large read (e.g., of a dataset's data, by NWB Inspector) took one
round trip per 256 KiB, i.e., a couple of MB/s however fast the
connection.  Use instead a block cache that fetches each run of
consecutive missing blocks with a single request, and keeps up to 512
blocks (128 MiB) per open file rather than 32, which h5py's scattered
reads of metadata kept evicting and fetching again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@CodyCBakerPhD
CodyCBakerPhD force-pushed the claude/datalad-fuse-readable branch from 2d461dd to 608f3ab Compare October 15, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@CodyCBakerPhD CodyCBakerPhD

Labels

cmd-validate enhancement New feature or request minor Increment the minor version when merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@CodyCBakerPhD @claude