Configurable fsspec reads and coalescing block cache

[CodyCBakerPhD]

On the latest dandi-cli PRs for streaming validation, we are found the resulting inefficiency:

• With fsspec's default read-ahead, reading an NWB file's structure downloads about as much as the whole file each time it is opened.

I will attempt to open a PR on fsspec for the new block cache, but no guarantee they will accept or merge that. If they do, I'll come back and reduce the custom code for that added here.

🤖 AI text below 🤖

Changes

• Configurable fsspec reads. DatasetAdapter takes fsspec_options, which are passed to fsspec's HTTPFileSystem (block_size, cache_type, cache_options). FsspecAdapter passes them along.
• datalad_fuse.caching.CoalescingBlockCache, registered with fsspec as "coalescing-blockcache".
  • Like fsspec's "blockcache", it keeps the least recently used blocks. That suits HDF5's reads of small pieces scattered over a file, which fsspec's default 5 MiB read-ahead cache keeps discarding.
  • Unlike "blockcache", it fetches each run of consecutive missing blocks with one request, instead of one request per block.
  • It can be dropped once/if fsspec's BlockCache does the same (TODO: update target BlockCache: fetch each run of consecutive missing blocks with one request filesystem_spec#1).

Split out of #2, together with #4 (opt-out of initializing git-annex). Both PRs add a parameter to DatasetAdapter.__init__, so whichever lands second has a small conflict there: keep both parameters.

The parameter name fsspec_options matches the variant based on datalad#131, so callers look the same with either. Whichever of this and datalad#131 lands second needs its conflicts resolved: the same few lines move from DatasetAdapter.__init__ in fsspec.py to FsspecBackend / adapter.py.

Tests

New datalad_fuse/tests/test_caching.py:

• A unit test of the cache.
• The fsspec options reach the HTTPFileSystem.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code