ENH: add remfile backend with pluggable backend architecture

[yarikoptic]

Summary

• Add pluggable backend system for remote file access with --backends CLI option (comma-separated, priority-ordered)
• Add remfile backend (optional pip install datalad-fuse[remfile]) optimized for HDF5 access patterns (.nwb, .h5, .hdf5, etc.)
• Refactor monolithic fsspec.py into focused modules: backends.py (ABC), remfile.py, adapter.py, slimmed fsspec.py
• Preserve backward compatibility for old datalad_fuse.fsspec import paths via DeprecationWarning

How it works

The backend chain walks --backends=remfile,fsspec (default) in order. Each backend's can_handle(key, mode) decides if it should handle a file based on the annex key's extension. First match tries all URLs; on failure, falls through to the next backend. Configuration via --backends flag, datalad.fusefs.backends config variable, or programmatic API.

New module layout

Module	Contents
backends.py	Backend ABC,DEFAULT_BACKENDS
remfile.py	RemfileBackend, RemfileWrapper
adapter.py	RemoteFilesystemAdapter (renamed from FsspecAdapter), DatasetAdapter, FileState
fsspec.py	FsspecBackend, HTTP helpers, backward-compat __getattr__

Commits

• f8cc531 RF: address code review findings
• 3dfd2d0 TST: add backward compatibility deprecation tests
• 990eae0 ENH: add remfile backend with pluggable backend architecture

Test plan

• Manual testing
• Unit tests pass: pytest datalad_fuse/tests/test_backends.py datalad_fuse/tests/test_deprecations.py -k "not network"
• Integration tests with pinned S3 objects (ros3test.hdf5, ros3test.nwb, README.md): pytest datalad_fuse/tests/test_backends.py -k network
• Existing tests unaffected: pytest datalad_fuse/tests/test_util.py
• Backward compat: old from datalad_fuse.fsspec import FsspecAdapter still works with DeprecationWarning

🤖 Generated with Claude Code

Overall it grew larger than initial quick "fix" was (first commit) since the whole setup had 'fsspec' naming throughout it.

[codecov]

Codecov Report

❌ Patch coverage is 98.25000% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.04%. Comparing base (83a0a30) to head (b4dbee6).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
datalad_fuse/adapter.py	96.00%	13 Missing ⚠️
datalad_fuse/tests/test_backends.py	98.82%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #131      +/-   ##
==========================================
+ Coverage   92.84%   97.04%   +4.20%     
==========================================
  Files          13       18       +5     
  Lines        1188     2100     +912     
==========================================
+ Hits         1103     2038     +935     
+ Misses         85       62      -23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a pluggable remote-file backend architecture (defaulting to remfile,fsspec) and introduces a remfile backend optimized for HDF5-style access while preserving legacy datalad_fuse.fsspec import paths via DeprecationWarning.

Changes:

• Introduces backend ABC + backend resolution/creation logic and refactors the adapter layer into adapter.py
• Adds RemfileBackend + wrapper, and refactors fsspec.py into a backend + compatibility shim
• Updates CLI interfaces and tests to support --backends and validate backward compatibility

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tox.ini	Installs additional extras and expands mypy targets to cover new modules
setup.cfg	Adds new extras (remfile, full) for optional backend dependencies
datalad_fuse/tests/test_forgejo.py	Updates imports to the new module locations
datalad_fuse/tests/test_deprecations.py	Adds tests for legacy import-path deprecation behavior
datalad_fuse/tests/test_backends.py	Adds unit/integration tests for backend chaining, remfile, and fsspec behavior
datalad_fuse/remfile.py	Implements RemfileBackend and RemfileWrapper
datalad_fuse/fuse_.py	Switches FUSE operations layer to use RemoteFilesystemAdapter + adds backend option
datalad_fuse/fsspec_head.py	Adds --backends CLI option and switches to new adapter
datalad_fuse/fsspec_cache_clear.py	Updates imports to the new adapter module
datalad_fuse/fsspec.py	Refactors into FsspecBackend, keeps async HTTP helpers, adds compatibility __getattr__
datalad_fuse/backends.py	Adds Backend ABC and DEFAULT_BACKENDS
datalad_fuse/adapter.py	Adds backend-agnostic dataset/remote filesystem adapter and backend chain logic
datalad_fuse/init.py	Adds --backends option to the main FuseFS interface

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[yarikoptic-gitmate]

Review and test drive of 3c3bdd8

The backend chain and the module split look good to me. The fsspec.py shim works, CI is green, and in a real FUSE mount remfile reads HDF5 noticeably faster. I found two real bugs in RemfileWrapper that I'd fix before merging, and two behaviour changes in the new default (remfile,fsspec) that deserve a decision or documentation. Details are in the inline comments; summary below.

Should fix before merge

1. RemfileWrapper.info() makes a fresh HEAD request. FUSE reaches it via getattr(path, fh) on open handles. Against a server that rejects HEAD (which presigned S3 GET URLs do), fstat on an open file fails with HTTPError 403; fsspec gets the size fine. The fix needs no network call: self._f.length.
2. RemfileWrapper.read(): with no size it returns b"" and moves the position back by 1. It never clamps at EOF. For files whose size is a multiple of remfile's 100 KiB chunk, a read at EOF raises 416 Range Not Satisfiable, which also breaks line iteration and so fsspec-head -n. The suggested clamp fixes all three. Also minor: seek() returns None.

Behaviour changes to decide on or document

3. Dead URLs stall about 30 s each under remfile (its open probe retries 8× with backoff), versus about 1 s under fsspec. I measured 30.2 s vs 1.07 s on dandiset 000027. In fusefs this happens under the global rwlock, so the whole mount stalls.
4. --caching=ondisk no longer applies to HDF5 files once remfile is installed: no cache files were written, and fsspec-cache-clear has nothing to clear for them.

Smaller points

• URL/VURL keys (addurl --fast/--relaxed) have no suffix, so .nwb files with such keys go to fsspec. Falling back to relpath's extension would fix that.
• datalad.fusefs.backends is read from the global cfg, so dataset-local config is ignored. Same as the existing cache-clear option, but ds.config is right there.
• When a backend falls through, get_urls() and the exporttree lookup (whereis subprocess, boto3) run again.
• The README isn't updated: nothing on --backends, datalad.fusefs.backends, or the [remfile]/[full] extras, and the intro still says fsspec is used. The --caching help says "fsspec'ed".
• Nit: with --backends remfile on README.md, the error is Could not open README.md within … (backends=remfile) with no cause attached. Saying that no configured backend handles .md would be clearer.

What I ran

Setup: Ubuntu 24.04, Python 3.13, datalad 1.7.1, git-annex 10.20260901, remfile 0.1.15, fsspec 2026.9.0, h5py 3.16, libfuse2.

• Unit and network tests: test_backends.py test_deprecations.py test_exporttree.py test_util.py → 134 passed, including the S3 integration tests.
• Full suite with --libfuse: 211 passed. The rest is environmental: 8 test_forgejo errors (no docker in my sandbox) and 1 failure of test_fuse.py::test_parallel_access, because archive.org is blocked by the sandbox network policy. That test fails the same way on master.
• flake8 and mypy (the tox lint/typing invocations): clean.
• End to end, own dataset: MD5E .nwb/.hdf5/.md files registered to the pinned dandiarchive.s3 objects, then dropped.
  • fsspec-head dispatched correctly: remfile for .nwb/.hdf5, fsspec for .md.
  • --backends fsspec|remfile|bogus|' , ' and -c datalad.fusefs.backends=fsspec all behaved as described.
  • Through fusefs, the sha256sum of all three files matched the originals, and h5py walked both HDF5 files (42 and 5 objects).
• Real dandiset 000004, a 73 MB ecephys+image.nwb read through fusefs:
  • Full sequential read: 4.6 s with remfile vs 22.6 s with fsspec. The checksum matched the SHA256E key in both cases.
  • h5py open, traverse and a small slice: comparable and noisy (≈5.2 s vs 3.1–6.9 s across runs).
  • Peak RSS: similar (~146 vs ~141 MiB).
• Deprecation shim: old names warn, unknown names still raise AttributeError/ImportError, and FsspecBackend imports without a warning.

---

Generated by Claude Code

[yarikoptic-gitmate]

Bug: the HEAD request breaks fstat() on open FUSE handles when the server rejects HEAD.

This is not "rarely called". DataLadFUSE.getattr(path, fh) takes the "File already open" branch and calls file_getattr(f) → f.info() whenever the kernel asks for attributes of an open handle (fgetattr). I saw this in a real mount log while h5py was opening ros3test.nwb. It does not depend on the key carrying a size.

The HEAD is also the request remfile itself avoids, because presigned S3 GET URLs (e.g. what DANDI's api.dandiarchive.org/.../download/ redirects to) reject HEAD. When that happens, urllib raises HTTPError, which is not a FileNotFoundError, so file_getattr does not map it to ENOENT and fusepy gets an unhandled exception.

Reproduced deterministically: a dataset whose only URL is a local server that answers HEAD with 403 and serves ranged GETs. I called ops = DataLadFUSE(ds, caching=False, backends=...), then fh = ops.open(path, O_RDONLY), then ops.getattr(path, fh):

== remfile,fsspec
open -> 1000 RemfileWrapper
getattr(path, fh) FAILED: HTTPError HTTP Error 403: Forbidden
== fsspec
open -> 1000 HTTPFile
getattr(path, fh) size: 177728

fsspec survives because HTTPFile.info() returns the details it cached at open time. remfile also already knows the size, since RemFile.__init__ fetched Content-Length with an aborted GET. So no network call is needed here:

Suggested change

	def info(self) -> dict[str, Any]:
	def info(self) -> dict[str, Any]:
	"""Minimal info dict matching the fsspec convention.
	remfile already determined the size (via an aborted GET, which unlike
	HEAD also works for presigned URLs) when the file was opened.
	"""
	return {"type": "file", "size": self._f.length}

With this change, the urllib.request import and the two test_info_* tests that mock urlopen need adjusting.

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. info() now returns {"type": "file", "size": self._f.length} and makes no request at all. The urllib.request import stays in the module, but only for the new size probe in open_url() (see the reply on the retry/stall thread). The two test_info_* tests that mocked urlopen are replaced by test_info_uses_remfile_length, which asserts urlopen is not called.

[yarikoptic-gitmate]

Bug: read() with no size returns b"" and moves the position backwards, and reading at EOF can raise.

RemFile.read() does not support size=-1. With -1, its chunk range comes out empty, so it returns b"" and then does self._position += size. Against the real S3 object:

f.seek(10); f.read()          -> 0 bytes; tell() == 9
fsspec: g.seek(10); g.read()  -> 177718 bytes

RemoteFilesystemAdapter.open() returns this object typed as IO, so any API user calling .read() silently gets empty data.

RemFile.read(n) has two related problems:

• It always advances by n, even past EOF. read(100) at length-5 leaves tell() == length+95.
• When the size is an exact multiple of its 100 KiB chunk, a read at EOF issues an unsatisfiable Range request instead of returning b"". That breaks __next__'s EOF detection, so fsspec-head -n fails. On a 204800-byte .h5 served locally:

iteration FAILED at EOF: _PermanentHTTPError Error fetching bytes 204800-204799: 416 Range Not Satisfiable

Clamping to the remaining length fixes all of these:

Suggested change

	def read(self, size: int = -1) -> bytes:
	return self._f.read(size) # type: ignore[no-any-return]
	def read(self, size: Optional[int] = -1) -> bytes:
	# remfile requires an explicit size, does not clamp at EOF, and may
	# issue an unsatisfiable Range request when reading at EOF
	remaining = max(self._f.length - self._f.tell(), 0)
	if size is None or size < 0 or size > remaining:
	size = remaining
	if not size:
	return b""
	return self._f.read(size) # type: ignore[no-any-return]

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, with the suggested clamp. __next__ now reads through self.read() rather than self._f.read(), so line iteration gets the clamp too — otherwise the 416 at EOF still broke fsspec-head -n.

The mock in _make_mock_remfile was made faithful to remfile.RemFile rather than well-behaved (requires an explicit non-negative size, advances past EOF, raises 416 at EOF, seek() returns None); with that, the existing test_iteration/test_iteration_empty went red before the fix. Checked against the real S3 object too: seek(size); read(10) → b"", over-long read returns the tail and leaves tell() at EOF, read() with no size returns all 4008 bytes.

[yarikoptic-gitmate]

Minor: RemFile.seek() has no return statement, so this returns None despite -> int. mypy doesn't catch it because self._f is Any. In the test drive, f.seek(100) gave None, while io.BytesIO(...).seek(100) gives 100.

Suggested change

	def seek(self, offset: int, whence: int = 0) -> int:
	return self._f.seek(offset, whence) # type: ignore[no-any-return]
	def seek(self, offset: int, whence: int = 0) -> int:
	self._f.seek(offset, whence)
	return self.tell()

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, as suggested. Covered by test_seek_returns_new_position.

[yarikoptic-gitmate]

Unreachable URLs stall about 30× longer than with fsspec, and in FUSE that blocks the whole mount.

remfile.File(url) probes Content-Length and retries transient failures (connection errors, 5xx) 8 times with exponential backoff, about 25 s plus connect time. Only 4xx fails fast. The adapter tries URLs one after another, so each dead URL costs that much before the next one is tried.

Test: dandiset 000027. Its first whereis URL is api.dandiarchive.org/..., which my sandbox's network policy blocks. Its second URL is a versioned dandiarchive.s3 one. Running datalad fsspec-head -c 8 sub-RAT123/sub-RAT123.nwb:

--backends	wall time
remfile,fsspec (default)	30.2 s
fsspec	1.07 s

In fusefs, DataLadFUSE.open() calls self._adapter.open() while holding the global rwlock, so every other read on the mount waits the whole time. Dead mirrors, firewalled hosts, offline special-remote URLs and the several candidate URLs that get_urls() builds per HTTP remote all hit this.

Possible fix: do a cheap probe here with no retries and a short timeout, for example a ranged GET bytes=0-0 that also reads Content-Range to get the size, so URL fall-through stays fast. Then pass the known size to remfile (key.size is almost always available) so it skips its own probe. remfile only exposes that as the underscore-private _size=, though. It may be worth asking remfile upstream for public size= and retry parameters, or pinning the remfile version.

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, along the lines you suggested. RemfileBackend.open_url() now does the cheap probe itself — one GET bytes=0-0 with a 10 s timeout and no retries — reads the total from Content-Range (falling back to Content-Length when the server ignores Range, and to None when it says neither), and passes it to remfile.File(..., _size=...) so remfile skips its own retrying probe. For a live URL that is the same single round trip it was already paying.

Measured against a missing object in the same bucket: 0.20 s, down from ~30 s. The pinned-S3 integration tests still pass unchanged.

_size= is private API, so setup.cfg now pins remfile ~= 0.1.15. Asking upstream for a public size= (and retry parameters) still seems worth doing — happy to file that separately.

[yarikoptic-gitmate]

Dispatching only on key.suffix misses HDF5 files whose keys come from a non-*E backend. git annex addurl --fast/--relaxed (and datalad addurls --fast) produce URL/VURL keys, and AnnexKey.parse sets suffix=None for those:

fast.nwb      VURL-s177728--https://dandiarchive.s3.amazonaws.com/ros3test.nwb
relaxed.hdf5  VURL--https://dandiarchive.s3.amazonaws.com/ros3test.hdf5
fast.nwb: backend remfile cannot handle (suffix=None, mode=rb)
fast.nwb: opening via backend fsspec

The dandisets/* datasets use SHA256E, so they are fine. DatasetAdapter.open() already has relpath, though. Passing it to can_handle() (e.g. can_handle(key, mode, relpath)) and falling back to PurePosixPath(relpath).suffix when the key has no suffix would make the choice follow what the user sees in the tree.

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. Backend.can_handle() takes a third relpath argument (optional, so the signature stays usable standalone), and RemfileBackend falls back to PurePosixPath(relpath).suffix when the key carries no suffix. DatasetAdapter.open() passes the relpath it already has. A key suffix still wins when present.

[yarikoptic-gitmate]

--caching=ondisk is now silently a no-op for HDF5 files once remfile is installed. caching is passed only to FsspecBackend. With the new default remfile,fsspec, .nwb/.h5 files bypass the on-disk cache entirely:

datalad fsspec-head --caching ondisk --backends remfile,fsspec -c 8 ros3test.nwb  -> 0 files in .git/datalad/cache
datalad fsspec-head --caching ondisk --backends fsspec         -c 8 ros3test.nwb  -> 2 files in .git/datalad/cache

So installing datalad-fuse[full] quietly changes what --caching ondisk does for exactly the large files where it matters most. Options:

• wire remfile.DiskCache into e.g. .git/datalad/cache/remfile, and clear it in RemfileBackend.clear() so fsspec-cache-clear and datalad.fusefs.cache-clear keep working;
• have RemfileBackend decline when caching=True;
• at least document it, and update the --caching help, which still says "cache fsspec'ed files".

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching — the first of your three options. RemfileBackend now takes (path, caching) like FsspecBackend, builds a remfile.DiskCache at .git/datalad/cache/remfile when caching is on, passes it to remfile.File(..., disk_cache=...), and removes the directory in clear(), so fsspec-cache-clear and datalad.fusefs.cache-clear cover HDF5 files again. Verified end to end against the real S3 object: a 100-byte read writes a chunk under that directory, and clear() removes it.

The --caching help text still says "fsspeced" — that and the README are in the follow-up noted on the PR.

[yarikoptic-gitmate]

The global datalad.cfg doesn't read the dataset's own .git/config or .datalad/config. So git config datalad.fusefs.backends fsspec inside a dataset is ignored, whether the command runs from inside it or with -d. I checked both: remfile was still used. This matches how datalad.fusefs.cache-clear is read today, so it is not a regression. DatasetAdapter.__init__ already builds Dataset(path), though, so ds.config.get("datalad.fusefs.backends") would honor per-(sub)dataset settings at no extra cost.

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. resolve_backends() takes an optional config, and DatasetAdapter.__init__ passes ds.config. That is a superset of the global one rather than a replacement: ConfigManager.__init__ does self.overrides = datalad.cfg.overrides.copy(), so -c datalad.fusefs.backends=... still wins. The global cfg remains the fallback when no config is supplied.

I left datalad.fusefs.cache-clear alone, since that is outside this PR.

[yarikoptic-gitmate]

Minor (efficiency): when the first backend exhausts its URLs, the next backend calls get_urls() and get_exporttree_urls() again. That repeats the git annex whereis subprocess (batch=False), the two examinekey calls and, on the exporttree path, the boto3 list_object_versions call. Collecting the URLs lazily into a list the first time and replaying it for later backends would avoid that, and the lazy boto3 behaviour would stay the same.

---

Generated by Claude Code

[yarikoptic-gitmate]

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. Added a small replayable() helper: the chained get_urls() + get_exporttree_urls() iterator is pulled at most once, and each backend replays what has already been seen before pulling anything new — so the laziness of the boto3 call is unchanged, but whereis/examinekey/list_object_versions run once per open() rather than once per backend. test_urls_enumerated_once_across_backends pins the call count; test_urls_not_enumerated_when_no_backend_handles pins the laziness.

[read-the-docs-community]

Documentation build overview

📚 datalad-fuse | 🛠️ Build #34979867 | 📁 Comparing b4dbee6 against latest (83a0a30)

  🔍 Preview build  

23 files changed · + 3 added · ± 20 modified

+ Added

• _modules/datalad_fuse/adapter.html
• _modules/datalad_fuse/backends.html
• _modules/datalad_fuse/remfile.html

± Modified

• api.html
• cli.html
• concepts.html
• contributing.html
• genindex.html
• index.html
• installation.html
• python.html
• troubleshooting.html
• tutorial.html
• and 10 more...