ENH: add remfile backend with pluggable backend architecture

CONTRIBUTING.md

@@ -33,6 +33,21 @@ Requires FUSE system libraries (`apt-get install fuse` on Debian/Ubuntu):
 tox -e py3 -- --libfuse
 ```
 
+### Backend tests
+
+Remote files are read by a backend (`fsspec`, always installed, and the
+optional `remfile`).  `remfile` comes from the `full` extra, which tox
+installs only in envs whose name carries `full` or `libfuse`; tests that
+need it are skipped elsewhere via the `requires_remfile` marker.
+
+```bash
+# with remfile (and --libfuse)
+tox -e py314-full
+
+# only the backend tests, in whatever the current env has installed
+tox -e py3 -- datalad_fuse/tests/test_backends.py
+```
+
 ### Forgejo-aneksajo integration tests
 
 These tests start an ephemeral [Forgejo-aneksajo](https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo)

README.md

@@ -5,19 +5,23 @@
 `datalad-fuse` lets you read files of [DataLad](https://www.datalad.org)
 datasets and [git-annex](https://git-annex.branchable.com) repositories
 without downloading them first: only the parts of files that are actually
-read are fetched, via [fsspec](https://filesystem-spec.readthedocs.io), from
-the URLs that git-annex knows.  Unlike `datalad get`, which downloads whole
-files before you can use them, this pays off when you need only parts of
-large files, e.g. a few arrays from NWB/HDF5 files or the headers of many
-images.  Use it through a FUSE mount, so that any program can open the files,
-or directly from Python.
+read are fetched, from the URLs that git-annex knows, via
+[fsspec](https://filesystem-spec.readthedocs.io) or — for NWB/HDF5 files, if
+installed — [remfile](https://github.com/flatironinstitute/remfile).  Unlike
+`datalad get`, which downloads whole files before you can use them, this pays
+off when you need only parts of large files, e.g. a few arrays from NWB/HDF5
+files or the headers of many images.  Use it through a FUSE mount, so that
+any program can open the files, or directly from Python.
 
 **Documentation: https://datalad-fuse.readthedocs.io**
 
 ## Installation
 
     python3 -m pip install datalad-fuse
 
+Add the `remfile` backend, which is faster for NWB/HDF5 files, with
+`python3 -m pip install "datalad-fuse[remfile]"`.
+
 [git-annex](https://git-annex.branchable.com/install/) is required, and FUSE
 for mounting datasets (e.g. `sudo apt-get install fuse3 libfuse2t64` on
 Ubuntu 24.04).  See the
@@ -52,7 +56,7 @@ from contextlib import closing
 import h5py
 import pynwb
 
-from datalad_fuse.fsspec import DatasetAdapter
+from datalad_fuse.adapter import DatasetAdapter
 
 # caching=False: keep fetched data in memory only
 with closing(DatasetAdapter("000582", caching=False)) as dsa:

datalad_fuse/__init__.py

@@ -14,7 +14,7 @@
 )
 from datalad.interface.base import Interface, build_doc, eval_results
 from datalad.interface.results import get_status_dict
-from datalad.support.constraints import EnsureNone
+from datalad.support.constraints import EnsureNone, EnsureStr
 from datalad.support.param import Parameter
 
 from ._version import get_versions
@@ -94,7 +94,17 @@ class FuseFS(Interface):
             args=("--caching",),
             choices=["none", "ondisk"],
             default="none",
-            doc="Whether to cache fsspec'ed files on disk on not at all",
+            doc="Whether to cache data fetched from remote URLs on disk or not at all",
+        ),
+        "backends": Parameter(
+            args=("--backends",),
+            doc=(
+                "Comma-separated list of backends to try for remote file"
+                " access, in priority order.  Available: remfile, fsspec."
+                "  Default: remfile,fsspec (remfile for HDF5 files,"
+                " fsspec for everything else)"
+            ),
+            constraints=EnsureStr() | EnsureNone(),
         ),
         # TODO: (might better become config vars?)
         # --cache=persist
@@ -112,6 +122,7 @@ def __call__(
         mode_transparent: bool = False,
         allow_other: bool = False,
         caching: str | None = None,
+        backends: str | None = None,
     ) -> Iterator[Dict[str, Any]]:
         from fuse import FUSE
 
@@ -133,6 +144,7 @@ def __call__(
                 ds.path,
                 mode_transparent=mode_transparent,
                 caching=caching == "ondisk",
+                backends=backends,
             ),
             mount_path,
             foreground=foreground,