Add custom_fingerprint to memoize_path, and a git-annex key fingerprint

[CodyCBakerPhD]

Upstreams the mechanism from dandi/dandi-cli#1941, which otherwise has to reimplement it outside fscacher.

memoize_path keys its cache on a stat() of the path argument. Some resources can vouch for their content better, e.g. locked git-annex'ed files through their content-hash key, which also survives the file being moved or its content dropped; and non-path objects (e.g. dandi-cli's Readables) have no stat() at all.

Change

• memoize_path(..., custom_fingerprint=None): an alternative to the built-in stat() fingerprint. The callable is called with the top-level argument only and returns a fingerprint, or None to fall back to stat(). The rest of the logic is shared: _get_fingerprint() returns a CustomFingerprint or a PathFingerprint (the stat() one with its path, keyed exactly as before, so existing caches stay valid), then the None → direct call, modified-recently window and key injection run once for both. A custom fingerprint has no modified-recently window: it must change whenever the result may.
• Directories: the callable may return a fingerprint of the whole tree; if it returns None, the directory is walked and stat()-ed exactly as on master.
• fscacher.annex_key_fingerprint(path, *, pair_with_path=True): fingerprints a locked annexed file (a symlink into .../annex/objects/.../KEY/KEY, read without calling git-annex) by its key, if the backend hashes the content (SHA*, SHA3_*, SKEIN*, BLAKE2*, MD5, ± E). Unlocked files (whose key goes stale when edited, incl. adjusted branches/Windows), WORM, URL/VURL and external keys fall back to stat(). The key is paired with the path by default. A locked file's cached result is returned even after its content is dropped.
• Docs: a short README section and docstrings; the design decisions and the contract for callables (cheap, return None rather than raise for anything unrecognized, picklable with a stable repr(), equal fingerprints share results), plus edge cases such as Windows/WSL, are in docs/design/git-annex-content.md.

Deferred to follow-ups

• Fingerprinting annexed files inside directories (e.g. Zarrs with dropped chunks), via per-entry fingerprints in the walk or a tree hash of committed directories.
• A way for a callable to say "do not cache" (e.g. a fscacher.NO_CACHE constant); for now an exception from the callable propagates.

Tests

test_annex.py: the helper on a fake annex layout (backends accepted/rejected, pairing, relative paths, non-annexed and non-path values, dropped content), memoize_path end to end on it (twins across paths and extensions with and without pairing, dropped content, stat() fallbacks incl. an edited unlocked file, a directory left to the stat() walk), and against real git-annex when available (locked twins, unlock + edit, drop, WORM). Generic custom-fingerprint tests in test_cache.py. Locally: 50 passed, 2 skipped (no git-annex here); black, isort, flake8 clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

[codecov]

Codecov Report

❌ Patch coverage is 98.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.95%. Comparing base (d4f228b) to head (4f89cd7).

Files with missing lines	Patch %	Lines
src/fscacher/tests/test_annex.py	96.61%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #113      +/-   ##
==========================================
+ Coverage   92.07%   94.95%   +2.88%     
==========================================
  Files           4        6       +2     
  Lines         530      813     +283     
  Branches       40       54      +14     
==========================================
+ Hits          488      772     +284     
  Misses         25       25              
+ Partials       17       16       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[CodyCBakerPhD]

@yarikoptic As you say, mostly waiting on your facelift to fix CI here

[CodyCBakerPhD]

The red checks here are not from this change:

• test (ubuntu-latest, 3.9, py-dev): tox fails installing joblib @ git+https://github.com/joblib/joblib.git under Python 3.9, before any test runs (joblib's main branch no longer supports 3.9).
• benchmark (*): fail the same way on both heads of this PR; the Test workflow is also red on master for [gh-actions](deps): Bump actions/setup-python from 6 to 7 #109–[gh-actions](deps): Bump actions/checkout from 6 to 7 #111.

All other test jobs pass, and codecov reports all new lines covered. Leaving these for the CI facelift mentioned above rather than widening this PR.

---

Generated by Claude Code

[yarikoptic]

facelift was finished, I will rebase now "I""you" ;) merged the master to take into account

[CodyCBakerPhD]

Heads-up: I had just merged master (with #106) into the branch as a8ed903 (clean, tests pass locally). Feel free to rebase over it instead if you prefer linear history; I won't push to the branch in the meantime.

---

Generated by Claude Code

[yarikoptic]

THANK YOU! Overall - looks pretty good! And the main aspect from direct review -- the need to centralize the logic, as here I see the custom fingerprinting to be just an alternative to built in stat based one -- so ideally it should just swap one function with the provided one, and be done, the rest of the code should stay the same.

Also I think it might be worth to immediately develop here the target helper of fingerprinting based on annex keys, since that would "ground" this custom implementation and potentially drive some of its decisions. So, while thinking about that, I recalled that some annex backends should not even be considered; or likely we need to pair with the path (the argument) since otherwise if actual function is file path dependent (not just content; e.g. like smth loading from BIDS dataset from both data file and sidecar .json it deduces) -- we would return wrong cached result.

Here is more on that and related from me+claude:

The only consumer so far is a synthetic Blob, though. I'd rather develop a real fingerprint function together with the hook in this PR. A git-annex key-based one would be the obvious candidate, and it could be tested as part of fscacher and reused by dandi-cli and others. It could go in a follow-up PR, but doing it here is the best way to check that the hook is complete. Thinking it through already raises the points below.

Which keys can vouch for content

• Only content-hash backends (SHA*, SHA3_*, SKEIN*, BLAKE2*, MD5, and their *E variants) pin the content. WORM keys are size + mtime + filename. URL/VURL keys identify a URL whose content may change. External X* backends are unknown. Those should not be trusted, at least not by default.
• Only locked files qualify. For an unlocked file, the key stays the same after the file is edited (git annex lookupkey still reports the old one), so the key would serve stale results. The same goes for adjusted branches and Windows. All of these must fall back to stat().

Contract for the callable
It now runs on every call, including plain str/Path arguments that used to go straight to stat(), so it must be cheap and must return None, not raise, for anything it doesn't recognize. Today lambda x: x.digest makes any call with a plain path fail. The docstring should state this contract.

How it differs from the stat() mode

1. Sharing scope. stat() mode already keys on the dereferenced path. For locked annex files that is the object file, so twins within a repo already share an entry. That holds even across extensions for non-E keys (c.dat and d.nwb stored under the same SHA256 key). Content keys widen the sharing to any location or clone. That's the point, but it's unsafe for functions whose result depends on the path or extension (suffix-dispatched readers, file names in metadata). Whether results are paired with the path should be an explicit choice, with pairing as the default.
2. Missing content. After git annex drop, stat() mode can't fingerprint the broken symlink and calls the function directly. Content mode returns the cached result without the content being present. That's useful, but it is a behavior change and should be documented and tested.
3. The modified-recently window is skipped. That's justified only for locked, hash-based keys.
4. Directories. The hook sees only the top-level argument. For a directory with annexed files (e.g. a .zarr in a dandiset), a custom callable would have to fingerprint the whole tree itself. Otherwise it falls back to _get_dir_fingerprint, which follows symlinks: a single dropped file then disables caching for the whole directory, and annexed files' mtimes still feed the modified-recently window. Per-file content fingerprints inside the directory walk would address this. Even if that comes later, it should inform the shape of the hook now.

Tests

• Run a real function end to end: the listed backends accepted or rejected, locked vs. unlocked (including an edited unlocked file), content present vs. dropped, twins across paths and extensions, and a directory with some files dropped.
• Use a fake annex layout, so the tests run without git-annex, plus real git-annex tests where it's available.

[CodyCBakerPhD]

Addressed the review in c43232e:

• Single flow: custom_fingerprint is now just an alternative to the stat() fingerprint (see the inline reply).
• fscacher.annex_key_fingerprint(path, *, pair_with_path=True), developed here as the real consumer:
  • only reads the symlink (no git-annex call), so it is cheap and returns None for anything else;
  • accepts only locked files (a link into .../annex/objects/.../KEY/KEY) with content-hash backends (SHA*, SHA3_*, SKEIN*, BLAKE2*, MD5, ± E); unlocked files (incl. adjusted branches/Windows), WORM, URL/VURL and X* keys fall back to stat();
  • pairs the key with the path (absolute, as given) by default; pair_with_path=False shares results across locations/clones.
• Contract documented in the docstring and README: called on every call (and per directory entry), must be cheap, must return None rather than raise; no modified-recently window for custom fingerprints; equal fingerprints share results, so include the path unless the result doesn't depend on it.
• Missing content: a locked file's cached result is returned after git annex drop; documented and tested. A file whose content was never present is still called (and fails) every time.
• Directories: the directory walk now also consults the callable per entry, so a tree with dropped annexed files is fingerprinted by their keys (and those entries don't feed the modified-recently window) instead of failing on the broken symlink.
• Tests (test_annex.py): fake annex layout (backends accepted/rejected, pairing, dropped, non-annexed/non-path values; twins across paths and extensions with and without pairing; stat() fallbacks incl. an edited unlocked file; a directory with some files dropped, and changes to it) plus a real git-annex test (locked twins, unlock + edit, drop, WORM), skipped when git-annex is missing. Locally, with git-annex installed: 52 passed, 3 runs.

---

Generated by Claude Code

[CodyCBakerPhD]

facelift was finished, I will rebase now "I""you"

me Clauded-me with auto-watching enabled (Claude settings now forbid such noise)

Only content-hash backends (SHA*, SHA3_, SKEIN, BLAKE2*, MD5, and their *E variants) pin the content.

Yeah that was a baked in assumption that somehow was not made clear through all this

The same goes for adjusted branches and Windows.

Poor, poor Windows

Missing content. After git annex drop, stat() mode can't fingerprint the broken symlink and calls the function directly. Content mode returns the cached result without the content being present. That's useful, but it is a behavior change and should be documented and tested.

Oh, that's kind of nice, didn't think about that! TBH since my main use for this is 100% streaming I wasn't considering how it would work for dynamically fetching/dropping content

The modified-recently window is skipped. That's justified only for locked, hash-based keys.

Yup, as intended

Directories. The hook sees only the top-level argument. For a directory with annexed files (e.g. a .zarr in a dandiset), a custom callable would have to fingerprint the whole tree itself. Otherwise it falls back to _get_dir_fingerprint, which follows symlinks: a single dropped file then disables caching for the whole directory, and annexed files' mtimes still feed the modified-recently window. Per-file content fingerprints inside the directory walk would address this. Even if that comes later, it should inform the shape of the hook now.

Yeah... As per usual, the Zarr side of this will need a good deal more thought

For now early-exiting on detection of that case, can decide more in follow-ups

[CodyCBakerPhD]

This file is for more of those lower-level design concepts for the AI to refer to (a human can perhaps glance and clarify major details)

It also contains thoughts/discussions of edge cases like WSL for reference

(hiding away in commit messages makes it a tad harder to introspect / search IMO)

[yarikoptic]

in general -- I do not mind those. But that is where I feel adhering to some framework like spec-kit is benefitical since keeping such files in sync with the code is part of the process. Otherwise they can get "out of sync"

[CodyCBakerPhD]

what part of spec-kit keeps already-specified docs up-to-date?

[yarikoptic]

the /speckit-analyze:
https://github.github.com/spec-kit/quickstart.html#step-7-speckit-analyze--check-consistency
https://github.com/github/spec-kit/blob/main/templates/commands/analyze.md?plain=1

there is also apparently now a new command - /speckit-converge as well
https://github.github.com/spec-kit/quickstart.html#step-9-speckit-converge--verify-completeness

which relates

[CodyCBakerPhD]

the /speckit-analyze

Sure but that doesn't work automatically 'on its own' and I bet it would work on design docs like these just fine; all one would need is a 'routine' (weekly? monthly? when releases are cut?) to regularly scan the repo with that command

[yarikoptic]

While walking the dog yesterday I came up with ingenious idea for which I did not see an immediate use case here but who knows: there is also Ellipsis, so I thought we could make it explicit

• None -- stops chaining, use original function: this would allow for callbacks to enforce full processing whenever user knows that such paths need to be processed fully (cute idea but I failed to come up immediately with immediate use case, FIFO pipes?)
• Ellipsis -- fall back to stat (kinda -- follow three dots to the next one)

but oh well, if we decide later we need it, we could add that Ellipsis.... but also I guess raising an exception should do the same desired effect of not proceeding to stat?

[CodyCBakerPhD]

Don't think we really need it yet (a FIFO or device file is a problem for plain stat() caching too, not something specific to custom fingerprints in this PR)

On the ellipsis, I prefer explicit over implicit: if ever needed in a follow-up, I'd prefer a named constant, e.g. fscacher.NO_CACHE

Raising doesn't give that effect - an exception from custom_fingerprint propagates, so the decorated call fails.

I'd keep it that way rather than catch it, because such an exception is more likely a bug in the callable, which should fail loudly rather than silently disable caching (IMO)

Then we return None for anything not recognized

[yarikoptic]

the /speckit-analyze:
https://github.github.com/spec-kit/quickstart.html#step-7-speckit-analyze--check-consistency
https://github.com/github/spec-kit/blob/main/templates/commands/analyze.md?plain=1

there is also apparently now a new command - /speckit-converge as well
https://github.github.com/spec-kit/quickstart.html#step-9-speckit-converge--verify-completeness

which relates

[yarikoptic]

unclear to me yet why we need to do this here as presumably this is the part of the original 'mstat' code path, so we wouldn't even get here if custom_fingerprint handles it... might be just a matter of documentation on how custom_fingerprint to interact with folder paths???

[CodyCBakerPhD]

Sure, the fingerprint function is now only called on the top-level argument, not on each entry inside a directory

Per-entry composition or tree-hash fingerprint for committed directories is now fully deferred to a follow-up

[yarikoptic]

overall I think it is good, just thta docs/design/git-annex-content.md might have diverged. make agent go through diff and it to ensure that it reflects the situation (or may be even remove by now? ;-) )

[yarikoptic]

note to paranoid me (smth negging in the back of my mind) -- abspath does not resolve links!

❯ python -c 'import os.path; print(os.path.abspath("sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz"))'
/home/yoh/datalad/dbic/QA/sub-qa/ses-20200102/anat/sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz
❯ dg sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz
get(ok): sub-qa/ses-20200102/anat/sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz (file) [from origin...]                                                            
❯ python -c 'import os.path; print(os.path.abspath("sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz"))'
/home/yoh/datalad/dbic/QA/sub-qa/ses-20200102/anat/sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz
❯ python -c 'import os.path; print(os.path.abspath("./sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz"))'
/home/yoh/datalad/dbic/QA/sub-qa/ses-20200102/anat/sub-qa_ses-20200102_acq-MPRAGE_T1w.nii.gz

[CodyCBakerPhD]

Yeh, many reasons why I personally prefer pathlib, which then has .resolve() as an action to be explicit

[yarikoptic]

need to remember about this fsdecode... not sure I ever used it ...

[yarikoptic]

I think this is wrong -- we do include path by default, don't we?

[CodyCBakerPhD]

the doc was correct, but it didn't say who includes the path...

Now it says fscacher adds the path to the key only for plain stat() fingerprints