`git annex test` fails on NFS: exact high-resolution mtime comparison in linkAnnex / prepSendAnnex

[yarikoptic-gitmate]

Tracking issue for the nightly test-annex (nfs-home, ubuntu-24.04) failure, root-caused with the instrumented harness in #293. Nothing here needs a patch in this repository; it is a git-annex upstream matter plus a workaround for NFS users.

Symptom

test-annex (nfs-home) fails on roughly two nightlies out of three, in a different test each night, and is usually the only failing job in the run. The messages alternate between:

• <file> failed to link to annex — git annex add of an unlocked file
• git-annex: unlock failed — git annex unlock
• Exception: content changed while it was being sent — testremote transfers

Root cause

All three are the same assumption. git-annex records a file's InodeCache — (inode, size, high-resolution mtime) — copies the file, stats it again, and compares the two with compareStrong, i.e. exact equality including the nanosecond field. If they differ it concludes the file changed underneath, deletes the destination and fails.

On NFS a client can return two different mtimes for a file nothing wrote to. Instrumented runs caught it four times, always with inode and size unchanged and only the mtime moving:

file	mtime before → after	delta
worktree file (import)	1790084329.078602968 → .090671216	+12.07 ms
worktree file (add subdirs)	1790178434.030902619 → .309979362	+279 ms
annex object (edit)	1790084752.023778495 → 1790084806.769318488	+54.75 s
annex object (conversion annexed to git)	1790178735.562779131 → 1790178776.778504714	+41.22 s

The sub-second ones are the client's value being replaced by the server's within the same second; the tens-of-seconds ones are a cached attribute going stale (default acregmax is 60 s). In the object cases nothing in the test writes to the file at all.

Two call sites are affected, so this is not confined to one command:

• Annex/Content.hs:646 — checksrcunchanged in linkAnnex → failed to link to annex / unlock failed
• Annex/Content.hs:730 — sameInodeCache in prepSendAnnex → content changed while it was being sent

compareInodeCaches only relaxes to compareWeak (2-second tolerance) when the inode sentinal says inodes changed, which never happens here, so the comparison stays exact.

Confirmed by experiment

Run 35882579073 ran the nfs-home flavor three ways, three times each, same build and same runner batch:

arm	mount	result
default	as the nightlies mount	3/3 failed
actimeo0	-o actimeo=0	3/3 passed
noac	-o noac	3/3 passed

actimeo=0 disables attribute caching and changes nothing else, so this is the attribute cache, not the write path. Cost is roughly 4×: a passing default job did the same 26 groups in 8m20s against 32m01s with actimeo=0.

Workaround for NFS users: mount with actimeo=0. noac also works but additionally forces synchronous writes and is not needed.

Is it a regression?

Not in the git-annex code that fails — in the CI environment or in timing.

• test-annex (nfs-home) passed on 2026-03-13 (run 23036066273, where only test-datalad failed) and was failing by 2026-08-08 (run 31238024732). Most June–July nightlies failed at build-package, so the suite did not run and the transition is masked in the GitHub-side logs; the tinuous archive has those runs if anyone wants to bisect the date exactly.
• On the git-annex side nothing moved: since 2026-01-01 exactly one commit touched Annex/Content*.hs, Utility/InodeCache.hs, Annex/InodeSentinal.hs or Utility/CopyFile.hs — 95520e391 "generalize type", which only changes downloadUrl. No commit has touched the compareStrong usage since at least 2025-01-01, and both the strict comparison and the high-resolution mtime predate 2022-08 (as far back as this repo's upstream/master mirror can be deepened).

So the code has assumed this for years; what changed is how often the NFS client actually violates it on the runners.

Reproducing

Two scripts, proposed for eval-under in con/eval-under#11:

• mtime-stability — write, stat, copy as git-annex copies, stat again, compare; reports a rate, no git-annex involved. This is the minimal reproducer: the failure is visible with four lines of Python.
• git-annex-linkannex — loops unlock and unlocked add, reporting a failure rate in minutes rather than a pass/fail of the whole suite.

That PR also adds eval-under nfs --mount-opts, without which actimeo=0 could not be tested locally at all.

Suggested next step

Report upstream at git-annex.branchable.com: exact equality on a high-resolution mtime is not a safe assumption on NFS, at both checksrcunchanged and sameInodeCache, with the numbers above and the actimeo=0 control. A fix confined to linkAnnex would leave transfers failing.

[yarikoptic-gitmate]

It is the runner's kernel: 6.8 passes, 6.17 fails. Export mode is irrelevant

Run 36491917512 — nfs-home, 2 runner images × 2 export modes × 2 reps, one build shared by all 8 jobs (10.20260901+git71-g0b0bd01297, the OsPath build):

runner	export	rep 1	rep 2
ubuntu-22.04	rw (sync)	✅ pass	✅ pass
ubuntu-22.04	rw,async	✅ pass	✅ pass
ubuntu-24.04	rw (sync)	❌ fail	❌ fail
ubuntu-24.04	rw,async	❌ fail	❌ fail

4/4 on each side. sync vs async on the export changes nothing; the runner image decides it. What differs, as each job now logs:

	ubuntu-22.04 (passes)	ubuntu-24.04 (fails)
kernel	6.8.0-1064-azure	6.17.0-1022-azure
nfs-utils	1:2.6.1-1ubuntu1.2	1:2.6.4-3ubuntu5.1
export line	/tmp/nfsmount_ localhost(rw)	identical
mount options	…,hard,proto=tcp6,…	…,hard,fatal_neterrors=none,proto=tcp6,…
git-annex	10.20260901+git71-g0b0bd01297	identical
outcome	all 26 groups, 7m31s	10 FAILs: import, lock, and storeKey/present True in testremote

fatal_neterrors is a client option only the newer kernel advertises — the mount option lists are otherwise character-identical, so the NFS client is the only thing that moved.

The failing arm reproduced the same mechanism, again mtime-only with inode and size unchanged:

unlock foo
  DIAGNOSTIC: linkAnnex: inode cache of .git/annex/objects/Kj/0x/SHA256E-s20--e394…/SHA256E-s20--e394…
  changed while copying it; before: 9446136 20 1790635113 563914024; after: 9446136 20 1790635153 0176511
  git-annex: unlock failed

inode 9446136 and size 20 both unchanged; mtime +39.44 s — another stale cached attribute, against the 60 s default acregmax.

This also answers "why did it pass before?"

Nothing in git-annex changed. test-annex moved to ubuntu-24.04 in 4cf75f8d8 (2026-01-03) and still passed on 2026-03-13, because the 24.04 image then carried a 6.8-series kernel — the same series ubuntu-22.04 still runs today. GitHub has since advanced that image to 6.17, and the newer NFS client returns a changed high-resolution mtime for an untouched file often enough to break git-annex's exact compareStrong comparison. A years-old assumption, a moving client kernel.

Options, in order of cost

1. Mount nfs-home with actimeo=0 — proven fix on 24.04 (run 35882579073, 3/3 → 0/3), at roughly 4× the wall clock. Keeps testing a current kernel, which is the point of the flavor.
2. Pin the nfs-home cell to ubuntu-22.04 — green and fast, but then the flavor stops exercising a modern NFS client, and this failure would go unnoticed for the users who will hit it.
3. Leave it red and treat it as the upstream report's evidence.

My preference is 1: it keeps the coverage honest and makes the job green, and the ~4× cost lands on one cell.

Either way the upstream report gets stronger: the trigger is a stock NFS client on a current kernel, not an exotic configuration.

---

Generated by Claude Code