Shared physical extents (APFS clones, btrfs/XFS reflinks) are counted once per file

[laurigates]

Summary

Files that share physical blocks — APFS clones (cp -c, clonefile(2)) on macOS, and btrfs/XFS reflinks (cp --reflink) on Linux — are counted at full size once per file. On systems where copy-on-write copies are common this inflates totals substantially.

This is the same problem reported in #83 (2020). That thread stalled on an approach that can't work, and the issue is marked closed, but it still reproduces on v1.2.4. I've since found a syscall that does expose the information, so I wanted to reopen the topic with evidence and a concrete design — and to ask whether you'd want it before anyone writes code.

Reproduction (dust 1.2.4, macOS 26.5.1, APFS)

$ dd if=/dev/urandom of=big.bin bs=1m count=200

$ df -h .                       # before cloning
  used=571Gi  avail=323Gi

$ for i in 1 2 3 4 5; do cp -c big.bin "clone$i.bin"; done

$ df -h .                       # after 5 clones — unchanged
  used=571Gi  avail=323Gi

$ du -sh .
1.2G    .

$ dust -d1 .
 200M   ┌── big.bin   │█████████                                           │  17%
 200M   ├── clone1.bin│█████████                                           │  17%
 200M   ├── clone2.bin│█████████                                           │  17%
 200M   ├── clone3.bin│█████████                                           │  17%
 200M   ├── clone4.bin│█████████                                           │  17%
 200M   ├── clone5.bin│█████████                                           │  17%
 1.2G ┌─┴ .           │███████████████████████████████████████████████████ │ 100%

Ground truth is 200 MB. du agrees with dust at 1.2G, so dust is not doing anything wrong relative to its inputs — st_blocks genuinely reports full allocated size for every clone.

This isn't a synthetic concern. uv (Python) and bun (JS) both materialise their global package caches into per-project .venv / node_modules using clonefile on macOS. I measured a second project's install costing 0 bytes of real disk while du/dust billed it at full size. On a machine with many projects the overstatement compounds — it's what sent me down this path.

Why #83 couldn't have worked

That thread tried to find the signal in stat() / MetadataExt. It isn't there. A clone has a distinct inode and reports full st_blocks — it is indistinguishable from an independent copy via stat:

file	inode	st_blocks
orig.bin	167359983	full
clone.bin	167359986	full
copy.bin (real copy)	167359987	full

What does work: physical extents

fcntl(fd, F_LOG2PHYS_EXT, …) (declared in sys/fcntl.h, fcntl 65) maps a logical offset to a device offset. Clones share physical extents; real copies don't:

orig.bin    extents: [phys=369673572352 len=9437184] [phys=668907188224 len=8388608] [phys=787040804864 len=3145728]
clone.bin   extents: [phys=369673572352 len=9437184] [phys=668907188224 len=8388608] [phys=787040804864 len=3145728]   ← identical
copy.bin    extents: [phys=362318385152 len=20971520]                                                                  ← different

So the dedup key is a set of (device_offset, length) extents rather than (ino, dev).

The Linux analogue is FIEMAP, which exposes the same physical extents for btrfs/XFS reflinks and is what filefrag and btrfs compsize use. I have only verified the macOS side empirically — I have no btrfs system to test on — but if the two backends line up, this stops being a macOS feature and becomes generic shared-extent accounting that also fixes cp --reflink on Linux. That seems like the version worth having, and I mention it because #83 foundered partly on you not having a Mac to test with.

Where it would slot in

clean_inodes() in dir_walker.rs already does exactly the right shape of work: a post-walk pass holding a global HashSet<(u64,u64)>, iterating in deterministic inode-sorted order, dropping already-seen entries and recomputing node sizes bottom-up. Shared-extent accounting is the same operation with a different key — and because that pass already recomputes sizes, it can subtract partially-shared extents rather than only dropping whole files (a clone that's been partially written to owns some extents and shares others). platform.rs::get_metadata is already cfg-gated per OS, so the extent read has an obvious home.

Cost — this would have to be opt-in

Reading extents needs an open() per file plus one fcntl per extent; dust currently gets by with lstat from the walk. Measured on this machine:

tree	lstat only (today)	open + fcntl	ratio
~/.cargo/registry (87 k files)	0.42 s	2.61 s	6.2×
~/.cache/uv (1.26 M files)	17.7 s	51.4 s	2.9×

Given that speed is dust's whole point, I'd assume this belongs behind a flag (--shared-blocks?) and stays off by default.

Memory looks fine: extents came to ≈1.02 per file (1 288 215 extents across 1 263 593 files), so the dedup set is comparable to the existing inode set.

Two cases need a graceful fallback to st_blocks:

• files that can't be opened (permissions)
• the sealed system volume — /usr/bin returns ENOTSUP. I checked whether that was file compression; it isn't, it's the read-only sealed snapshot. Data-volume files work fine, and immutable system files aren't where user clones live anyway.

The question

Is this in scope? I noticed #309 (btrfs compsize) was closed as out of scope, and I don't want to conflate the two — that request was about compression ratios, which is genuinely a different feature. This is narrower: dust already promises "disk usage" and already dedupes hardlinks to avoid double-counting; shared extents are arguably the same correctness concern reached by a different mechanism.

If you'd want it, I'm happy to implement the macOS backend and the generic plumbing, behind an off-by-default flag. I'd need help or CI for the btrfs side since I can't test it. If you'd rather not carry per-filesystem code, that's a completely reasonable call and it'd be useful to say so on #83 too, since that one currently reads as fixed when it isn't.