Skip to content

backup: Reduce the number of GetBlobs. - #503

Open
mathieu-plak wants to merge 1 commit into
mainfrom
mm/reduce_getblobs
Open

backup: Reduce the number of GetBlobs.#503
mathieu-plak wants to merge 1 commit into
mainfrom
mm/reduce_getblobs

Conversation

@mathieu-plak

Copy link
Copy Markdown
Contributor
  • When getting an object out of the cache it's not needed to recheck that it exists in the repo. This used to be true because the vfs cache was local and so it could be lying (repo changed due to another user). Now that we base the vfs cache on a previous snapshot we know by definition that the object exists and so BlobExists will always be true.

  • BlobExists is a "slow" function and is the hottest path of them all, reducing its usage is always a win.

  • On a non changing backup this divides by 4 the number of calls we emit to the state cache.

  • This is a scary diff, I've been sitting on it for a while, so this needs to get some extra considerations from reviewers.

* When getting an object out of the cache it's not needed to recheck
  that it exists in the repo. This used to be true because the vfs cache
  was local and so it could be lying (repo changed due to another user).
  Now that we base the vfs cache on a previous snapshot we know _by
  definition_ that  the object exists and so BlobExists will always be
  true.

* BlobExists is a "slow" function and is the hottest path of them all,
  reducing its usage is always a win.

* On a non changing backup this divides by 4 the number of calls we emit
  to the state cache.

* This is a scary diff, I've been sitting on it for a while, so this
  needs to get some extra considerations from reviewers.
Copilot AI review requested due to automatic review settings July 24, 2026 10:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces repository BlobExists lookups on the backup hot path by trusting the VFS cache (seeded from a previous snapshot) and reusing cached object/entry MACs without re-validating their presence in the repository.

Changes:

  • Removed BlobExists(RT_OBJECT, ...) checks when reusing cached content objects.
  • Removed BlobExists(RT_VFS_ENTRY, ...) checks when reusing cached VFS directory/file entry MACs.
  • Keeps the rest of the backup pipeline (scanlog -> indexes -> persist) unchanged, relying on cached blobs already being present.
Comments suppressed due to low confidence (2)

snapshot/backup.go:861

  • writeDirectoryEntry now reuses cachedPath.MAC without verifying that the referenced RT_VFS_ENTRY blob exists in the destination repository. This removes the only safety net when the VFS cache is accidentally sourced from a different repo (or otherwise not guaranteed to be present), which can persist a VFS btree pointing at missing entry blobs.
	if cachedPath != nil {
		dirEntryMAC = cachedPath.MAC
		serialized, err := dirEntry.ToBytes()
		if err != nil {
			return err

snapshot/backup.go:902

  • writeFileEntry now trusts cachedPath.MAC (RT_VFS_ENTRY) on cache hits without confirming it exists in the destination repository. If the VFS cache comes from another repository (possible because WithVFSCache does not validate) this will record file entries whose VFS_ENTRY blobs are missing, yielding a corrupted snapshot.
	if cachedPath != nil {
		fileEntryMAC = cachedPath.MAC
		if fileEntry.Object == (objects.MAC{}) && cachedPath.ObjectMAC != (objects.MAC{}) {
			fileEntry.Object = cachedPath.ObjectMAC
		}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread snapshot/backup.go
Comment on lines 826 to +830
if cachedPath != nil && cachedPath.ObjectMAC != (objects.MAC{}) {
if snap.repository.BlobExists(resources.RT_OBJECT, cachedPath.ObjectMAC) {
return &contentMeta{
ObjectMAC: cachedPath.ObjectMAC,
Size: cachedPath.FileInfo.Size(),
Chunks: cachedPath.Chunks,
Entropy: cachedPath.Entropy,
ContentType: cachedPath.ContentType,
}, nil
}
return &contentMeta{
ObjectMAC: cachedPath.ObjectMAC,
Size: cachedPath.FileInfo.Size(),
Chunks: cachedPath.Chunks,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that would require a mac colision -_-. Though as kloset is a library maybe we want to add a small assertion in WithVFSCache that the repos UUID match. But to be 100% clear this can't happen without a hash collision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants