[WIP] support for cache volume mounts from host

[sipsma]

As mentioned here, moving this from POC to fleshed out implementation ended up calling for a very different approach.

Will flesh out description more, but high level:

• Needed to fork buildkit; corresponding PR for dagger's buildkit fork is here [WIP) volume snapshotter support buildkit#3
  • Trying to keep modifications there minimally invasive and mostly plumbing-only
• Cache volumes are implemented via a custom snapshotter (which lives in this dagger repo)
  • Makes it much easier to reuse existing buildkit code without special casing
  • Allows the implementation to be shareable across concurrent engines (by avoiding use of boltdb for metadata, instead we use flat files, flock and some careful maneuvering with atomic renames and such)

Currently passing tests other than a few expected failures from tests for cache volume ownership/mode, which is a TODO below.

---

TODOs:

• Add support for pruning just cache volumes (including specific ones) to the API added in core: add API for local engine cache and integ tests #7767
• Way more tests (need to backfill coverage + add more, including sharing of volumes between separate engines)
• Support for sharing mode private
• Support for base layers underneath cache mount (which also gets us support for permissions+mode)
• Double-check against cloud cache.
  • New implementation is compatible with it now in theory, but need to test to confirm and also think through whether it even makes sense to have both working together.

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.

[sipsma]

(finishing up #7804 first, then getting back to this)

[sipsma]

Update on this:

• Been struggling with the split responsibilities between buildkit + dagger that come along with a fleshed out implementation of this
• Tried to just layer on top of buildkit but proving to not be tenable, so for now going with a fork of buildkit that includes all the customizations we need here
• After trying a few things there, think I've honed in on a good approach:
  1. Support for these special "mutable-only, shareable" mounts will be a custom Snapshotter implementation (i.e. the interface from containerd).
     • This allows us to enforce all the properties we want in a mostly self-contained place rather than leaking too much all over. Also potentially more re-usable in the future.
     • E.g. we can enforce the mutability, re-use based on key, locking, etc. there. We can also store the metadata however we want, so we won't be tied to a non-shareable boltdb implementation while still re-using most of the existing buildkit+dagger machinery
  2. For pruning, cache mounts stored on the host will be able to be automatically pruned after all (different than I originally thought), but I will update our default pruning policy to exclude them so out-of-the-box users will need to manually prune them
     • This requires fleshing out the pruning API added in core: add API for local engine cache and integ tests #7767 to support pruning specific cache refs (which in turn is not handled by buildkit's pruning implementation yet and will require some changes)

EDIT: Got the above idea basically working, updated PR description

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.

[bitgandtter]

Has something been done in this direction? I have been following the PR for a long time, and the GH bot has more interactions than the code

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.