[lyrixx]

Fixes #448, refs #430

Thanks a lot @joelwurtz for the investigation and the reproducer in #448!

As explained there, the intermittent builder rebuild of #430 comes from a BuildKit bug, moby/buildkit#6418: when several imported caches describe the same stages, BuildKit only searches one of them for the next steps, the first one to answer. It happens in two cases:

• frontend and builder share the php-base stage, and are built in a single BuildKit session (castor start, castor docker:push)
• a single target imports two caches of the same stages: the registry cache and the GitHub Actions cache added for Intermittent type=registry cache misses in CI (mode=max cache-to) — consider type=gha for CI builds #430

The fix, moby/buildkit#7218, is not merged yet. Meanwhile, this PR:

1. Removes the GitHub Actions cache (docker-compose.ci.yml, the local expose-github-runtime action): it makes things worse instead of fixing them.
2. Builds each service separately, in parallel: castor docker:build (without --service) runs one docker compose build per service, and castor docker:push one bake per service. Each build only imports its own cache. The output of each build is prefixed with the name of its service, so the parallel builds stay readable. The services with a cache_from are built first, then the others: a service without cache sharing their stages (e.g. a worker) would otherwise build these stages from scratch meanwhile, and make them miss their cache (found by @damienalexandre).

Tests

The reproducer of #448 (two targets sharing a stage with a COPY from the build context, each with its own registry cache), with a fresh buildx builder for each run:

Build	Runs	Result
docker buildx bake a b	8	one target rebuilt every time (a 4×, b 4×)
docker compose build (what castor start uses)	8	one target rebuilt every time (a 5×, b 3×)
two separate bakes, in parallel	8	both cached, 8/8

On docker-starter, with this PR: 3 local castor docker:build runs with a fresh builder, importing the registry caches of ghcr.io/jolicode/docker-starter. No RUN step was executed, 12 to 18s. castor docker:push --dry-run prints one bake definition per service.

The measures of #448 on a project of 11 services go the same way: 83s with a single build (builder rebuilt), 53s with one build per service and the registry cache only (everything cached).

Side effect

In the CI of docker-starter, the PHP 8.3 and 8.4 jobs lose their cache: the registry cache is only pushed for PHP 8.5, and the GitHub Actions cache was the only one they had. Projects build a single PHP version, they are not affected.

[damienalexandre]

Validated #450 on a downstream project (4 built services: frontend and builder with a registry cache, worker_messenger and router without). The per-service approach alone was not enough there: the first CI runs still rebuilt the builder layers every time. Inspecting them surfaced one more trigger of the same BuildKit behavior (#6418).

Cache-less services racing the cached ones

worker_messenger shares the php-base stage but has no cache_from. Built in parallel in the same BuildKit daemon, its solve starts resolving php-base from scratch while the frontend and builder solves still import theirs — its divergent local records poison their chain resolution, and the imported cache can no longer match the builder-specific steps.

Reproduced deterministically, without any CI involved:

docker buildx create --use --driver docker-container          # fresh daemon
castor docker:build --service worker_messenger                # builds php-base, no cache import
castor docker:build --service builder                        # imports its cache... and misses the builder steps

The second build imports its manifest but rebuilds from builder 1/11 on. Without the first build, it is a full hit. In our CI the 4 parallel builds lost this race three runs in a row.

Fix downstream: build the services with a cache_from first, the cache-less ones after. The second wave finds the shared stages already resolved and builds nothing (0 executed RUN), so it costs nothing. With that, three consecutive CI runs execute zero RUN step — everything comes from the registry cache alone.

[lyrixx]

Thanks @damienalexandre! Reproduced on docker-starter with worker_messenger enabled: builder was rebuilt 3 runs out of 3 when built in parallel with it, and fully cached 3 runs out of 3 with the services having a cache_from built first. Pushed in the PR.

[damienalexandre]

Should we wait for moby/buildkit#7218 or just go with this solution?

[lyrixx]

I don't know 🤷🏼. What do you think?