ONBUILD executed in multi-stage build grandchild

[MikeMcC399]

Observed Behavior

If a docker build is executed using the Docker image cypress/factory in a two-stage Dockerfile build, with the recommended

# syntax=docker/dockerfile:1

from Custom Dockerfile syntax then the ONBUILD RUN instructions from cypress/factory are executed correctly in the first build stage, and fail in the second build stage. The ONBUILD instructions can only be executed once, since they clean up after themselves and they delete resources which are no longer supposed to be necessary.

cypress/factory is built with factory/factory.Dockerfile and this includes:

# Global Cleanup
ONBUILD RUN apt-get purge -y --auto-remove \
    curl \
    bzip2 \
    gnupg \
    dirmngr\
  && rm -rf /usr/share/doc \
  && rm -rf /var/lib/apt/lists/* \
  # Remove cypress install scripts
  && rm -rf /opt/installScripts

This is a regression in docker/dockerfile:1.11.0 and above:

docker/dockerfile version	Result
undefined	success
1	fail
1.10.0	success
1.11.0	fail
1.11.1	fail
1.12.0	fail

Expected Behavior

The set of ONBUILD instructions from an existing Docker file should only be used in the first stage of a multi-stage build, not in the second or any following stages.

Steps to reproduce

See https://github.com/MikeMcC399/cy-docker-multistage

On Ubuntu 24.04.1 LTS, Node.js v22.12.0 LTS, Docker Desktop 4.36.0

git clone https://github.com/MikeMcC399/cy-docker-multistage
cd cy-docker-multistage
npm ci
docker build . -f Dockerfile.1 -t cy-test --no-cache

using Dockerfile.1:

# syntax=docker/dockerfile:1
FROM cypress/factory AS first

COPY . /opt/app
WORKDIR /opt/app
RUN npm install cypress --save-dev
RUN npx cypress install

FROM first AS second

RUN apt-get update
RUN apt-get install chromium -y

Logs

$ docker build . -f Dockerfile.1 -t cy-test
[+] Building 94.7s (20/28)                                                                                                                                                                         docker:default
 => [internal] load build definition from Dockerfile.1                                                                                                                                                       0.1s
 => => transferring dockerfile: 267B                                                                                                                                                                         0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                                                                                                                    1.8s
 => docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5                                                                                     2.3s
 => => resolve docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5                                                                                         0.0s
 => => sha256:61782c2b067b8aa6d6a5a44dda352a26cc0bcb2dcf01b45b6eea6fca5a518bba 850B / 850B                                                                                                                   0.0s
 => => sha256:b968edff90bd031526591b03dfbc60323746c2d57ae359d0bff26f2e971b0094 1.26kB / 1.26kB                                                                                                               0.0s
 => => sha256:dae058f2fffd4fce12a0791304a7160b8e92593dd7bb824d0c2aa12f346f9f2b 12.78MB / 12.78MB                                                                                                             2.0s
 => => sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5 8.40kB / 8.40kB                                                                                                               0.0s
 => => extracting sha256:dae058f2fffd4fce12a0791304a7160b8e92593dd7bb824d0c2aa12f346f9f2b                                                                                                                    0.1s
 => [internal] load build definition from Dockerfile.1                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/cypress/factory:latest                                                                                                                                            1.3s
 => [internal] load .dockerignore                                                                                                                                                                            0.1s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => [first 1/5] FROM docker.io/cypress/factory:latest@sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450                                                                               31.6s
 => => resolve docker.io/cypress/factory:latest@sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450                                                                                      0.0s
 => => sha256:bc0965b23a04fe7f2d9fb20f597008fcf89891de1c705ffc1c80483a1f098e4f 28.23MB / 28.23MB                                                                                                             4.2s
 => => sha256:7e0a9cd7b743bea5b7e3c15257e5a44693f6b7adcfdd21b42349842ca82e5fec 163.09MB / 163.09MB                                                                                                          22.2s
 => => sha256:3dac0345d6caafaef6a8d9e008355a1ce7c6f48919891b30c8042ce22675f903 4.66kB / 4.66kB                                                                                                               0.8s
 => => sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450 683B / 683B                                                                                                                   0.0s
 => => sha256:7ba1dee0d47c173e3b04bd88b44062d513adb02b022b63ffe8b181815e68d044 899B / 899B                                                                                                                   0.0s
 => => sha256:2a6063523a72719fcd3ff210bf3f88d5b722caa38e79dffb5015261e775b2b26 3.09kB / 3.09kB                                                                                                               0.0s
 => => extracting sha256:bc0965b23a04fe7f2d9fb20f597008fcf89891de1c705ffc1c80483a1f098e4f                                                                                                                    2.2s
 => => extracting sha256:7e0a9cd7b743bea5b7e3c15257e5a44693f6b7adcfdd21b42349842ca82e5fec                                                                                                                    8.9s
 => => extracting sha256:3dac0345d6caafaef6a8d9e008355a1ce7c6f48919891b30c8042ce22675f903                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                            1.1s
 => => transferring context: 26.98MB                                                                                                                                                                         1.0s
 => [first  2/20] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0                                                                                                                 16.5s
 => [first  3/20] ONBUILD RUN node /opt/installScripts/yarn/install-yarn-version.js ${YARN_VERSION}                                                                                                          0.4s
 => [first  4/20] ONBUILD RUN node /opt/installScripts/chrome/install-chrome-version.js ${CHROME_VERSION}                                                                                                    0.4s
 => [first  5/20] ONBUILD RUN node /opt/installScripts/edge/install-edge-version.js ${EDGE_VERSION}                                                                                                          0.5s
 => [first  6/20] ONBUILD RUN node /opt/installScripts/firefox/install-firefox-version.js ${FIREFOX_VERSION}                                                                                                 0.5s
 => [first  7/20] ONBUILD RUN node /opt/installScripts/cypress/install-cypress-version.js ${CYPRESS_VERSION}                                                                                                 0.6s
 => [first  8/20] ONBUILD RUN apt-get purge -y --auto-remove     curl     bzip2     gnupg     dirmngr  && rm -rf /usr/share/doc   && rm -rf /var/lib/apt/lists/*   && rm -rf /opt/installScripts             1.6s
 => [first  9/20] COPY . /opt/app                                                                                                                                                                            0.6s
 => [first 10/20] WORKDIR /opt/app                                                                                                                                                                           0.0s
 => [first 11/20] RUN npm install cypress --save-dev                                                                                                                                                         1.6s
 => [first 12/20] RUN npx cypress install                                                                                                                                                                   34.1s
 => ERROR [second  1/17] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0                                                                                                           0.3s
------
 > [second  1/17] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0:
0.289 bash: /opt/installScripts/node/install-node-version.sh: No such file or directory
------
Dockerfile.1:9
--------------------
   7 |     RUN npx cypress install
   8 |
   9 | >>> FROM first AS second
  10 |
  11 |     RUN apt-get update
--------------------
ERROR: failed to solve: process "/bin/sh -c bash /opt/installScripts/node/install-node-version.sh ${APPLIED_FACTORY_DEFAULT_NODE_VERSION}" did not complete successfully: exit code: 127

Workaround

Pin syntax to 1.10.0 in Dockerfile.1.10.0:

# syntax=docker/dockerfile:1.10.0

To test workaround, as above, except:

docker build . -f Dockerfile.1.10.0 -t cy-test --no-cache # pin to older version
npm test                                        # run Cypress test

Reference

Dockerfile ONBUILD

Triggers are cleared from the final image after being executed. In other words they aren't inherited by "grand-children" builds.

[ap-wtioit]

I'm also seeing a similar issue with https://github.com/Tecnativa/doodba where groupadd and useradd can only be done once

[MikeMcC399]

I added a simpler reproduction to https://github.com/MikeMcC399/cy-docker-multistage which does not rely on cypress/factory:

git clone https://github.com/MikeMcC399/cy-docker-multistage
cd cy-docker-multistage
docker build . -f Dockerfile.parent -t parent --no-cache
docker build . -f Dockerfile.child-grandchild -t child-grandchild --no-cache

Dockerfile.parent

# syntax=docker/dockerfile:1
FROM debian:12-slim AS parent

RUN echo "parent"
ONBUILD RUN echo "parent to child"

Build Docker image parent:

$ docker build . -f Dockerfile.parent -t parent --no-cache
[+] Building 2.1s (9/9) FINISHED                                                                         docker:default
 => [internal] load build definition from Dockerfile.parent                                                        0.0s
 => => transferring dockerfile: 157B                                                                               0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                          1.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e7  0.0s
 => [internal] load build definition from Dockerfile.parent                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:12-slim                                                  0.5s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => CACHED [1/2] FROM docker.io/library/debian:12-slim@sha256:1537a6a1cbc4b4fd401da800ee9480207e7dc1f23560c21259f  0.0s
 => [2/2] RUN echo "parent"                                                                                        0.3s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:119f68760291ec6495317ed8e1b072b2bc6cd0f74931b70de40541c523185c14                       0.0s
 => => naming to docker.io/library/parent                                                                          0.0s

Dockerfile.child-grandchild

# syntax=docker/dockerfile:1
FROM parent AS child

RUN echo "child"

FROM child AS grandchild

RUN echo "grandchild"

Build Docker image child-grandchild from parent:

$ docker build . -f Dockerfile.child-grandchild -t child-grandchild --no-cache
[+] Building 2.7s (12/12) FINISHED                                                                       docker:default
 => [internal] load build definition from Dockerfile.child-grandchild                                              0.0s
 => => transferring dockerfile: 171B                                                                               0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                          0.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e7  0.0s
 => [internal] load build definition from Dockerfile.child-grandchild                                              0.0s
 => [internal] load metadata for docker.io/library/parent:latest                                                   0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [child 1/2] FROM docker.io/library/parent:latest                                                               0.0s
 => [child 2/3] ONBUILD RUN echo "parent to child"                                                                 0.3s
 => [child 3/3] RUN echo "child"                                                                                   0.5s
 => [grandchild 1/2] ONBUILD RUN echo "parent to child"                                                            0.6s
 => [grandchild 2/2] RUN echo "grandchild"                                                                         0.5s
 => exporting to image                                                                                             0.1s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:cfa55e2c974b1b397e904ff10a8bb0c7b60c9d00287f73d619680ed80bec24a1                       0.0s
 => => naming to docker.io/library/child-grandchild                                                                0.0s

The following log line from above should not appear. This ONBUILD RUN instruction should only be run in the child, not the grandchild:

 => [grandchild 1/2] ONBUILD RUN echo "parent to child"

[ap-wtioit]

My current "simple" reproduction is:
Dockerfile.onbuild

FROM ubuntu AS onbuild
RUN touch /history.txt
ONBUILD ARG also_used_later=""
ONBUILD RUN echo "$(date) onbuild also_used_later: $also_used_later" >> /history.txt
ONBUILD RUN dd if=/dev/random count=$((1024*2)) > /random.bin
ONBUILD RUN sha256sum /random.bin >> /history.txt
ONBUILD ARG build_date
ONBUILD LABEL my_label=$build_date

Dockerfile.use_onbuild

ARG BASE_IMAGE
FROM $BASE_IMAGE AS first
RUN echo "$(date) first also_used_later: $also_used_later" >> /history.txt
FROM first AS second
RUN echo "$(date) second" >> /history.txt
FROM first AS clean
RUN rm /history.txt /random.bin
FROM scratch AS data
COPY --from=clean / /
COPY --from=first /history.txt /history-first.txt
COPY --from=second /history.txt /history-second.txt
ARG also_used_later
ENV also_used_later="$also_used_later"
RUN grep -RH '.' /history*.txt

# with clean build caches (docker system prune)
docker build -f Dockerfile.onbuild -t onbuild_image .
docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image .
docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image --build-arg "also_used_later=true" .

When runnning with DOCKER_BUILDKIT=1 BUILDKIT_PROGRESS=plain i get the following output from the grep (dumping the created history.txt) of the third image build (building a variant of the image in the second step):

#17 [data 4/4] RUN grep -RH '.' /history*.txt
#17 0.116 /history-first.txt:Wed Dec 11 14:41:26 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-first.txt:1ffad5e071a575ea9a87654ea03c6e0c8a4f312d79c8eeb7738de8722aafa96e  /random.bin
#17 0.116 /history-first.txt:Wed Dec 11 14:41:27 UTC 2024 first also_used_later: true
#17 0.116 /history-second.txt:Wed Dec 11 14:41:26 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-second.txt:1ffad5e071a575ea9a87654ea03c6e0c8a4f312d79c8eeb7738de8722aafa96e  /random.bin
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 first also_used_later: true
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-second.txt:a962687565ca6a6e2f1d7a948805c6dad30db692c293a1bd4cfcbcebb3680d26  /random.bin
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 second

When running with DOCKER_BUILDKIT=0 the following result is achived

Step 14/14 : RUN grep -RH '.' /history*.txt
 ---> Running in d88e97868486
/history-first.txt:Wed Dec 11 14:41:08 UTC 2024 onbuild also_used_later: true
/history-first.txt:e142fb0793d6ea24f79f959f837ac30ca98ffb79157c203df08a65a5d0095b38  /random.bin
/history-first.txt:Wed Dec 11 14:41:09 UTC 2024 first also_used_later: true
/history-second.txt:Wed Dec 11 14:41:08 UTC 2024 onbuild also_used_later: true
/history-second.txt:e142fb0793d6ea24f79f959f837ac30ca98ffb79157c203df08a65a5d0095b38  /random.bin
/history-second.txt:Wed Dec 11 14:41:09 UTC 2024 first also_used_later: true
/history-second.txt:Wed Dec 11 14:41:09 UTC 2024 second

for us it does not depend on the dockerfile syntax, but we discovered it when updating our build servers one by one from:
dpkg -l | grep docker

ii  docker-buildx-plugin                   0.17.1-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                              5:27.3.1-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:27.3.1-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:27.3.1-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.29.7-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  python3-docker                         5.0.3-1                                 all          Python 3 wrapper to access docker.io's control socket

to

ii  docker-buildx-plugin                   0.19.2-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                              5:27.4.0-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:27.4.0-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:27.4.0-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.31.0-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  python3-docker                         5.0.3-1                                 all          Python 3 wrapper to access docker.io's control socket

So i'm not sure if it is the same issue or just a similar one.

Edit: also the following actions change the result to be equivalent in buildkit and non buildkit builds:

• removing the ONBUILD ARG and ONBUILD LABEL for build_date
• or skipping the docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image . build
  • but it then also happens if you run docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image --build-arg "also_used_later=true" . twice

[tonistiigi]

@ap-wtioit This probably broke with #5490 . As a workaround you can pin to 1.10 as previous comment suggests.