Ubuntu:focal + latest docker-in-docker feature fails to prebuild due to Moby package validation issue

[sayhiben]

Hi there,

I manage a devcontainer config for Codespaces that uses the ubuntu:focal image (like the universal image does) alongside the latest docker-in-docker feature. As of around 10am Pacific time today, our Codespaces prebuilds began to fail with the following error:

#28 18.36 Err:1 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-buildx amd64 0.12.1-ubuntu20.04u1
#28 18.36   File has unexpected size (34152700 != 34091250). Mirror sync in progress? [IP: 13.90.21.104 443]

GitHub Actions Prebuild logs:

#28 0.707 ===========================================================================
#28 0.707 Feature       : Docker (Docker-in-Docker)
#28 0.707 Description   : Create child containers *inside* a container, independent from the host's docker instance. Installs Docker extension in the container along with needed CLIs.
#28 0.707 Id            : ghcr.io/devcontainers/features/docker-in-docker
#28 0.707 Version       : 2.8.1
#28 0.707 Documentation : ********/devcontainers/features/tree/main/src/docker-in-docker
#28 0.707 Options       :
#28 0.707     VERSION="latest"
#28 0.707     MOBY="********"
#28 0.707     DOCKERDASHCOMPOSEVERSION="v1"
#28 0.707     AZUREDNSAUTODETECTION="********"
#28 0.707     DOCKERDEFAULTADDRESSPOOL=""
#28 0.707     INSTALLDOCKERBUILDX="********"
#28 18.36 Err:1 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-buildx amd64 0.12.1-ubuntu20.04u1
#28 18.36   File has unexpected size (34152700 != 34091250). Mirror sync in progress? [IP: 13.90.21.104 443]
#28 18.36   Hashes of expected file:
#28 18.36    - SHA256:0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03
#28 18.36    - Filesize:34091250 [weak]
#28 18.38 Get:2 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-cli amd64 24.0.9-ubuntu20.04u1 [15.6 MB]#28 18.58 Get:3 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-runc amd64 1.1.12-ubuntu20.04u1 [6680 kB]#28 18.64 Get:4 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-containerd amd64 1.6.28-ubuntu20.04u1 [45.9 MB]
#28 18.99 Get:5 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-tini amd64 0.19.0-ubuntu20.04u1 [350 kB]
#28 18.99 Get:6 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-engine amd64 24.0.9-ubuntu20.04u1 [32.2 MB]
#28 19.22 Fetched 101 MB in 1s (113 MB/s)
#28 19.22 E: Failed to fetch https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb  File has unexpected size (34152700 != 34091250). Mirror sync in progress? [IP: 13.90.21.104 443]
#28 19.22    Hashes of expected file:
#28 19.22     - SHA256:0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03
#28 19.22     - Filesize:34091250 [weak]
#28 19.22 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
#28 19.23 (!) Packages for moby not available in OS ubuntu focal (amd64). To resolve, either: (1) set feature option '"moby": false' , or (2) choose a compatible OS version (eg: 'ubuntu-20.04').
#28 19.24 ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install! Look at the documentation at ********/devcontainers/features/tree/main/src/docker-in-docker for help troubleshooting this error.
#28 ERROR: process "/bin/sh -c cp -ar /tmp/build-features-src/docker-in-docker_6 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/docker-in-docker_6  && cd /tmp/dev-container-features/docker-in-docker_6  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/docker-in-docker_6" did not complete successfully: exit code: 1
------
 > [dev_containers_target_stage 10/10] RUN --mount=type=bind,from=dev_containers_feature_content_source,source=docker-in-docker_6,target=/tmp/build-features-src/docker-in-docker_6     cp -ar /tmp/build-features-src/docker-in-docker_6 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/docker-in-docker_6  && cd /tmp/dev-container-features/docker-in-docker_6  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/docker-in-docker_6:
18.99 Get:5 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-tini amd64 0.19.0-ubuntu20.04u1 [350 kB]
18.99 Get:6 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 moby-engine amd64 24.0.9-ubuntu20.04u1 [32.2 MB]
19.22 Fetched 101 MB in 1s (113 MB/s)
19.22 E: Failed to fetch https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb  File has unexpected size (34152700 != 34091250). Mirror sync in progress? [IP: 13.90.21.104 443]
19.22    Hashes of expected file:
19.22     - SHA256:0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03
19.22     - Filesize:34091250 [weak]
19.22 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
19.23 (!) Packages for moby not available in OS ubuntu focal (amd64). To resolve, either: (1) set feature option '"moby": false' , or (2) choose a compatible OS version (eg: 'ubuntu-20.04').
19.24 ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install! Look at the documentation at ********/devcontainers/features/tree/main/src/docker-in-docker for help troubleshooting this error.
------
Dockerfile-with-features:183
--------------------
 182 |     ENV DOCKER_BUILDKIT="1"
 183 | >>> RUN --mount=type=bind,from=dev_containers_feature_content_source,source=docker-in-docker_6,target=/tmp/build-features-src/docker-in-docker_6 \
 184 | >>>     cp -ar /tmp/build-features-src/docker-in-docker_6 /tmp/dev-container-features \
 185 | >>>  && chmod -R 0755 /tmp/dev-container-features/docker-in-docker_6 \
 186 | >>>  && cd /tmp/dev-container-features/docker-in-docker_6 \
 187 | >>>  && chmod +x ./devcontainer-features-install.sh \
 188 | >>>  && ./devcontainer-features-install.sh \
 189 | >>>  && rm -rf /tmp/dev-container-features/docker-in-docker_6
 190 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c cp -ar /tmp/build-features-src/docker-in-docker_6 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/docker-in-docker_6  && cd /tmp/dev-container-features/docker-in-docker_6  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/docker-in-docker_6" did not complete successfully: exit code: 1
Stop: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-root/container-features/0.56.1-1707501892313/Dockerfile-with-features -t vsc-web-abc123 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-root/container-features/0.56.1-1707501892313 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/lib/docker/codespacemount/workspace/web/.devcontainer
Error: Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-root/container-features/0.56.1-1707501892313/Dockerfile-with-features -t vsc-web-abc123 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-root/container-features/0.56.1-1707501892313 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/lib/docker/codespacemount/workspace/web/.devcontainer
    at j$ (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:465:1933)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Lw (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:464:1831)
    at async iK (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:464:608)
    at async gAA (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:481:3660)
{"outcome":"error","message":"Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-root/container-features/0.56.1-1707501892313/Dockerfile-with-features -t vsc-web-abc123 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-root/container-features/0.56.1-1707501892313 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/lib/docker/codespacemount/workspace/web/.devcontainer","description":"An error occurred building the image."}
    at async BC (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:481:4775)
    at async xeA (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:614:11265)
    at async UeA (/.codespaces/agent/bin/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:614:11006)
devcontainer process exited with exit code 1
Failed to create container.
Error: Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-root/container-features/0.56.1-1707501892313/Dockerfile-with-features -t vsc-web-abc123 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-root/container-features/0.56.1-1707501892313 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/lib/docker/codespacemount/workspace/web/.devcontainer
Error code: 1302 (UnifiedContainersErrorFatalCreatingContainer)
Container creation failed.
Jobs failed, exiting the agent. Error code: 1302 (UnifiedContainersErrorFatalCreatingContainer)
Error: Process completed with exit code 1.

devcontainer.json:

 "build": {
    "dockerfile": "./Dockerfile",
    "context": "."
  },
  "runArgs": [ "--init", "--privileged" ], // for Docker-in-Docker
  "overrideCommand": false, // for Docker-in-Docker

  "features": {
    "ghcr.io/devcontainers/features/common-utils:2": {
      "username": "codespace",
      "userUid": "1000",
      "userGid": "1000"
    },
    "ghcr.io/devcontainers/features/docker-in-docker:2": {
      "version": "latest"
    },
    ...
  },

  "overrideFeatureInstallOrder": [
    "ghcr.io/devcontainers/features/common-utils",
    ...
    "ghcr.io/devcontainers/features/docker-in-docker",
  ],

Dockerfile:

FROM ubuntu:focal

COPY first-run-notice.txt /tmp/scripts/

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    # Restore man command
    && yes | unminimize 2>&1

ENV LANG="C.UTF-8"

# Install basic build tools
RUN apt-get update \
    && apt-get upgrade -y \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    [all the stuff from the universal image]

# Verify expected build and debug tools are present
RUN apt-get update \
    && apt-get -y install build-essential cmake cppcheck valgrind clang lldb llvm gdb python3-dev \
    # Install tools and shells not in common script
    && apt-get install -yq vim vim-doc xtail software-properties-common libsecret-1-dev \
    # Install additional tools (useful for 'puppeteer' project)
    && apt-get install -y --no-install-recommends l
    [stuff from universal]

# Default to bash shell (zsh shells available at /usr/bin/zsh)
ENV SHELL=/bin/bash \
    DOCKER_BUILDKIT=1

# Mount for docker-in-docker
VOLUME [ "/var/lib/docker" ]

# Fire Docker/Moby script if needed
ENTRYPOINT [ "/usr/local/share/docker-init.sh", "/usr/local/share/ssh-init.sh"]
CMD [ "sleep", "infinity" ]

[sayhiben]

I'm going to attempt to pin the docker-in-docker version back to 2.8.0 - I suspect something with 2.8.1 caused this, but I'm uncertain yet

edit: This didn't work. Apparently I misunderstood what the feature's version attr references; it seems it actually controls the underlying Moby version:

#28 12.12 Reading package lists...#28 15.25 (!) No full or partial Docker / Moby version match found for "2.8.0" on OS ubuntu focal (amd64). Available versions:
#28 16.30  24.0.9-ubuntu20.04u1 
#28 16.30  24.0.8-ubuntu20.04u1 
#28 16.30  24.0.7-ubuntu20.04u1 
#28 16.30  23.0.7+azure-ubuntu20.04u1 
#28 16.30  23.0.6+azure-ubuntu20.04u2 
#28 16.30  20.10.25+azure-ubuntu20.04u2 
#28 16.30  20.10.25+azure-ubuntu20.04u1 
...

I'm going to try changing to 20.10.25+azure-ubuntu20.04u2 and see what happens

edit2: That version is either the same or is similarly not working

[nashafa]

Just commenting to document that I'm running into the same issue, but with debian bookworm (arm64). The host machine is an Apple Silicon device. Neither downgrading the docker-in-docker feature or clearing any local caches resolves the issue.

[rchui]

Also running into this issue. Apple Silicon, Debian Bullseye

[leonitousconforti]

Also have the same issue

Logs:

#0 0.160 ===========================================================================
#0 0.160 Feature       : Docker (docker-outside-of-docker)
#0 0.160 Description   : Re-use the host docker socket, adding the Docker CLI to a container. Feature invokes a script to enable using a forwarded Docker socket within a container to run Docker commands.
#0 0.160 Id            : ghcr.io/devcontainers/features/docker-outside-of-docker
#0 0.160 Version       : 1.3.2
#0 0.160 Documentation : https://github.com/devcontainers/features/tree/main/src/docker-outside-of-docker
#0 0.160 Options       :
#0 0.160     VERSION="latest"
#0 0.160     MOBY="true"
#0 0.160     DOCKERDASHCOMPOSEVERSION="v2"
#0 0.160     INSTALLDOCKERBUILDX="true"
#0 0.160 ===========================================================================
#0 0.166 Distro codename  'bullseye'  matched filter  'bookworm buster bullseye bionic focal jammy'
#0 0.455 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
#0 0.519 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
#0 0.543 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
#0 0.590 Get:4 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye InRelease [3649 B]
#0 0.648 Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8062 kB]
#0 0.792 Get:6 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main all Packages [1214 B]
#0 0.863 Get:7 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main amd64 Packages [135 kB]
#0 1.189 Get:8 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [267 kB]
#0 1.203 Get:9 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
#0 1.973 Fetched 8696 kB in 2s (5366 kB/s)
#0 1.973 Reading package lists...
#0 2.214 Reading package lists...
#0 2.450 Building dependency tree...
#0 2.498 Reading state information...
#0 2.556 Recommended packages:
#0 2.556   pigz
#0 2.566 The following NEW packages will be installed:
#0 2.566   moby-buildx moby-cli
#0 2.738 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
#0 2.738 Need to get 49.7 MB of archives.
#0 2.738 After this operation, 113 MB of additional disk space will be used.
#0 2.738 Get:1 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main amd64 moby-buildx amd64 0.12.1-debian11u1 [34.1 MB]
#0 2.738 Err:1 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main amd64 moby-buildx amd64 0.12.1-debian11u1
#0 2.738   File has unexpected size (34140374 != 34084902). Mirror sync in progress? [IP: 13.82.67.141 443]
#0 2.738   Hashes of expected file:
#0 2.738    - SHA256:72ba1ec05f3db690e87bb342e94c5bf20e413c1ef8541c1efb7cce562d1a70d4
#0 2.738    - Filesize:34084902 [weak]
#0 2.968 Get:2 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main amd64 moby-cli amd64 24.0.9-debian11u1 [15.6 MB]
#0 5.268 E: Failed to fetch https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-debian11u1_amd64.deb  File has unexpected size (34140374 != 34084902). Mirror sync in progress? [IP: 13.82.67.141 443]
#0 5.268    Hashes of expected file:
#0 5.268     - SHA256:72ba1ec05f3db690e87bb342e94c5bf20e413c1ef8541c1efb7cce562d1a70d4
#0 5.268     - Filesize:34084902 [weak]
#0 5.268 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
#0 5.268 Fetched 15.6 MB in 3s (5807 kB/s)
#0 5.268 ERROR: Feature "Docker (docker-outside-of-docker)" (ghcr.io/devcontainers/features/docker-outside-of-docker) failed to install!

[sayhiben]

It appears that the issue here is Microsoft's package repository claiming the package should be one size but the downloaded package being another size, and then Ubuntu / apt being unhappy with the mismatch. Some quick searching says this likely has to be fixed on Microsoft's package repo

[sayhiben]

Here's a bash script that can be run to reproduce this issue locally (at least, on one of my still-working Codespaces in ubuntu focal):

edit: It does not matter what the version suffixes are set to; the issue is moby-buildx 0.12.1 failing its checksum in apt due to a filesize/SHA difference versus what's defined in Microsoft's package registry here. This could be indicative of a supply chain attack (unlikely, but possible)

#!/bin/bash

set -xeuo pipefail

MICROSOFT_GPG_KEYS_URI='https://packages.microsoft.com/keys/microsoft.asc'
VERSION_CODENAME='focal'
ID='ubuntu'

architecture=$(dpkg --print-architecture)
engine_package_name="moby-engine"
cli_package_name="moby-cli"

# Import key safely and import Microsoft apt repo
curl -sSL ${MICROSOFT_GPG_KEYS_URI} | gpg --dearmor > /usr/share/keyrings/microsoft-archive-keyring.gpg
echo "deb [arch=${architecture} signed-by=/usr/share/keyrings/microsoft-archive-keyring.gpg] https://packages.microsoft.com/repos/microsoft-${ID}-${VERSION_CODENAME}-prod ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/microsoft.list

apt-get update

engine_version_suffix='=20.10.25+azure-ubuntu20.04u2'
cli_version_suffix='=20.10.25+azure-ubuntu20.04u2'

apt-get -y install --allow-downgrades --no-install-recommends moby-cli${cli_version_suffix} moby-buildx moby-engine${engine_version_suffix}

[sayhiben]

Here's where things are going off the rails - docker-in-docker attempts to download the latest version of moby-buildx which has the following entry in Microsoft's package registry:

Package: moby-buildx
Version: 0.12.1-ubuntu20.04u1
Architecture: amd64
Section: admin
Priority: optional
Installed-Size: 75208
Maintainer: Microsoft <support@microsoft.com>
Description: A Docker CLI plugin for extended build capabilities with BuildKit
Homepage: https://github.com/docker/buildx
Conflicts: docker-buildx-plugin, docker-ce, docker-ee
Depends: libc6 (>= 2.3.4)
Recommends: moby-cli
Replaces: docker-buildx-plugin
SHA256: 0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03
Size: 34091250
Filename: pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb

The actual filesize of the downloaded package is 34152700 with SHA 707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c

Microsoft needs to look at their registry and fix their Packages manifest. I believe we can fix the docker-in-docker feature by allowing for the moby-buildx package to have its version set via devcontainer.json, but that will require a PR and change in this repo that will likely be temporary

[samruddhikhandale]

Thanks for reporting and providing a possible solution.

I believe we can fix the docker-in-docker feature by allowing for the moby-buildx package to have its version set via devcontainer.json, but that will require a PR and change in this repo that will likely be temporary

I'd quickly create a PR to temporary unblock everyone. Thanks for your patience.

[samruddhikhandale]

@sayhiben Beat me to it, thanks for #838
Reviewing!

[sayhiben]

Thanks for the assistance, @samruddhikhandale - for what it's worth, I believe this to also affect GitHub's ability to create their default Codespaces image using devcontainer/images for Universal linux