Cache the Arm FVP and toolchain downloads in the CI images (#24585)

Every CI job that runs examples/arm/setup.sh still downloads the default
FVPs and the Arm GNU toolchain from developer.arm.com, about 394 MB, and
the
NXP MCUXpresso test fetches the same toolchain with wget. Arm has asked
us
to stop hitting their CDN this often, and the pinned versions change a
few
times a year, so the CI images can carry them.

download_with_retry now looks in ARM_SETUP_DOWNLOAD_CACHE first. A file
there named after the URL is copied instead of downloaded, but only when
it
matches the pinned MD5, so a stale cache costs a download, never a wrong
archive. Each miss is logged as a GitHub warning, so a cache that stops
matching shows up on the run. The NXP script copies its toolchain from
the
same place.

The arm-sdk and zephyr-sdk images fill /opt/arm-download-cache at build
time with the Corstone-300 and Corstone-320 FVPs and the arm-none-eabi
toolchain, sourcing the same fvp_utils.sh and toolchain_utils.sh as
setup.sh, which build.sh copies into the build context as it already
does
for requirements-lintrunner.txt. The fill is best effort, so an outage
of
Arm's CDN cannot break the image build for jobs that never download from
it.

The image tag only hashes .ci/docker, so docker-builds.yml also triggers
on
those two files and a pin bump on main rebuilds the images under the
same
tag. Until a job gets the rebuilt image it downloads and warns: during
the
rebuild, on runner nodes that still hold the old image, on PRs based
before
the bump, and after a ciflow/docker build of a pin-bump PR, which
overwrites
main's images until the next main or weekly rebuild. Images are rebuilt
about once a day, mostly for the PyTorch pin PR, so image builds still
make
roughly ten downloads a day, against about 10,000 from jobs before.

Authored with Claude Code.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils
@Sebastian-Larsson @robell

Author: rascani

.ci/docker/build.sh

@@ -116,6 +116,10 @@ fi
 # Copy requirements-lintrunner.txt from root to here
 cp ../../requirements-lintrunner.txt ./
 
+# Copy the Arm setup scripts that pin the FVP and toolchain downloads
+mkdir -p arm-setup
+cp ../../backends/arm/scripts/{utils,fvp_utils,toolchain_utils}.sh arm-setup/
+
 # OSDC runners have no docker daemon, so the build runs on the in-cluster
 # BuildKit pool via a remote buildx builder. That builder has nowhere to load
 # an image into, so the result has to go straight to the registry.

.ci/docker/common/install_arm_downloads.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Cache the developer.arm.com downloads of examples/arm/setup.sh, so CI jobs
+# copy them from ARM_SETUP_DOWNLOAD_CACHE instead of fetching them each run.
+# Jobs download whatever is missing, so a failed download only costs the cache.
+
+set -exu
+
+ARCH="$(uname -m)"
+OS="$(uname -s)"
+ARM_SETUP_CURL_PROGRESS_ARGS=(--no-progress-meter)
+# Fill the cache rather than read from it
+cache_dir="${ARM_SETUP_DOWNLOAD_CACHE}"
+unset ARM_SETUP_DOWNLOAD_CACHE
+source ./arm-setup/fvp_utils.sh
+source ./arm-setup/toolchain_utils.sh
+
+cache_download() {
+    local url=$1
+    local md5=$2
+    local output="${cache_dir}/$(basename "${url}")"
+    download_with_retry "cache" "${url}" "${output}" "${md5}" || rm -f "${output}"
+}
+
+mkdir -p "${cache_dir}"
+cache_download "${corstone300_url}" "${corstone300_md5_checksum}"
+cache_download "${corstone320_url}" "${corstone320_md5_checksum}"
+gcc_select_toolchain
+cache_download "${toolchain_url}" "${toolchain_md5_checksum}"

.ci/docker/ubuntu/Dockerfile

@@ -103,6 +103,16 @@ COPY ./common/utils.sh utils.sh
 RUN if [ -n "${ZEPHYR_SDK}" ]; then bash ./install_zephyr.sh; fi
 RUN rm install_zephyr.sh zephyr_sdk_release_proxy.py utils.sh
 
+# Cache the developer.arm.com downloads of examples/arm/setup.sh, which the
+# Arm and Zephyr jobs run.
+ENV ARM_SETUP_DOWNLOAD_CACHE=/opt/arm-download-cache
+COPY ./common/install_arm_downloads.sh install_arm_downloads.sh
+COPY ./arm-setup arm-setup
+RUN if [ -n "${ARM_SDK}${ZEPHYR_SDK}" ]; then \
+    bash ./install_arm_downloads.sh || echo "Arm download cache not populated"; \
+    fi
+RUN rm -r install_arm_downloads.sh arm-setup
+
 ARG QNN_SDK
 
 ARG MEDIATEK_SDK

.github/workflows/docker-builds.yml

@@ -16,6 +16,8 @@ on:
       - .ci/docker/**
       - .github/workflows/docker-builds.yml
       - requirements-lintrunner.txt
+      - backends/arm/scripts/fvp_utils.sh
+      - backends/arm/scripts/toolchain_utils.sh
   schedule:
     - cron: 1 3 * * 3
 

backends/arm/scripts/utils.sh

@@ -76,6 +76,9 @@ function download_with_retry() {
     # both are caught here. --fail rejects HTTP errors,
     # --retry-all-errors handles transport errors, and verify_md5 catches
     # truncation / wrong-content via the published archive checksum.
+    # If the absolute path ARM_SETUP_DOWNLOAD_CACHE holds a file named after
+    # the URL with the expected MD5, it is copied instead; the CI images cache
+    # them there.
 
     # Arg 1: log context (passed to log_step)
     # Arg 2: URL to download
@@ -89,6 +92,16 @@ function download_with_retry() {
     local output="${3}"
     local expected_md5="${4}"
 
+    if [[ -n "${ARM_SETUP_DOWNLOAD_CACHE:-}" ]]; then
+        local cached="${ARM_SETUP_DOWNLOAD_CACHE}/$(basename "${url}")"
+        if [[ -f "${cached}" ]] && verify_md5 "${expected_md5}" "${cached}" \
+           && cp "${cached}" "${output}"; then
+            log_step "${context}" "Using cached ${cached}"
+            return 0
+        fi
+        echo "::warning::[Arm Setup/${context}] No usable ${cached}; downloading"
+    fi
+
     local max_attempts=5
     for attempt in $(seq 1 ${max_attempts}); do
         rm -f "${output}"

examples/nxp/mcuxpresso/imxrt700/executorch_cifarnet/test_build_from_scratch.sh

@@ -12,7 +12,13 @@ echo Downloading ARM GCC toolchain
 if [ ! -d arm-toolchain ]; then
     mkdir -p arm-toolchain
     pushd arm-toolchain
-    wget $ARM_TOOLCHAIN_URL
+    # The Arm CI images cache this archive (see backends/arm/scripts/utils.sh)
+    cached_toolchain="${ARM_SETUP_DOWNLOAD_CACHE:-}/$(basename $ARM_TOOLCHAIN_URL)"
+    if [ -n "${ARM_SETUP_DOWNLOAD_CACHE:-}" ] && [ -f "$cached_toolchain" ]; then
+        cp "$cached_toolchain" .
+    else
+        wget $ARM_TOOLCHAIN_URL
+    fi
     tar -xvf *.tar.xz
     rm *.tar.xz
     popd