Custom yum repos and package upgrade question

[Omripresent]

Hello,

I'm trying to configure a local set of yum repos hosted on a local pulp server and remove/disable the default repos to prevent internet access.

My user config looks like this

# Create local repo files
yum_repos:
  local-almalinux-appstream:
    name: almalinux-appstream
    baseurl: 'https://<INTERNAL_SERVER>/pulp/content/rpm/almalinux-appstream/'
    repo_gpgcheck: false
    gpgcheck: false
    enabled: true

# Package install
package_reboot_if_required: true
package_upgrade: true

# Startup command runs
runcmd:
- "rm -vf /etc/yum.repos.d/almalinux-*.repo"

Looking at the cloud-init log the order of operations is yum_add_repo > runcmd > package_upgrade.
But looking at the output log it doesn't seem to be the case

2025-06-20 18:18:16,453 - schema.py[WARNING]: cloud-config failed schema validation! You may run 'sudo cloud-init schema --system' to check the details.
Cloud-init v. 24.4-4.el9.alma.1 running 'modules:config' at Fri, 20 Jun 2025 18:18:21 +0000. Up 12.76 seconds.
Cloud-init v. 24.4-4.el9.alma.1 running 'modules:final' at Fri, 20 Jun 2025 18:18:22 +0000. Up 13.71 seconds.
AlmaLinux 9 - AppStream                         0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'appstream':
  - Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
2025-06-20 18:18:23,276 - log_util.py[WARNING]: Package update failed
AlmaLinux 9 - AppStream                         0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'appstream':
  - Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
2025-06-20 18:18:23,835 - log_util.py[WARNING]: Package upgrade failed
AlmaLinux 9 - AppStream                         0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'appstream':
  - Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
  - Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream?countme=1 [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.almalinux.org/mirrorlist/9/appstream [Failed to connect to mirrors.almalinux.org port 443: Connection refused]
2025-06-20 18:18:24,390 - log_util.py[WARNING]: Failure when attempting to install packages: ['proxmox-backup-client', 'ipa-client']
2025-06-20 18:18:24,391 - cc_package_update_upgrade_install.py[WARNING]: 3 failed with exceptions, re-raising the last one
2025-06-20 18:18:24,392 - log_util.py[WARNING]: Running module package_update_upgrade_install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3.9/site-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
removed '/etc/yum.repos.d/almalinux-appstream.repo'
removed '/etc/yum.repos.d/almalinux-baseos.repo'
removed '/etc/yum.repos.d/almalinux-crb.repo'
removed '/etc/yum.repos.d/almalinux-extras.repo'
removed '/etc/yum.repos.d/almalinux-highavailability.repo'
removed '/etc/yum.repos.d/almalinux-nfv.repo'
removed '/etc/yum.repos.d/almalinux-plus.repo'
removed '/etc/yum.repos.d/almalinux-resilientstorage.repo'
removed '/etc/yum.repos.d/almalinux-rt.repo'
removed '/etc/yum.repos.d/almalinux-saphana.repo'
removed '/etc/yum.repos.d/almalinux-sap.repo'
Cloud-init v. 24.4-4.el9.alma.1 finished at Fri, 20 Jun 2025 18:18:24 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0].  Up 16.19 seconds

Any idea what I'm doing wrong here?

In a perfect world I should be able to override any existing repo files with a force flag or something, right now it skips files that already exist.

[blackboxsw]

Thanks for this question @Omripresent. Runcmd is a confusing one because the scripe rendered

Per typical /etc/cloud/cloud.cfg module ordering runcmd should run at the tail end of the modules:config stage. But this module only renders a runcmd script to /var/lib/cloud/instance/scripts/runcmd which ultimately gets run during the module cc_scripts_user.

That module is ordered later than package_update_upgrade_install which is activated by your package_upgrade: true config because typically runcmd operations like to take advantage of packages provided by newly added repositories.

In your case, I'd suggest dropping the package_upgrade: true and pulling out your dnf update or dnf -y upgrade into your runcmd after you have removed the repos.

Another alternative, if you knew all repo files you needed to remove, you could provide write_files config which would zero out the repo files you are trying to remove:

write_files:
- path: /etc/yum.repos.d/almalinux-appstream.repo
  content: ""
- path: /etc/yum.repos.d/almalinux-baseos.repo
  content: ""
- path: /etc/yum.repos.d/almalinux-crb.repo
  content: ""
...

Since write_files is run before package_update_upgrade_install module, it'd allow you to zero out those undesired default upstream repo files.