deb 13 cloud-init hung at "Waiting on external services to complete"

[rjhornsby]

Have had cloud-init working on RHEL-based ec2 systems for years, but using that same config on (pretty vanilla) debian 13 is not behaving. More specifically, cloud-init never finishes. The last message in /var/log/cloud-init.log is

socket.py[INFO]: Sending sd_notify(STATUS=Waiting on external services to complete before starting the network stage.)

There are no errors (I can find) in /var/log/cloud-init.log. Given the last message, I can't figure out which service(s) cloud-init is waiting on.

I think because cloud-init is getting hung up, it's never finishing so the ec2-user user never gets created.

I ended up commenting out every cloud_init and cloud_config module from /etc/cloud/cloud.cfg[1] figuring if I skip all those I skip problems with those, and running cloud-init clean --logs --reboot to try again, but it doesn't seem to have changed anything.

Going through the "How to debug"[2] doc, I'm not finding much data that seems helpful. I put the output of the 5 different debug stages below. systemctl list-jobs --after seems to be the only step with anything -- but even then I don't know what cloud-init is waiting for or why it's blocking itself.

Any suggestions would be welcome. thanks.

1. dmesg

╰─○ sudo dmesg -T | grep -i -e warning -e error -e fatal -e exception
[Wed Sep 24 18:22:12 2025] Speculative Return Stack Overflow: WARNING: See https://kernel.org/doc/html/latest/admin-guide/hw-vuln/srso.html for mitigation options.
[Wed Sep 24 18:22:12 2025] i8042: Warning: Keylock active
[Wed Sep 24 18:22:12 2025] RAS: Correctable Errors collector initialized.

2. systemctl failed

╰─○ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

3. cloud-init status

╰─○ cloud-init status --long
status: running
extended_status: running
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:08 +0000
detail: DataSourceEc2Local
errors: []
recoverable_errors: {}

4. running services boot stage

╰─○ systemctl list-jobs --after
JOB UNIT                                                 TYPE  STATE
148 cloud-final.service                                  start waiting
└─      blocking job 147 (cloud-init.target/start)          -     -
147 cloud-init.target                                    start waiting
150 cloud-config.service                                 start running
└─      blocking job 147 (cloud-init.target/start)          -     -
└─      blocking job 148 (cloud-final.service/start) -     -

3 jobs listed.

5. pstree

╰─○ for pid in 148 147 150; do echo "pstree of pid $pid"; pstree $pid; done
pstree of pid 148
pstree of pid 147
pstree of pid 150

[1] https://gist.github.com/rjhornsby/378163b833f22c6cc319f7b74891632c
[2] https://cloudinit.readthedocs.io/en/latest/howto/debugging.html

[holmanb]

@rjhornsby This doesn't seem appropriate as a discussion topic, this should be a an issue. Also, I think that this one is likely a downstream issue.

[rjhornsby]

@holmanb thanks. I thought discussion fit better, assuming it was my problem or something I'm doing wrong, but I'll move this to an issue.

By "downstream" are you thinking this is a problem in Debian or with one of the (not cloud-init) services?

[rjhornsby]

Created issue #6487