SSH hangs: timeout during server version negotiation with AlmaLinux 9

[pietervogelaar]

I would like to run an amd64 AlmaLinux 9 Virtual Machine on an arm64 (silicon) MacBook Pro with Vagrant (2.4.3) and Qemu (9.2.0).

Vagrant.configure(2) do |config|
  config.vm.box = "almalinux/9"

  config.vm.provider "qemu" do |qe|
    qe.arch = "x86_64"
    qe.machine = "q35"
    qe.cpu = "qemu64"
    qe.net_device = "virtio-net-pci"
    qe.extra_netdev_args = "net=192.168.51.0/24,dhcpstart=192.168.51.10"
  end
end

However with this configuration that uses the almalinux/9 vagrant box, it keeps hanging at the SSH step:

vagrant up --provider qemu
Bringing machine 'default' up with 'qemu' provider...
==> default: Checking if box 'almalinux/9' version '9.2.20230513' is up to date...
==> default: Importing a QEMU instance
    default: Creating and registering the VM...
    default: Successfully imported VM
==> default: Warning! The QEMU provider doesn't support any of the Vagrant
==> default: high-level network configurations (`config.vm.network`). They
==> default: will be silently ignored.
==> default: Starting the instance...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:50022
    default: SSH username: vagrant
    default: SSH auth method: private key

If I use the centos/7 vagrant box as described in the README it just works.

I tried to debug it, but can't figure it out. It seems like the SSH server drops or refuses the connection as soon as the client shares its protocol. The line "local is" is normally followed by a "remote is" line.

DEBUG ssh: == Net-SSH connection debug-level log START ==
DEBUG ssh: D, [2025-01-05T20:16:26.594185 #17157] DEBUG -- net.ssh.transport.session[b68]: establishing connection to 127.0.0.1:50022
D, [2025-01-05T20:16:26.594651 #17157] DEBUG -- net.ssh.transport.session[b68]: connection established
I, [2025-01-05T20:16:26.594676 #17157]  INFO -- net.ssh.transport.server_version[b7c]: negotiating protocol version
D, [2025-01-05T20:16:26.594681 #17157] DEBUG -- net.ssh.transport.server_version[b7c]: local is `SSH-2.0-Ruby/Net::SSH_7.3.0 arm64-darwin'

DEBUG ssh: == Net-SSH connection debug-level log END ==

Is there a way to access the VM console screen to login as root? Then I can maybe see an error on the SSH server side.

I also asked a question at https://stackoverflow.com/questions/79331456/how-to-run-an-amd64-almalinux-9-vm-on-an-arm64-silicon-macbook-pro-with-vagran

[jesteban93]

A colleague of mine has the same issue (also Mac), and I just started experiencing it last Wednesday in Windows. With the following error:

An error occurred in the underlying SSH library that Vagrant uses.
The error message is shown below. In many cases, errors from this
library are caused by ssh-agent issues. Try disabling your SSH
agent or removing some keys and try again.

If the problem persists, please report a bug to the net-ssh project.

Net::SSH::ConnectionTimeout

[jesteban93]

Hey @pietervogelaar , I did manage to make it work by explicitly installing SSH in the vagrant box. I found out while trying a docker instance of Alma9 where I had a lot of trouble with ssh. I don't quite understand why it happens, but seems to be a matter of security restrictions in latest versions of Alma9, so now when it would create the machine with some basic packages, you may need to explicitly install them.

alma9.vm.provision "setup", type: "shell", preserve_order: true, inline: <<-EOF
      echo "Install OpenSSH"
      yum -y install openssh
EOF

[pietervogelaar]

@jesteban93 This doesn't work. Provisioners will only run after the VM is successfully set up. It's still stuck at the first SSH connection attempt.

[pietervogelaar]

To answer my question "Is there a way to access the VM console screen to login as root? Then I can maybe see an error on the SSH server side.":

For CentOS 7 the debug instruction that is described at https://github.com/ppggff/vagrant-qemu#debug works. I can login as root to the VM console through a serial connection, like nc -U /Users/.../qemu_socket_serial. However that only worked after vagrant replaced the SSH keys and the VM is successfully booted. So with AlmaLinux 9 the serial connection also hangs.