CS4.22 GPU Server Unable to create a VM

[tatay188]

problem

Unable to create VMs getting 504 error

The server is recognized a GPU enabled.
Using a regular template with UBUNTU 22.04

Using a Service created, the service is as all our server but has a GPU. HA, and i checked the Video just to test.
on the agent logs shows:

2025-12-09 20:33:14,351 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:3e3d3f80) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
2025-12-09 20:34:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:35:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:36:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:37:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:38:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:39:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:40:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:41:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:42:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:43:01,633 ERROR [cloud.agent.Agent] (AgentOutRequest-Handler-6:[]) (logid:) Ping Interval has gone past 300000. Won't reconnect to mgt server, as connection is still alive
2025-12-09 20:43:05,155 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:7f598e7f) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt

On the Server, there is no errors or disconnections.
the storage ID for this VM shows

ID
30aea531-8b82-478f-85db-e9991bf193f5

I am able to reach the primary storage from the GPU Host.

Except for this error and after 45 minutes the System keeps spinning on creating the VM "Launch Instance in progress"

Logs from the management server:

2025-12-09 20:46:49,858 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc70e228]) (logid:c4510d7b) No inactive management server node found

2025-12-09 20:46:49,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc70e228]) (logid:c4510d7b) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms

2025-12-09 20:46:51,322 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1aad6e7c]) (logid:829826e7) HA health check task is running...

2025-12-09 20:46:51,358 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1ff2c4b]) (logid:07a5943d) No inactive management server node found

2025-12-09 20:46:51,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1ff2c4b]) (logid:07a5943d) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms

2025-12-09 20:46:51,678 INFO  [c.c.a.m.A.MonitorTask] (AgentMonitor-1:[ctx-fe31f2a4]) (logid:825f839a) Found the following agents behind on ping: [75]

2025-12-09 20:46:51,683 DEBUG [c.c.a.m.A.MonitorTask] (AgentMonitor-1:[ctx-fe31f2a4]) (logid:825f839a) Ping timeout for agent Host {"id":75,"name":"ggpu","type":"Routing","uuid":"98715dad-759d-44c7-a633-524e9aa67431"}, do investigation

2025-12-09 20:46:51,685 INFO  [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Investigating why host Host {"id":75,"name":"ggpu","type":"Routing","uuid":"98715dad-759d-44c7-a633-524e9aa67431"} has disconnected with event

2025-12-09 20:46:51,687 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Checking if agent (Host {"id":75,"name":"ggpu","type":"Routing","uuid":"98715dad-759d-44c7-a633-524e9aa67431"}) is alive

2025-12-09 20:46:51,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Wait time setting on com.cloud.agent.api.CheckHealthCommand is 50 seconds

2025-12-09 20:46:51,690 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Seq 75-1207246175112003675: Routed from 250977680725600

2025-12-09 20:46:51,690 DEBUG [c.c.a.t.Request] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Seq 75-1207246175112003675: Sending  { Cmd , MgmtId: 250977680725600, via: 75(ggpu), Ver: v1, Flags: 100011, 
[{"com.cloud.agent.api.CheckHealthCommand":{"wait":"50","bypassHostMaintenance":"false"}}] }

2025-12-09 20:46:51,733 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 75-1207246175112003675: Processing:  { Ans: , MgmtId: 250977680725600, via: 75, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckHealthAnswer":{"result":"true","details":"resource is alive","wait":"0","bypassHostMaintenance":"false"}}] }

2025-12-09 20:46:51,734 DEBUG [c.c.a.t.Request] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Seq 75-1207246175112003675: Received:  { Ans: , MgmtId: 250977680725600, via: 75(ggpu), Ver: v1, Flags: 10, { CheckHealthAnswer } }

2025-12-09 20:46:51,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is alive

2025-12-09 20:46:51,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Agent (Host {"id":75,"name":"ggpu","type":"Routing","uuid":"98715dad-759d-44c7-a633-524e9aa67431"}) responded to checkHealthCommand, reporting that agent is Up

2025-12-09 20:46:51,734 INFO  [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) The agent from host Host {"id":75,"name":"ggpu","type":"Routing","uuid":"98715dad-759d-44c7-a633-524e9aa67431"} state determined is Up

2025-12-09 20:46:51,734 INFO  [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) Agent is determined to be up and running

2025-12-09 20:46:51,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-6:[ctx-f54bab9b]) (logid:58c1e81b) [Resource state = Enabled, Agent event = , Host = Ping]

2025-12-09 20:46:52,121 DEBUG [c.c.a.ApiServlet] (qtp1438988851-251223:[ctx-4df48857]) (logid:9ab16f87) ===START===  SOMEIPADDRESS  -- GET  jobId=22d89170-20e6-4151-a809-552938d734e9&command=queryAsyncJobResult&response=json&

2025-12-09 20:46:52,121 DEBUG [c.c.a.ApiServlet] (qtp1438988851-251223:[ctx-4df48857]) (logid:9ab16f87) Two factor authentication is already verified for the user 2, so skipping

2025-12-09 20:46:52,134 DEBUG [c.c.a.ApiServer] (qtp1438988851-251223:[ctx-4df48857, ctx-caa819c6]) (logid:9ab16f87) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"45a1be9e-2c67-11f0-a2e6-9ee6a2dce283"}]' is allowed to perform API calls:

I noticed the Isolated network the Virtual router is on another server, I do not have any server tags at the moment.

final error:

Error
Unable to orchestrate the start of VM instance {"instanceName":"i-2-223-VM","uuid":"a12748a3-7519-4732-8445-05dfa96046b7"}.

versions

The versions of ACS, hypervisors, storage, network etc..
ACS 4.22.0
KVM for the GPU and other hosts
CEPH RDB primary storage
NFS secondary storage
VXLAN running same as the other servers.
Ubuntu 22.04 as a Hypervisor
Ubuntu 22.04 as template - Same template used for other VMs.

The GPU is recognized by the system with no problems.

The steps to reproduce the bug

1. Using a GPU service offering with HA and GPU Display true - we do have disabled the OOB management.
2. Add a simple VM isolated network, using a GPU Offering 1GPU.
3. Everything starts ok, VR is created on a regular CPU server -automatically, Storage is created, Ip addresses allocated
4. Instance creation fails after 35+ minutes.

one more screen:

Please Guide us on the proper setting.

Thank you

What to do about it?

No response

[tatay188]

Just noticed after 50 minutes the ggpu serve becomes in alert mode and It's impossible to restart the GPU agent.
It just does not restart the agent on the GPU server.

Force-reconnect does not work

Have to restart the KVM Host.

So far the only servers not able to create a VM are the ones that contain a GPU.

[vishesh92]

@tatay188 I am not sure if this is because of the GPU server. Can you enable debug logs on the agent and share the logs again? Also, are you able to launch a non-GPU VM on GPU Enabled Host by manually selecting the host during VM start?

[tatay188]

@vishesh92
I did deployed on the GPU server a regular VM Ubuntu server, with no problem at all, the AGENT in debug mode shows several warnings with libvirt:

2025-12-10 12:05:49,735 WARN [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:962fb683) Execution of process [475638] for command [/bin/bash -c ovftool --version ] failed.
2025-12-10 12:05:49,735 WARN [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:962fb683) Process [475638] for command [/bin/bash -c ovftool --version ] encountered the error: [127].
2025-12-10 12:05:49,796 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:962fb683) Processing agent ready command, agent id = 75, uuid = 98715dad-759d-44c7-a633-524e9aa67431, name = ggpu
!
2025-12-10 12:05:49,805 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:962fb683) Ready command is processed for agent [id: 75, uuid: 98715dad-759d-44c7-a633-524e9aa67431, name: ggpu]
!
2025-12-10 12:05:49,868 WARN [utils.script.Script] (AgentRequest-Handler-4:[]) (logid:962fb683) Execution of process =[475660] for command [/bin/bash -c virt-v2v --version ] failed.
!
2025-12-10 12:05:49,869 WARN [utils.script.Script] (AgentRequest-Handler-4:[]) (logid:962fb683) Process [475660] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
!
2025-12-10 12:05:49,901 WARN [utils.script.Script] (AgentRequest-Handler-4:[]) (logid:962fb683) Execution of process [475680] for command [/bin/bash -c ovftool --version ] failed.
!
2025-12-10 12:05:49,901 WARN [utils.script.Script] (AgentRequest-Handler-4:[]) (logid:962fb683) Process [475680] for command [/bin/bash -c ovftool --version ] encountered the error: [127].
!
2025-12-10 12:06:47,660 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:485a8639) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
!
2025-12-10 12:06:47,690 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:485a8639) Asking libvirt to refresh storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a
!
2025-12-10 12:08:46,277 INFO [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-3:[]) (logid:4384b375) Setting quota=8300, period=10000 to VM domain 7f7480c0-e908-4d6c-b3f1-0d98ecfc3c39"
!
2025-12-10 12:08:46,317 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:4384b375) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
!
2025-12-10 12:08:48,541 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:4384b375) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
!
2025-12-10 12:08:48,697 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:4384b375) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
!
2025-12-10 12:08:48,727 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:4384b375) Trying to fetch storage pool e76f8956-1a81-3e97-aff6-8dc3f199a48a from libvirt
!
2025-12-10 12:08:49,193 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
!
2025-12-10 12:08:49,193 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Shell script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.sh' is not available. Transformations will not be applied.
!
2025-12-10 12:08:49,194 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Groovy scripting engine is not initialized. Data transformation skipped.
!
2025-12-10 12:08:51,786 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
!
2025-12-10 12:08:51,787 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Shell script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.sh' is not available. Transformations will not be applied.
!
2025-12-10 12:08:51,787 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-3:[]) (logid:4384b375) Groovy scripting engine is not initialized. Data transformation skipped.
!
And that's it, there is nothing else there. the non GPU VM is running with no problems, what do I do to eliminate those errors? as the libvirt QEMU is the version as it comes with cloudstack, not sure the hyperversor and libvirt versions have a mismatch, but did not want to change anything!

Let me know how you want me to proceed on this lab.

virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

[vishesh92]

Have you enabled IOMMU and disabled nouveau kernel module?

[tatay188]

@vishesh92
I do have IOMMU and I have NOUVEAU kernel diabled. The Kernel driver is nvidia, should I force it to vfio-pci ?

Is my understanding that for your platform Cloudstack only the VMs need the vfio-pci, not the KVM-host.
Please let me know.

lsmod | grep nvidia
nvidia_uvm 1781760 0
nvidia_drm 110592 0
nvidia_modeset 1622016 1 nvidia_drm
nvidia 103976960 19 nvidia_uvm,nvidia_modeset
drm_kms_helper 315392 8 mgag200,nvidia_drm
drm 622592 5 drm_kms_helper,nvidia,mgag200,nvidia_drm

!
!
Additionally we have the enable.additional.vm.configuration = false, we do not want to enable as true for now due to security.

[vishesh92]

Yes. Can you try setting driver to vfio-pci and try? Also, can you try with a new service offering without VM HA? And also try with display disabled.

[tatay188]

@vishesh92
I ended configuring the KVM as we did for OpenStack, and so far works using vfio-pci as I should be used from the beginning - I though the Cloudstack would/may take care of it via the management-agent communication with the KVM Host. We validated it does not, not a big deal - Now the VM immediately sees the GPU, that is a fantastic Improvement where we do not need to create all the Extraconfig, additionally there is no need to enable the "enable.additional.vm.configuration"