domain user metrics - request failed

[PPisz]

Hello,

After upgrading CS to 4.22, all domain user across all domains lost the ability to view VM metrics. A "Request failed" message displays, with no visible error in the mgmt log. All domain admins and root admins can see metrics as usual.
Has anyone else encountered this error? Or what might be causing it?

Piotr

[DaanHoogland]

@PPisz , I hadn’t heard this before. from what version did you upgrade?

[PPisz]

From 4.21, and it definitely worked there.

[PPisz]

On management log we have:
`2025-12-11 10:55:41,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Seq 4-4433512358169719428: Routed from 37951217891278
2025-12-11 10:55:42,442 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d]) (logid:d4eefa21) ===START=== 45.93.161.24 -- GET id=2a7e988d-6c4f-4ece-9420-be21e4c3c520&startDate=2025-12-11T09:55:43%2B01:00&endDate=2025-12-11T10:55:43%2B01:00&command=listVirtualMachinesUsageHistory&response=json&&projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
2025-12-11 10:55:42,442 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d]) (logid:d4eefa21) Two factor authentication is already verified for the user 307, so skipping
2025-12-11 10:55:42,450 DEBUG [c.c.a.ApiServer] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) CIDRs from which account 'Account [{"accountName":"usecrypt","id":104,"uuid":"d9591bbb-be72-4348-97ec-3dea3e76cb72"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-12-11 10:55:42,463 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-12-11 10:55:42,476 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) ===END=== 45.93.161.24 -- GET id=2a7e988d-6c4f-4ece-9420-be21e4c3c520&startDate=2025-12-11T09:55:43%2B01:00&endDate=2025-12-11T10:55:43%2B01:00&command=listVirtualMachinesUsageHistory&response=json&&projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
2025-12-11 10:55:42,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Seq 5-7783909005956528746: MgmtId 165190512198206 : Resp: Routing to peer
2025-12-11 10:55:42,886 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:[]) (logid:) Seq 4-8870402416059633629: Routing from 165190512198206
2025-12-11 10:55:42,909 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Seq 5-4433512358169719428: Received: { Ans: , MgmtId: 37951217891278, via: 5(sdc.dco.webdisk.io), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2025-12-11 10:55:43,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Seq 9-8870402416059633629: MgmtId 165190512198206 : Resp: Routing to peer
2025-12-11 10:55:43,085 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a60498c]) (logid:8583a3c8) No inactive management server node found
2025-12-11 10:55:43,085 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a60498c]) (logid:8583a3c8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-12-11 10:55:43,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds

If I use cmk as root admin:

(localcloud) 🐱 > list virtualmachinesusagehistory startdate="2025-12-11 09:55:43" enddate="2025-12-11 10:55:43" id=2a7e988d-6c4f-4ece-9420-be21e4c3c520 projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
{
"count": 1,
"virtualmachine": [
{
"displayname": "usecrypt-jump",
"id": "2a7e988d-6c4f-4ece-9420-be21e4c3c520",
"name": "i-107-2479-VM",
"stats": [
{
"cpuused": "0.54%",
"diskiopstotal": 56,
"diskioread": 0,
"diskiowrite": 56,
"diskkbsread": 0,
"diskkbswrite": 280,
"memoryintfreekbs": 444348,
"memorykbs": 2097152,
"memorytargetkbs": 2097152,
"networkkbsread": 12,
"networkkbswrite": 0,
"networkread": "0.01 MiB",
"networkwrite": "0.00 MiB",
"timestamp": "2025-12-11T09:56:13+0100"
},

If I use cmk as domain user:

(localcloud) 🐱 > list virtualmachinesusagehistory startdate="2025-12-11 09:55:43" enddate="2025-12-11 10:55:43" id=2a7e988d-6c4f-4ece-9420-be21e4c3c520 projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
(localcloud) 🐱 >

What else could I check?