"xpu-smi ps" and xpumcli using VRAM

[WizardlyBump17]

Everytime I do xpu-smi ps it uses exactly 211.444MB of VRAM.

davi@davi:/tmp$ xpu-smi ps
PID       Command                       DeviceID       SHR            MEM            
4387      .mutter-Xwaylandauth.PPPSH3   0              0              0              
7832      xpu-smi                       0              0              211444         
davi@davi:/tmp$ xpu-smi ps
PID       Command                       DeviceID       SHR            MEM            
4387      .mutter-Xwaylandauth.PPPSH3   0              0              0              
7882      xpu-smi                       0              0              211444         
davi@davi:/tmp$ xpu-smi ps
PID       Command                       DeviceID       SHR            MEM            
4387      .mutter-Xwaylandauth.PPPSH3   0              0              0              
7932      xpu-smi                       0              0              211444

The xpumcli uses 206MB of VRAM, with the blame going to xpumd, since everytime I run xpumd the VRAM usage gets increased by that amount.

davi@davi:/tmp$ xpumcli ps
PID       Command                       DeviceID       SHR            MEM            
4387      .mutter-Xwaylandauth.PPPSH3   0              0              0              
5060      lact                          0              0              0              
5083      dump                          0              0              206 

It would be very nice if both tools used 0% of the VRAM because if you are already using 100% of the VRAM you dont want anything else to touch the VRAM.

Ubuntu 25.04 with a B580

[eero-t]

(Note: I'm not XPUM dev, just another user.)

xpumd daemon collects all the information it's been configured to collect, and provides to xpumcli client a subset of that data it requested. xpu-smi is basically a wrapper around the daemon, where daemon is in the same process, does only a single collection iteration, with somewhat limited support of operations. I.e. what the user requests with the CLI, and what the daemon part does, are separate things.

To limit amount of things done / collected by the daemon part (not just what data is reported by the CLI UI), you could try telling daemon part to collect only relevant metrics, and see whether that helps. See: https://github.com/intel/xpumanager/blob/master/core/include/xpum_api.h#L44

[WizardlyBump17]

(Note: I'm not XPUM dev, just another user.)

xpumd daemon collects all the information it's been configured to collect, and provides to xpumcli client a subset of that data it requested. xpu-smi is basically a wrapper around the daemon, where daemon is in the same process, does only a single collection iteration, with somewhat limited support of operations. I.e. what the user requests with the CLI, and what the daemon part does, are separate things.

To limit amount of things done / collected by the daemon part (not just what data is reported by the CLI UI), you could try telling daemon part to collect only relevant metrics, and see whether that helps. See: https://github.com/intel/xpumanager/blob/master/core/include/xpum_api.h#L44

The issue is not what is collected, but how it is collected. It doesnt make sense for a simple diagnostic tool to use 211MB of VRAM and also "update" the clocks and temperatures (https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/6917)

[eero-t]

(Note: I'm not XPUM dev, just another user.)

The issue is not what is collected, but how it is collected.

How did you verify that?

Did you try the described environment variables, and did they not have any impact on memory usage? [1]

For example:

export XPUM_METRICS=0

(I don't currently have XPUM supported GPU to test it myself.)

It doesnt make sense for a simple diagnostic tool to use 211MB of VRAM

While I agree that 211MB [2] seems huge amount of memory for monitoring purposes, after looking at all the functionality XPUM provides [3] and the the code to implement that, "simple" is not the word that I would use... :-)

and also "update" the clocks and temperatures (https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/6917)

I commented on that ticket. Clocks & temperatures changing isn't necessarily something XPUM specific, it could be just a side-effect.

---

[1] If you run xpu-smi e.g. with strace -f -e execve, I think you'll be surprised by what you see it doing. That, comparing the same output from xpumd, and looking a bit into the code, is IMHO rather convincing evidence that xpu-smi is just a thin wrapper around XPUM daemon functionality.

I.e. to control what the xpu-smi does / how much it uses resources, you need to specify what the daemon part linked to it does. xpu-smi wrapper CLI options are not enough for that.

[2] if xpumcli ps counts just daemon GPU memory usage, and does it correctly (does not e.g. include CPU side mappings). Hundreds of megs is a lot.

[3] Among other things, XPUM builds few compute kernels, can be asked to do FW updates, manage SR-IOV VFs, run media transcoding to stress the GPU etc, and in case of the daemon, do those even remotely with the REST API server. AI workloads use just GPU compute APIs, XPUM is using many additional GPU APIs.

[eero-t]

(I don't currently have XPUM supported GPU to test it myself.)

I meant dGPU. I tried XPUM v1.3.5 on TGL iGPU.

if xpumcli ps counts just daemon GPU memory usage, and does it correctly

With only 2 XPUM instances, and no other GPU workloads running on a 32GB device (memory shared with CPU):

# xpumcli ps
PID       Command                       DeviceID       SHR            MEM

# head /sys/kernel/debug/dri/128/{clients,i915_gem_objects}
==> /sys/kernel/debug/dri/128/clients <==
             command  tgid dev master a   uid      magic
               xpumd 23522 128   n    y     0          0
               xpumd 30846 128   n    y     0          0

==> /sys/kernel/debug/dri/128/i915_gem_objects <==
11 shrinkable [0 free] objects, 45056 bytes
system: total:0x00000007a74f2000 bytes
stolen-system: total:0x0000000003c00000 bytes

XPUM had claimed 6GB of GPU memory being used [1]:

# xpumd &
# xpumcli dump -d 0 -m 5,18 -n 1
Timestamp, DeviceId, GPU Memory Utilization (%), GPU Memory Used (MiB)
15:26:54.386,    0, 19.71, 6181.65

If I remove the other XPUM instance, usage goes down by 0.1GB:

# xpumcli dump -d 0 -m 5,18 -n 1
Timestamp, DeviceId, GPU Memory Utilization (%), GPU Memory Used (MiB)
15:52:11.721,    0, 19.42, 6089.09

Which is half of what you were seeing.

Btw. If I enable only GPU memory usage metrics collection:
# XPUM_METRICS=8,9 xpumd &

XPUM daemon memory usage does not decrease, but that does not necessarily apply to dGPUs, because only memory, frequency & engine utilization GPU metrics are supported on iGPUs, so initial situation is very different from dGPU.

That could also explain why in my case XPUM instance seems to use half of the VRAM that you saw. I.e. it makes sense to test whether XPUM_METRICS=8,9 xpumd & changes anything for you.

---

[1] Other ~6GB iGPU VRAM "usage" is probably CPU processes, as iGPU shares memory with CPU.

[WizardlyBump17]

When installing the xpumanager package the gpu starts being used immediately with the same issues. even sudo env XPUM_METRICS=0 xpumd uses the GPU.

Yes, I did stop the previous service.

I am using the kernel 6.18.0-061800-generic, Mesa 25.3.2 - kisak-mesa PPA and compute-runtime 25.48.36300.8

[WizardlyBump17]

Maybe an important info I forgot: I am also using a GTX 1650 on the same system with its nvidia-driver-580, but I remember seeing this issue with the B580 alone

[WizardlyBump17]

A cool find: if I spam xpu-smi ps my temperatures start to go lower. I started watch -n 0.1 xpu-smi ps 3 minutes ago and I am down almost 10ºC already

[eero-t]

When installing the xpumanager package the gpu starts being used immediately with the same issues. even sudo env XPUM_METRICS=0 xpumd uses the GPU.

Good to know, thanks!

Maybe an important info I forgot: I am also using a GTX 1650 on the same system with its nvidia-driver-580, but I remember seeing this issue with the B580 alone

I don't think it would affect VRAM usage, but setups with other than Intel GPUs are unlikely to be part of regular XPUM validation, so it's good to mention it.

A cool find: if I spam xpu-smi ps my temperatures start to go lower. I started watch -n 0.1 xpu-smi ps 3 minutes ago and I am down almost 10ºC already

GPU temperature? When GPU is fully utilized?

If it's fully utilized, such behaviour is kind of expected as more of time would be spent switching between GPU tasks and on CPU<->GPU syncs, than on GPU activity requiring more power (GPU compute cores & BW usage). This would show up as lower perf for the interrupted GPU task.