4xB60 setup failing eval with 'Devices 0 and 1 do not have P2P capabilities'

[mwrothbe]

I'm running two Maxsun B60 dual cards in a system with Proxmox OS an iommu pass-through to Ubuntu. I ran the install.sh script which completed without errors, but after rebooting, running scripts/evaluation/platform_basic_evaluation.sh as per the title of this issue.

I don't seem to be able to add attachments to this, so here's the content of the log, and I can add the systeinfo tar if I can add files at some point. Any ideas what I might be doing wrong or what I can try to debug?

Thanks!

:: initializing oneAPI environment ...
platform_basic_evaluation.sh: BASH_VERSION = 5.2.37(1)-release
args: Using "$@" for setvars.sh arguments: --force
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: ipp -- latest
:: mkl -- latest
:: mpi -- latest
:: pti -- latest
:: tbb -- latest
:: umf -- latest
:: oneAPI environment initialized ::

[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) Pro B60 Graphics 20.1.0 [1.14.36300+8]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) Pro B60 Graphics 20.1.0 [1.14.36300+8]
[level_zero:gpu][level_zero:2] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) Pro B60 Graphics 20.1.0 [1.14.36300+8]
[level_zero:gpu][level_zero:3] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) Pro B60 Graphics 20.1.0 [1.14.36300+8]
[opencl:cpu][opencl:0] Intel(R) OpenCL, Intel(R) Xeon(R) w5-3435X OpenCL 3.0 (Build 0) [2025.19.4.0.18_160000.xmain-hotfix]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Pro B60 Graphics OpenCL 3.0 NEO [25.48.36300.8]
[opencl:gpu][opencl:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Pro B60 Graphics OpenCL 3.0 NEO [25.48.36300.8]
[opencl:gpu][opencl:3] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Pro B60 Graphics OpenCL 3.0 NEO [25.48.36300.8]
[opencl:gpu][opencl:4] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Pro B60 Graphics OpenCL 3.0 NEO [25.48.36300.8]
+-----------+--------------------------------------------------------------------------------------+
| Device ID | Device Information |
+-----------+--------------------------------------------------------------------------------------+
| 0 | Device Name: Intel(R) Arc(TM) Pro B60 Graphics |
| | Vendor Name: Intel(R) Corporation |
| | SOC UUID: 00000000-0000-0001-0000-0000e2118086 |
| | PCI BDF Address: 0000:01:00.0 |
| | DRM Device: /dev/dri/card0 |
| | Function Type: physical |
+-----------+--------------------------------------------------------------------------------------+
| 1 | Device Name: Intel(R) Arc(TM) Pro B60 Graphics |
| | Vendor Name: Intel(R) Corporation |
| | SOC UUID: 00000000-0000-0002-0000-0000e2118086 |
| | PCI BDF Address: 0000:02:00.0 |
| | DRM Device: /dev/dri/card1 |
| | Function Type: physical |
+-----------+--------------------------------------------------------------------------------------+
| 2 | Device Name: Intel(R) Arc(TM) Pro B60 Graphics |
| | Vendor Name: Intel(R) Corporation |
| | SOC UUID: 00000000-0000-0003-0000-0000e2118086 |
| | PCI BDF Address: 0000:03:00.0 |
| | DRM Device: /dev/dri/card2 |
| | Function Type: physical |
+-----------+--------------------------------------------------------------------------------------+
| 3 | Device Name: Intel(R) Arc(TM) Pro B60 Graphics |
| | Vendor Name: Intel(R) Corporation |
| | SOC UUID: 00000000-0000-0004-0000-0000e2118086 |
| | PCI BDF Address: 0000:04:00.0 |
| | DRM Device: /dev/dri/card3 |
| | Function Type: physical |
+-----------+--------------------------------------------------------------------------------------+
Timestamp, DeviceId, GPU Utilization (%), GPU Power (W), GPU Frequency (MHz), GPU Core Temperature (Celsius Degree), GPU Memory Temperature (Celsius Degree), GPU Memory Utilization (%), GPU Memory Used (MiB), PCIe Read (kB/s), PCIe Write (kB/s)
16:23:46.320, 0, 0.07, 43.84, 1550, N/A, 52.00, 5.64, 1382.05, 1174, 105575
16:23:46.320, 1, 0.00, 43.67, 1550, N/A, 52.00, 0.89, 219.95, 1174, 105575
16:23:46.320, 2, 0.00, 43.67, 1550, N/A, 52.00, 1.02, 219.95, 1174, 105575
16:23:46.320, 3, 0.00, 43.85, 1550, N/A, 52.00, 1.02, 219.89, 1174, 105575

Single Process Unidirectional tests

---

Bandwidth Write : Device( 0 )->Device( 1 )
Devices 0 and 1 do not have P2P capabilities
terminate called without an active exception

[james-tang17]

Please check if you have disabled the PCIe ACS in BIOS (access control service) feature and IOMMU (can paste your /proc/cmdline)

[mwrothbe]

Thanks for the fast reply. I don't see anything in the BIOS like "ACS". I did confirm that Vt-D is enabled. My board is asus pro ws w790e-sage se . Here is the /proc/cmdline for the proxmox os: initrd=\EFI\proxmox\6.17.4-2-pve\initrd.img-6.17.4-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

[james-tang17]

If possible, please try to disable vt-d and also intel iommu (intel_iommu=off in /proc/cmdline).

Also since you are using proxmox OS which we didn't verify before, could you please also attach the kernel config file located in /boot/config-6.17.4-2-pve? I want to check whether PCIe P2P is enabled by default in kernel side.

[mwrothbe]

Thanks. Attached is the config-6.17.4-2-pve file. I can give disabling vt-d in bios a try, but I think what will happen then is the Linux VM won't see the GPUs at all, and I'm running the scripts on the Linux VM. I may be doing this wrong, so let me describe in detail my setup.

Here's the config file:
config-6.17.4-2-pve.txt

BIOS:
PCI slots: Bifurcated (required for dual GPU cards)
Vt-D: Enabled
Extended APIC: Enabled

Proxmox OS (after system reboot):
as per instructions here: https://pve.proxmox.com/wiki/PCI_Passthrough
dmesg | grep -e DMAR -e IOMMU :
... [ 0.302624] DMAR: IOMMU enabled ...
pvesh get /nodes/proxmox/hardware/pci --pci-class-blacklist "" :
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_device │
│ 0x030000 │ 0xe211 │ 0000:36:00.0 │ 33 │ 0x8086 │ Battlemage G21 [Arc Pro B60] │ │ 0x806c │
│ 0x030000 │ 0xe211 │ 0000:3a:00.0 │ 38 │ 0x8086 │ Battlemage G21 [Arc Pro B60] │ │ 0x806c │
│ 0x030000 │ 0xe211 │ 0000:54:00.0 │ 20 │ 0x8086 │ Battlemage G21 [Arc Pro B60] │ │ 0x806c │
│ 0x030000 │ 0xe211 │ 0000:58:00.0 │ 25 │ 0x8086 │ Battlemage G21 [Arc Pro B60] │ │ 0x806c │

Then I assign the resources to the VM.

at this point, the linux VM seems to see the GPUs:

but when I run the eval script in the VM (after already running the install script), I get:

interestingly, after running the eval script in the VM, going back to Proxmox OS and re-running dmesg | grep -e DMAR -e IOMMU as before, I no longer see "IOMMU enabled" which I don't understand.

Any thoughts or ideas? Should I try disabling the techs in bios? I assume I should be running the scripts in the VM and not in the proxmox OS (which is not ubuntu.) Thanks!

[james-tang17]

Sorry for the late feedback. We only run it natively with Ubuntu.

Now understand your topo, Proxmox OS + Linux VM should be more complicated case which may requires the P2P support from many levels such as:

• BIOS: disable PCIe ACS (Access Control Service), enable PCIe 4G beyond decoding.

• Proxmox OS:
  kernel support: i checked your kernel config, CONFIG_PCI_P2PDMA=y is the correct configuration
  6.17 kernel with xe driver, should be as new as possible, i think no issue

• Linux VM
  Kernel support: seems you have installed with our offline installer, so the kernel should be 6.14.0-1008-intel, it's ok.
  UMD driver: i see you are using 35563.4 which is ok

Based on above, what gap I can see right now is about ACS disabling in BIOS or how you pass through the GPU to VM.

Lastly, have you ever tried to run these test with Ubuntu natively?

[mwrothbe]

Thanks for the reply! There is nothing in the BIOS like "ACS". Potentially related settings are: Extended APIC, Vt-d, and SR-IOV. But really not sure which, if any, of these are related. 4G and beyond is enabled. Also, I am passing through the GPUs to the VM. That's what the first and the second screenshot above show. In the Proxmox OS, I've blacklisted the XE driver, so it should be a direct passthrough to the VM.

I have not tried running Ubuntu natively on the server. That would be too much for me to try as I have many services on Proxmox that I can't interrupt.

Thanks for your help! I understand you can only support my particular environment so far.