High Latency Observed in Memory Copy Operations

[Kepontry]

Hi, jwludzik,

This is more of a question than an issue, but since discussions aren’t enabled for this repo, I’m posting it here.

I attempted to measure memory copy latency by sequentially copying a large array using zeCommandListAppendMemoryCopy and recording timestamps. The test accesses the first 64B of each 4KB page. The results showed approximately 170 timestamps for each access, and with each timestamp corresponding to 52ns on my machine, this translates to 8.84µs per copy.

I'm benchmarking the LNL NPU and using the v1.13.0 driver. Could there be an issue with my approach? Any insights would be greatly appreciated!

[jwludzik]

Hi @Kepontry, let's keep this discussion in "Issues" for now.

Keep in mind that LNL has power-saving features that may lower the NPU performance. You might want to add a warm-up workload to allow the NPU to operate at a higher clock frequency.

Can you explain what your pipeline looks like? Specifically, how do you use zeCommandList and zeCommandQueue?

What do you mean by 'recording timestamps'? Are you referring to the use of zeCommandListAppendWriteGlobalTimestamp?

[Kepontry]

Hi jwludzik,

Thank you for your quick response, and I apologize for the lack of clarity in my previous message.
I also observed the warm-up period during testing and have set the DVFS mode to performance using: echo 1 > /sys/kernel/debug/accel/0000:00:0b.0/dvfs_mode.

For my testing, I used the MeasureCommandCopyUsingTimestamp test in umd-test and modified the copy data size in zeCommandListAppendMemoryCopy to 64B. Additionally, I set numOfCopyCommands = 1024. The results showed that it costs 161 VPU clock units per command.

Moreover, I noticed that while the timestampFreq in the driver for the NPU timestamp is 26ns per clock unit (correspond to 38,400,000Hz), it appears to be 52ns per clock on my machine (Yoga Slim 7 15ILL9). I verified this by using zeDeviceGetGlobalTimestamps and comparing it with the host timestamp (1ns per clock). The 400,000,000Hz one also behaves not as expected and is 5ns per clock unit.

[jwludzik]

Please, keep in mind that dvfs_mode is only a hint for processor

For my testing, I used the MeasureCommandCopyUsingTimestamp test in umd-test and modified the copy data size in zeCommandListAppendMemoryCopy to 64B. Additionally, I set numOfCopyCommands = 1024. The results showed that it costs 161 VPU clock units per command.

The 64B is very small chunk of memory to be copied by DMA engine in NPU. To get more from NPU you should use as big chunk of memory as possible. Using 64B copy size you are wasting the NPU cores to handle a lot of small commands. You should be able to increase the size from 64B to (16KB - 1) to use DMA fully

Moreover, I noticed that while the timestampFreq in the driver for the NPU timestamp is 26ns per clock unit (correspond to 38,400,000Hz), it appears to be 52ns per clock on my machine (Yoga Slim 7 15ILL9). I verified this by using zeDeviceGetGlobalTimestamps and comparing it with the host timestamp (1ns per clock)

This is strange. We lately merge a fixed that decrease by half the value of timer resolution.

linux-npu-driver/umd/vpu_driver/source/device/vpu_40xx/vpu_hw_40xx.cpp

Line 46 in b4d5ffb

.timerResolution = PERF_FREQUENCY_DEFAULT_HZ / 2,

Where PERF_FREQUENCY_DEFAULT_HZ is defined in hw_info.hpp

linux-npu-driver/umd/vpu_driver/source/device/hw_info.hpp

Line 23 in b4d5ffb

constexpr uint64_t PERF_FREQUENCY_DEFAULT_HZ = 38'400'000;

Are you sure that the "linux-npu-driver" installed in your system is in 1.13.0 version?

The 400,000,000Hz

Is it a "CoreClockRate" from ze_device_properties_t structure? If yes, there is a similar issue as I described above in kernel driver. The "CoreClockRate" should be divided by 2. But the "CoreClockRate" should not be used to compute the timestamp

[Kepontry]

Thanks, Jozef,

You should be able to increase the size from 64B to (16KB - 1) to use DMA fully

Yes, DMA latency should be measured during a large memory copy operation, it's my mistake.

Are you sure that the "linux-npu-driver" installed in your system is in 1.13.0 version?

In version 1.13.0, the calculated timestampFreq is actually 52. I had recently upgraded the driver from 1.10.0 to 1.13.0 but was still using the old value by mistake.

The "CoreClockRate" should be divided by 2.

You're right — the 400 MHz frequency actually corresponds to the PLL_PROFILING_FREQ_HIGH macro in the KMD. The actual frequency should also be divided by 2.