Add -DGGML_USE_VMM=ON to build workflow to fix ROCm VRAM retention / OOM on RDNA 3.5/4

[Aaronontheweb]

Operating System

Ubuntu 24.04

OS Details

Kernel 6.14.0-37-generic

GPU Model

AMD Radeon AI PRO R9700 (gfx1201, RDNA 4), two of them in the same host.

llamacpp-rocm Build Version

b1245

Model Used

unsloth/gemma-4-26B-A4B-it-GGUF (gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf plus the BF16 mmproj).

Does this issue also occur with Vulkan on upstream llama.cpp?

Not tested locally. Issue #79 does report that switching to Vulkan fixes the same class of OOM on an RX 9060 XT (gfx1200), which suggests the ROCm allocator path is the problem, not the model or flash-attention code.

Issue Description

Your binaries print NO_VMM = 1 in system_info on every startup, regardless of GPU:

system_info: ... | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | ...
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 32624 MiB

That's because build-llamacpp-rocm.yml doesn't pass -DGGML_USE_VMM=ON to cmake. Without it, the VMM-backed pool allocator (ggml_cuda_pool_vmm in ggml/src/ggml-cuda/ggml-cuda.cu) gets #if defined(GGML_USE_VMM)'d out of libggml-hip.so. The NO_VMM = 1 string is literally printed from the matching #ifndef block in the same file.

The fallback pool (which is what you're actually shipping) runs on direct hipMalloc. It grows on demand but never gives pages back, so each new peak in flash-attn scratch or KV cache permanently raises the process's resident VRAM floor. Leave it running long enough with varied load and a routine allocation trips the ceiling:

/path/to/ggml-cuda.cu:97: ROCm error
ROCm error: out of memory
  current device: 0, in function alloc at .../ggml-cuda.cu:430
  ggml_cuda_device_malloc(&ptr, look_ahead_size, device)

Our stack traces go through ggml_cuda_flash_attn_ext_tile_case<512, 512>, which is Gemma 4's 512-dim global attention kernel.

Reproduction on our 2x R9700 box under real agent traffic: fresh process is sitting at 17 GB of VRAM (model + mmproj, which is what we'd expect). After roughly 170 requests have come and gone with zero slots currently active, the process is still parked on 24 GB. That's 7 GB the allocator is holding with nothing actually running. Next concurrent long-prompt burst asks for another 4-8 GB, blows past 32 GB, process core-dumps. systemd restart brings it back to 17 GB and we repeat the cycle every 60-90 minutes.

This seems to be the same underlying problem as issues #79, #52, and #81.

Additional Information

The fix is one line in both the Linux and Windows cmake blocks of build-llamacpp-rocm.yml:

   -DGGML_HIP=ON \
+  -DGGML_USE_VMM=ON \
   -DGGML_CUDA_FORCE_CUBLAS=OFF \

It shouldn't be a breaking change. Even when VMM is compiled in, llama.cpp does a per-device runtime check against hipDeviceAttributeVirtualMemoryManagementSupported and falls back to the legacy pool if the device doesn't claim support, so older GPUs that can't do VMM keep their current behavior.

I wrote a small HIP probe to make sure the runtime actually supports VMM on our hardware before suggesting this:

hipDeviceGetAttribute(&v, hipDeviceAttributeVirtualMemoryManagementSupported, 0);
// hipSuccess, v = 1
hipMemGetAllocationGranularity(&g, &prop, hipMemAllocationGranularityRecommended);
// hipSuccess, g = 4096

Both R9700s report VMM supported with 4096-byte granularity on ROCm 7.2.26015. So the flag flip should help RDNA 3.5/4 targets (gfx1151, gfx1150, gfx120X) right away, which is where most of the memory-pressure reports seem to be coming from.

For reference, the VMM pool in llama.cpp reserves a 32 GB virtual address range per device with cuMemAddressReserve, then maps physical pages on demand via cuMemCreate / cuMemMap and returns them with cuMemRelease. That's the behaviour your current builds don't have access to.

Happy to open the PR if you want it.

[Aaronontheweb]

Update: I went ahead and tested the proposed fix on the host that filed this. Built llama.cpp master (commit 606fa42) with -DGGML_HIP_NO_VMM=OFF -DGPU_TARGETS=gfx1201 -DAMDGPU_TARGETS=gfx1201 against system ROCm 7.2.26015, dropped the resulting llama-server + libggml*.so into place, and ran real traffic. The build picks up VMM — system_info shows VMM: yes, no NO_VMM = 1 line — and the very first real request reproducibly hangs the GPU. Rolled back to the Lemonade b1245 binaries after capturing the evidence. I don't think #87 should be merged as proposed.

Reproduction: gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf + BF16 mmproj, first chat completion with a ~8K-token prompt triggers it. Process state after the hang: the main thread sits at 91.6% CPU pegged in rocr::core::InterruptSignal::WaitRelaxed inside libhsa-runtime64.so.1, waiting for a GPU completion signal that never arrives. GPU 0's shader clock drops to 41 MHz while the CPU spins (so nothing is actually computing). 19 worker threads are all idle waiting on the main thread to post a result. /health still returns 200 because the handler is trivial, but /slots hangs within seconds because the slot manager is parked behind the stuck main thread.

dmesg is where the story gets concrete:

amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:40 vmid:8 pasid:32774)
  Process llama-server pid 127765 thread llama-server pid 127765
  in page starting at address 0x00007fd420000000 from client 10
  GCVM_L2_PROTECTION_FAULT_STATUS:0x00841051
  Faulty UTCL2 client ID: TCP (0x8)
  MORE_FAULTS: 0x1
  WALKER_ERROR: 0x0
  PERMISSION_FAULTS: 0x5
  MAPPING_ERROR: 0x0
  RW: 0x1

Repeats on consecutive 4K pages (0x00007fd420000000, 0x00007fd420001000, ...), preceded by amdgpu: Freeing queue vital buffer, queue evicted entries as the kernel scheduler boots the hung queues. PERMISSION_FAULTS: 0x5 combined with MAPPING_ERROR: 0x0 says the pages are mapped into the GPU's VA space but the shader has no permission to read them. TCP (Texture Cache Processor) is the shader-level client issuing the failing read, on the flash-attention / KV-cache data path. The CPU-side hang in WaitRelaxed is just downstream — the kernel was dispatched, the hardware page-faulted before the shader ran anything useful, the kernel aborted silently, and so the HSA signal never fires.

My guess is ggml_cuda_pool_vmm in ggml/src/ggml-cuda/ggml-cuda.cu is producing mappings that cuMemMap accepts but that aren't getting the right access granted on gfx1201 when the shader tries to read. Either a cuMemSetAccess call is missing after the map, or it's there with flags that don't translate to what RDNA 4's page tables actually enforce. I haven't read enough of the allocator to know which, but the page-fault pattern points squarely at the mapping layer.

Worth noting that #79 looks like the same class of problem on gfx1200 under the current non-VMM build, and that reporter says Vulkan handles the same workload fine. Vulkan doesn't route through cuMemMap for this, so that's consistent. My best read is that RDNA 4 + ROCm 7.13a just doesn't grant shader access correctly for cuMemMap-mapped pages right now, and flipping the default would ship a regression for every RDNA 4 user who picks up the next build.

Given that, I'd walk back my original suggestion. Either leave the default alone (same as today), or gate it by AMDGPU_TARGETS so RDNA 4 stays off while archs that actually work get VMM on, or land a fix in ggml_cuda_pool_vmm first and flip the default afterwards. Any of those is fine by me — just not the one-line flip I originally proposed.

I'll open a narrower issue on ggml-org/llama.cpp with the same page-fault dump, since the allocator-side fix belongs there rather than here. Sorry for the bad initial recommendation.

[Aaronontheweb]

sorry for the LLM generated comment, but the TL;DR; is there's probably an upstream issue in llama.cpp that makes my proposed fix not workable for ROCm right now, based on the evidence I gathered off of my AMD rig.

[issac-cyber]

my 2x 9070xt never work on llama cpp rocm in tensor spilt, however my rocm built did work at DGGML_CUDA_NO_PEER_COPY=ON, and only work at -sm layer, no luck at -sm row or tensor

[Aaronontheweb]

I'm running a full copy of the model on each instance, no tensor parallelism

[danielholanda]

@ppanchad-amd can you take a look?