llama-rocm Exception Code: 0xC0000005 2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE41F24,

[vugenti]

This happens every few hours. When it happens lemonade thinks the server is still running and will continue sending queries. Unload / reload the model to return to normal behavior for several more hours:

2026-03-17 22:56:25.643 [Info] (Process) slot update_slots: id  3 | task 26076 | prompt processing progress, n_tokens = 18432, batch.n_tokens = 2048, progress = 0.943634
2026-03-17 22:56:55.653 [Info] (Server) Model already loaded: Qwen3.5-122B-A10B-GGUF
2026-03-17 22:56:55.653 [Info] (Server) POST /api/v1/chat/completions - Streaming
2026-03-17 22:56:55.771 [Info] (Process) srv  params_from_: Chat format: peg-constructed
2026-03-17 22:57:01.930 [Info] (Process) slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = 3346841494
2026-03-17 22:57:01.930 [Info] (Process) srv  get_availabl: updating prompt cache
2026-03-17 22:57:01.930 [Info] (Process) srv   prompt_save:  - saving prompt with length 172, total state size = 153.097 MiB
2026-03-17 22:57:10.971 [Info] (Process) srv          load:  - looking for better prompt, base f_keep = 0.017, sim = 0.000
2026-03-17 22:57:10.972 [Info] (Process) srv          load:  - found better prompt with f_keep = 0.259, sim = 0.408
2026-03-17 22:57:10.978 [Info] (Process) Exception Code: 0xC0000005
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE41F24, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama.dll(0x00007FF95FDD0000) + 0x71F24 byte(s), llama_log_set() + 0x9CE4 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE5226F, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama.dll(0x00007FF95FDD0000) + 0x8226F byte(s), llama_log_set() + 0x1A02F byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE51E24, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama.dll(0x00007FF95FDD0000) + 0x81E24 byte(s), llama_log_set() + 0x19BE4 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE5CAB4, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama.dll(0x00007FF95FDD0000) + 0x8CAB4 byte(s), llama_log_set() + 0x24874 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF95FE0B370, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama.dll(0x00007FF95FDD0000) + 0x3B370 byte(s), llama_state_seq_set_data_ext() + 0x60 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF7019B6E1E, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x386E1E byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF7016CE400, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x9E400 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF7016CB950, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x9B950 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF70185723D, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x22723D byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF701634293, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x4293 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FF701AF84BC, C:\Users\Vincent\.cache\lemonade\bin\llamacpp\rocm\llama-server.exe(0x00007FF701630000) + 0x4C84BC byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FFA2C06E8D7, C:\windows\System32\KERNEL32.DLL(0x00007FFA2C040000) + 0x2E8D7 byte(s), BaseThreadInitThunk() + 0x17 byte(s)
2026-03-17 22:57:10.991 [Info] (Process) 0x00007FFA2E0CC48C, C:\windows\SYSTEM32\ntdll.dll(0x00007FFA2E040000) + 0x8C48C byte(s), RtlUserThreadStart() + 0x2C byte(s)
2026-03-17 22:57:24.290 [Info] (Server) Model already loaded: Qwen3.5-122B-A10B-GGUF
2026-03-17 22:57:24.290 [Info] (Server) POST /api/v1/chat/completions - Streaming
2026-03-17 22:57:47.156 [Info] (Server) Model already loaded: Qwen3.5-122B-A10B-GGUF
2026-03-17 22:57:47.156 [Info] (Server) POST /api/v1/chat/completions - Streaming
2026-03-17 22:58:30.166 [Info] (Server) Model already loaded: Qwen3.5-122B-A10B-GGUF

[humb1t]

Not 100% the same, but my local lemonade + llama-server also sometimes fail to clean up unused models, eat the whole RAM and make PC freeze.

[vugenti]

When I see this happen (on Windows 11) the RAM is deallocated from the model, so I don't have any issues with OS running out of memory when this happens. Seems like llamacpp process may have died but the component managing the llamacpp lifetime doesn't notice.

[humb1t]

I see that every new model load via lemonade increases the llama-server RAM consumption by 2-3 Gb, so it seems like some clean up is missing. Do you same behavior on Windows (because I use arch btw)

[jeremyfowers]

@danielholanda

[danielholanda]

[Transfered issue here so we can triage it better]

@vugenti I understand this happens when using lemonade + llama-server. Are you able to confirm this only happens when ROCm is used? Or do you see a similar behavior when using Vulkan?

[vugenti]

[Transfered issue here so we can triage it better]

@vugenti I understand this happens when using lemonade + llama-server. Are you able to confirm this only happens when ROCm is used? Or do you see a similar behavior when using Vulkan?

I haven't really tried much with vulkan so I can't say for certain. But based on the logs I think there are 2 distinct issues:

• 0xc0000005 from llamacpp - not sure if this is rocm-specific or whether it happens with vulkan. I don't really use vulkan
• lemonade-server / lemonade-router / whatever component manages the lifetime of the backend process doesn't realize it has exited and keeps sending requests indicating "model already loaded" (but the model has clearly been unloaded from VRAM when this is happening) - this is potentially happening with any backend that abends.

[danielholanda]

Issue 1 - 0xc0000005 from llamacpp

I can confirm I can run Qwen3.5-122B-A10B-GGUF runs well on my STX Halo (gfx1151). Are you also running this on a RyzenAI Max+ 395? Can you share additional information about your system?

@benrichard-amd is this something you could help investigate?

Issue 2 - "model already loaded" when model is unloaded

Created an issue back on Lemonade for this sub issue lemonade-sdk/lemonade#1510

[vugenti]

Issue 1 - 0xc0000005 from llamacpp

I can confirm I can run Qwen3.5-122B-A10B-GGUF runs well on my STX Halo (gfx1151). Are you also running this on a RyzenAI Max+ 395? Can you share additional information about your system?

The model does run perfectly fine the vast majority of the time. I do not experience this issue every day.

It is running on Ryzen AI Max+ 395, 128GB RAM, Windows 11 Pro configured for 64/64 split

Perhaps of note - I was previously using this driver amd-software-adrenalin-edition-25.20.01.17-win11-pytorch-combined based on directions I found at https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-WINDOWS-PYTORCH-7-1-1.html where it is described as AMD Software Driver Version 25.20.01.17 for Windows® 11 (Windows Driver Store Version 32.0.22001.17002) - but it looks like I currently have installed Driver Version 25.20.18-250917a-419976C-Sixunited (AMD Windows Driver Version 32.0.22018.5) - is this ok / could it be contributing?

[danielholanda]

@ppanchad-amd Please take a look as this seems ROCm/driver related. Providing guidance here would be great.

[tcgu-amd]

He @vugenti , sorry for the late response. Which version of Lemonade and llama.cpp are you running? Better, do you know the ROCm version? How often does this issue occur and has it occurred recently? Thanks!

[mapatel-amd]

Hey all, I noticed this issue and as @danielholanda said, this is a driver issue I believe due to the exception code 0xC0000005. This has to do with the OpenCL backend, specifically that the iGPU runtime seems to be running through the dGPU. I have filed a ticket with the driver team. If someone can confirm, I believe this issue can be categorized as a duplicate of this issue. There is some more information within that issue as well as to what was going on. Here is a minimal reproducer:

import pyopencl as cl
platforms = cl.get_platforms()
# Platform 1 = iGPU runtime (v3652.0), Device 0 = gfx1201 (dGPU)
ctx = cl.Context([platforms[1].get_devices()[0]])
cl.CommandQueue(ctx, platforms[1].get_devices()[0])  # SEGFAULT here

As for @vugenti, this is a very random problem with the drivers. Sometimes the dGPU will be run with its own runtime (which doesn't cause the error) and sometimes it will run through the iGPU runtime (which causes the issue). Either platform runtime can be selected so I have 2 current recommendations for you to try:

1. Turn off integrated graphics in BIOS (I recommend this one)
2. Revert your adrenalin driver version to 26.3.1 (Might not have as good performance or may have other bugs, but should work for this case)

If you all have any other questions let me know, and refer to this issue for more details and the jira integration for the driver team.