Using AMD GPUs

[taske666]

Hi,

I am a very inexperienced user if it comes to LLMs in general.

I tried to follow the guide as close as I could, but I realized CPU is getting utilized instead GPU.
Got an AMD 9070XT and, as I wasnt sure if it was needed, also installed the ROCm SDK.

Is it not possible to use an AMD GPU for Qwen2.5 or where can I find the config file for which hardware should be utilized?

kind regards

[t41372]

LLM inference is not a part of the Open-LLM-VTuber. If you are following the quick guide, I assume that you are using ollama for your LLM inference.

I did some digging, and apparently ollama does not support your GPU on Windows as of now. Once these support got merged, you will probably be able to use GPU acceleration in ollama.
ollama/ollama#10430
ollama/ollama#9633

For now, you can try:

• run ollama inside wsl
• use LM Studio
• use vllm within wsl or docker
• use API for now and check the status of the issues mentioned above in the mean time

[taske666]

Thanks a lot for the answer.

If I use LM Studio it utilizes my GPU, with the qwen2.5 7B i get up to 102 token/sec.

Is there somewhere documented how to use LM Studio instead Ollama?

I read it a few times but couldn't find an answer

Edit: I meant how to use LM Studio as backend (if that's the right term) in LLM Vtuber

[t41372]

Yeah I just realized we don't have that in our doc. We should have one. It's pretty easy though, there is an option called llm_provider. Set that to lm Studio. There is also some options named lm Studio in conf yaml, you can change relevant settings there.

In lm Studio, you might have to switch to dev mode and start the API server. I will add docs later today so if you don't know what I'm saying you can wait a bit

[taske666]

thanks sooo much! That way I can use my AMD GPU, will going very slowly through all the configuration now
You're a great person!

[philpw99]

I am using LM Studio. To use it, it's just a simple edit of config.yaml

1. Under Agent Config -> agent settings -> basic_memory-agent , change "llm_provider" to "lmstudio_llm"
2. Under Agent Config -> agent settings -> llm_configs -> lmstudio-llm, change "model" to the model name from LM studio, like "qwen2.5-7b-instruct".
3. Restart the app.