Fix TransformersModelManager: Prevents crash by dispatching model on all GPUs

[vectrozz]

Enable multi GPU usage in swicth_model function. Free memory properly in clear_model function

[TheGhoul21]

Thanks for the PR. I see that the previous behaviour is not maintained, if you change this we can proceed!
The main problem is that in our current code we preload all models on the CPU and then we use CUDA streams to move them on GPU when a new request arrives. In your updated code instead of moving the model from CPU to GPU you basically only load the model in the GPU.

By the way, the approach is correct and perfect, if you can make this change it'd be perfect.

To debug and test you can make multiple requests to different models and verify the time it takes to clear and load the model: it should not take more than 2-3 seconds (that is actually a lot)

[vectrozz]

Thank you for your reply.
I’ll make the changes to preserve the current behavior if I can find a way.
However, I’m unable to test the model switch — DeepSeek responds the same way regardless of the model I specify.

Whatever the model I call (casperhansen/mistral-small-24b-instruct-2501-awq , Qwen/QwQ-32B-AWQ ,
deepseek-ai/DeepSeek-R1-0528-Qwen3-8B or SentientAGI/Dobby-Mini-Unhinged-Llama-3.1-8B) I get always the same response from DeepSeek

Example :

curl -X POST http://192.168.x.xxx:8888/run
-H "Content-Type: application/json"
-H "Accept: application/json"
-H "User-Agent: UOMI-Client/1.0"
-d '{
"model": " SentientAGI/Dobby-Mini-Unhinged-Llama-3.1-8B",
"input": {
"messages": [
{
"role": "user",
"content": "hello, who are you ? "
}
]
}
}'

==> Deepseek response

[amashmendis]

Thanks for the PR. I see that the previous behaviour is not maintained, if you change this we can proceed! The main problem is that in our current code we preload all models on the CPU and then we use CUDA streams to move them on GPU when a new request arrives. In your updated code instead of moving the model from CPU to GPU you basically only load the model in the GPU.

By the way, the approach is correct and perfect, if you can make this change it'd be perfect.

To debug and test you can make multiple requests to different models and verify the time it takes to clear and load the model: it should not take more than 2-3 seconds (that is actually a lot)

Thank you