Multiple CUDA GPU's not being utilised

[michaelarmstrong]

Hi, without setting up separate distributed inference instances I cannot seem to achieve a single instance of ds4-server utilising multiple CUDA capable GPU's on a single system (use case is dual RTX Pro 6000). Please advise if this is currently supported or not as I'd like to look into it if not.

[btothec]

I run this on one RTX Pro 6000 with the command below:

CUDA_VISIBLE_DEVICES=0
DS4_CUDA_COPY_MODEL=1
DS4_CUDA_NO_Q8_F16_CACHE=1
DS4_METAL_PREFILL_CHUNK=2048
./ds4-server
--cuda
-m ./ds4flash.gguf
--ctx 450000
--host 0.0.0.0
--port 8002
--cors
--kv-disk-dir ~/.ds4/server-kv
--kv-disk-space-mb 65536
--warm-weights

To be honest I run vllm / sglang 99% of the time so I am not sure, but I think if you just pass CUDA_VISIBLE_DEVICES=0,1 and the rest of my launch command you should be fine.