Feature Request: Additional commandline arguments for llama.cpp (custom offloading MoE Layers)

[MichK-tech]

Normally, -ot ".ffn_.*_exps.=CPU" offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU" This offloads up and down projection MoE layers.

Try -ot ".ffn_(up)_exps.=CPU" if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example -ot ".(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(gate|up|down)_exps.=CPU" means to offload gate, up and down MoE layers but only from the 6th layer onwards.

From here: https://docs.unsloth.ai/models/gpt-oss-how-to-run-and-fine-tune#improving-generation-speed

Please also consider: The latest llama.cpp release also introduces high throughput mode. Use llama-parallel. Read more about it here. You can also quantize the KV cache to 4bits for example to reduce VRAM / RAM movement, which can also make the generation process faster.

[fuutott]

-n-moe-cpu is coming at some point in near future.

I'd suggest to change this to feature request for high throughput mode.

[MichK-tech]

It is not only the number of experts but more, which experts should be offloaded. The originial request is 3 months old, even in the linked reddit thread some guy states 'Hope the LM Studio devs read this. Please just give us -ot with the ability to set a custom regexp, or, even better, provide the ability to override CLI args. Make life easier for yourselves and users, it’s not sustainable to expose all possible args as nice UI inputs or elements. Just drop in an “arg override” text field with a disclaimer.' - which was also my suggestion in braces.

So better change the feature request to: Override lama.cpp arguments in developer mode. - Maybe with a hint, which are the current arguments coming from UI...