Skip to content

Commit c0a206b

Browse files
authored
chore(model gallery): add qwen3-30b-a1.5b-high-speed (#5311)
Signed-off-by: Ettore Di Giacinto <[email protected]>
1 parent 01bbb31 commit c0a206b

File tree

1 file changed

+30
-0
lines changed

1 file changed

+30
-0
lines changed

gallery/index.yaml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -442,6 +442,36 @@
442442
- filename: Smoothie-Qwen3-8B.Q4_K_M.gguf
443443
sha256: 36fc6df285c35beb8f1fdb46b3854bc4f420d3600afa397bf6a89e2ce5480112
444444
uri: huggingface://mradermacher/Smoothie-Qwen3-8B-GGUF/Smoothie-Qwen3-8B.Q4_K_M.gguf
445+
- !!merge <<: *qwen3
446+
name: "qwen3-30b-a1.5b-high-speed"
447+
icon: https://huggingface.co/DavidAU/Qwen3-30B-A1.5B-High-Speed/resolve/main/star-wars-hans-solo.gif
448+
urls:
449+
- https://huggingface.co/DavidAU/Qwen3-30B-A1.5B-High-Speed
450+
- https://huggingface.co/mradermacher/Qwen3-30B-A1.5B-High-Speed-GGUF
451+
description: |
452+
This repo contains the full precision source code, in "safe tensors" format to generate GGUFs, GPTQ, EXL2, AWQ, HQQ and other formats. The source code can also be used directly.
453+
454+
This is a simple "finetune" of the Qwen's "Qwen 30B-A3B" (MOE) model, setting the experts in use from 8 to 4 (out of 128 experts).
455+
456+
This method close to doubles the speed of the model and uses 1.5B (of 30B) parameters instead of 3B (of 30B) parameters. Depending on the application you may want to use the regular model ("30B-A3B"), and use this model for simpler use case(s) although I did not notice any loss of function during routine (but not extensive) testing.
457+
458+
Example generation (Q4KS, CPU) at the bottom of this page using 4 experts / this model.
459+
460+
More complex use cases may benefit from using the normal version.
461+
462+
For reference:
463+
464+
Cpu only operation Q4KS (windows 11) jumps from 12 t/s to 23 t/s.
465+
GPU performance IQ3S jumps from 75 t/s to over 125 t/s. (low to mid level card)
466+
467+
Context size: 32K + 8K for output (40k total)
468+
overrides:
469+
parameters:
470+
model: Qwen3-30B-A1.5B-High-Speed.Q4_K_M.gguf
471+
files:
472+
- filename: Qwen3-30B-A1.5B-High-Speed.Q4_K_M.gguf
473+
sha256: 2fca25524abe237483de64599bab54eba8fb22088fc21e30ba45ea8fb04dd1e0
474+
uri: huggingface://mradermacher/Qwen3-30B-A1.5B-High-Speed-GGUF/Qwen3-30B-A1.5B-High-Speed.Q4_K_M.gguf
445475
- &gemma3
446476
url: "github:mudler/LocalAI/gallery/gemma.yaml@master"
447477
name: "gemma-3-27b-it"

0 commit comments

Comments
 (0)