QWEN3 - MOE structure - not working ?

[David-AU-github]

Hi;

Tried to build:

base_model: D:/Qwen3-0.6B
gate_mode: random
architecture: qwen
dtype: bfloat16
experts_per_token: 2
experts:

• source_model: D:/qwen3-0.6b-creative-writing
• source_model: D:/Qwen3-0.6B-dreamwriter-0.6b-beta
• source_model: D:/Qwen3-0.6B-Konjac-0.6B
• source_model: D:/Qwen3-0.6B-Josiefied-Qwen3-0.6B-abliterated-v1
• source_model: D:/Qwen3-0.6B-Qwill-0.6B-IT-FULL
• source_model: D:/qwen3-0.6b-writing

shared_experts:

• source_model: D:/Qwen3-0.6B

Using:

mergekit-moe --clone-tensors f:/mergefiles/Qwen3-MOE-6X0.6B-Creative.txt e:/Qwen3-MOE-6X0.6B-Creative

ERROR:

ERROR:root:No output architecture found that is compatible with the given models.
ERROR:root:All supported output architectures:
ERROR:root: * Mixtral
ERROR:root: * DeepSeek MoE
ERROR:root: * Qwen MoE

NOTE:
Not sure if Qwen3 supports "MOE" structure composed of "0.6B" Qwen 3 models. (??)

[David-AU-github]

@cg123

Update:

Tried with new patch/pull:

Fix construction of ModelArchitecture for mixtral/qwen3_moe (https://github.com/arcee-ai/mergekit/pull/586[)](https://github.com/arcee-ai/mergekit/commit/040374af7157edbd0765cf28a4b220da0b776ba4)

Still same error.

With Qwen 2.5s ; you could "MOE" any size together so to speak.

It looks like 30B-A3B/235B... Qwen 3 moe is "one type" whereas all the others are another type?
Use Qwen 2.5s "moe everything" script and duplicate for "other" Qwen 3s ?

NOTE: Still not sure if "other" Qwen 3s are only a specific size to "MOE" . should be
0.6B right up to 32B "reg" Qwen 3s. (???)

Here are the config diff for "reg" qwen 3s VS qwen 3 "MOE" 30B-A3B :

REG: (4b)

{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2560,
"initializer_range": 0.02,
"intermediate_size": 9728,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.0",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

30B-A3B - 30B , modified with YARN/EXP change to 4.

{
"architectures": [
"Qwen3MoeForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"decoder_sparse_step": 1,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 6144,
"max_position_embeddings": 65536,
"max_window_layers": 48,
"mlp_only_layers": [],
"model_type": "qwen3_moe",
"moe_intermediate_size": 768,
"norm_topk_prob": true,
"num_attention_heads": 32,
"num_experts": 128,
"num_experts_per_tok": 4,
"num_hidden_layers": 48,
"num_key_value_heads": 4,
"output_router_logits": false,
"rms_norm_eps": 1e-06,
"rope_scaling": {
"type": "yarn",
"factor": 2.0,
"original_max_position_embeddings": 32768
},
"rope_theta": 1000000.0,
"router_aux_loss_coef": 0.001,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.0",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

[cg123]

The issue here is that currently only the Qwen 2.5 MoE architecture is implemented. Qwen 3 MoE is different (in that it has qk norm parameters and doesn't support shared experts) and Qwen 3 models can't go into a 2.5 MoE. I'll implement it as a new target architecture pretty soon when I get the chance!

[David-AU-github]

Thank you for all you do @cg123

[stabgan]

Oh damn, we can combine multiple 0.6B models into a single mixture of experts ?
Then what is your plan ? Do we need to again train it ? or use it as it is ?
How will the gating router understand how to select which expert ?

[David-AU-github]

@stabgan

You got it, I have 24 0.6B Qwen 3s ready to go, and today added 18 Qwen 3 4Bs. ;
Just drop em in, test, adjust, gate and go.
Gating can be set per model/expert with pos and/or neg prompts.

Ready to cook!

[geoman2]

Hey David, did you read about this merging https://huggingface.co/lemon07r/Qwen3-R1-SLERP-Q3T-8B

What do you think about using tokenizers from another models, do you use the same technique?

[David-AU-github]

@geoman2

You can use / state the tokenizer to use in mergekit and/or use "union" to merge them.

As for "switching it up" ; I have not done this intentionally , except when there is oddball issues with the merge.
In that case several changes may be required - including config.json, vocab, tokenizer etc etc, special tokens and the like.
Especially if there is an issue with BOS/EOS tokens or a conflict here.

In the case of lemon07r ; there is a special reason here: Reasoning.
If you don't use the correct files/settings/token etc , reasoning will fail to activate and/or be "dodgy".
That I have run into.

[David-AU-github]

@cg123

Dying to build Qwen 3 moes ; any way I can help?

Build 81 coder models in the past 2 weeks or so ; so I have an "army of coders" on standby from .8B to 87B
MOE, Fused, Brainstorm ... all ready to cook.

I am also a programmer by trade ; just need the info/specs to modify the "MOE CODE" ...

Built some Qwen3 coders too ; just need the Qwen 3 MOEs to complete the collection.

https://huggingface.co/collections/DavidAU/coder-programming-moe-reasoning-reg-imatrix-fused-686357166b1f0d4322ad3e2c

Thank you in advance.
Additional links to Mergekit have also been added from several repos and docs.

[NextGenOP]

Is it not possible to use Mixtral?

[David-AU-github]

Is it not possible to use Mixtral?

Mistral(s), Qwen 2.5, Llama, etc etc work great ; Qwen 3 is different - no shared expert (vs Qwen 2.5) - this seems to be the issue.

[David-AU-github]

@cg123

Update; patched and created module myself, successfully created multiple working Qwen 3 MOEs.
8 up at my repo as of this writing... a lot more to come.

https://huggingface.co/DavidAU

Blistering 4x0.6Bs, 4x1.7Bs, 6x ... ; on FIRE.
Reduction in thinking tokens, and far higher intelligence to boot.
Stupid fast T/S at 200 T/S on mid range card.

Building some crazy ones too.

Shining Madness is one...

Marching forward.

NOTE: Unsure/unclear how to upload changes/files to merge - 1 file changed, 1 file to add.

[FrozzDay]

Is it not possible to use Mixtral?

Mistral(s), Qwen 2.5, Llama, etc etc work great ; Qwen 3 is different - no shared expert (vs Qwen 2.5) - this seems to be the issue.

I mean using qwen3 inside mixtral architecture. Is that not possible?

[David-AU-github]

@FrozzDay
Sorry not clear: The Qwen3 Moes mimic the Mistral MOEs in terms of structure, specifically compression of the models.
It is not using a mistral structure per say - these are Qwen 3.