Eval bug: Gemma Models with QAT is not loading

[itayo1010]

Name and Version

llama-cli --version
version: 8673 (8df5be1)
built with GNU 14.2.0 for Linux aarch64

Operating systems

Linux

GGML backends

CPU

Hardware

its the Radxa rock 5t with armbian.

Models

gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf

Problem description & steps to reproduce

Model I have tried to load :
gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf

While running this commands :
ulimit -n 65536
taskset -c 4-7 ./rk-llama.cpp/build/bin/llama-cli -m ./gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -t 4 --temp 0 -c 512 --no-mmap -p "What is the meaning of life?"

I have got this error :

Loading model... /llama_model_load: error loading model: missing tensor 'blk.15.attn_k.weight'
llama_model_load_from_file_impl: failed to load model
|llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
-llama_model_load: error loading model: missing tensor 'blk.15.attn_k.weight'
llama_model_load_from_file_impl: failed to load model
/common_init_from_params: failed to load model './gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf'
srv load_model: failed to load model, './gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf'
Failed to load the model

First Bad Commit

No response

Relevant log output

I will reproduce if needed

[javer]

@itayo1010 You need the recent changes in the upstream to make it work.

You can use my fork, which is based on the much newer version b9538 of the upstream, unless this repo gets all the recent changes from the upstream.

build      : b9555-b023bca90
model      : gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
modalities : text

[danielferr85]

Thank you @javer

Yout fork works great for gemma 4 QAT models. Also for MTP models...

[romancone]

@javer Does it work with the 4QAT and MTP model?

[danielferr85]

Hi @romancone . I use the fork from @javer for the QAT version. Works greats. MTP also works but I didint notice any speed improvement (remenber to pass the argument to activate mtp in llama.cpp)

[javer]

@romancone Yes, it has the same capabilities as the upstream.

Here is the branch updated to the latest (today) changes (release b9782): https://github.com/javer/rk-llama.cpp/tree/b9782-rknpu2

[javer]

@danielferr85 I wouldn't say that there is no speed improvement with MTP, because I see 1.5-2x boost from MTP on Orange Pi 5 Plus.

Gemma 4 12B qat NO MTP:

$ RKNPU_HYBRID="W8A8_STANDARD" taskset -c 4-7 ./llama-server --host 127.0.0.1 --port 8888 --model ~/Models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf --threads 4 --threads-http 2 --flash-attn on --ctx-size 65536 --kv-unified --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap --temperature 1.0 --top-k 64 --top-p 0.95 --min-p 0.0

$ ./mtp-bench.py --url http://127.0.0.1:8888
  code_python        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
  code_cpp           pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
  explain_concept    pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
  summarize          pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
  qa_factual         pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
  translation        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=2.1
...

Gemma 4 12B qat WITH MTP:

$ RKNPU_HYBRID="W8A8_STANDARD" taskset -c 4-7 ./llama-server --host 127.0.0.1 --port 8888 --model ~/Models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf --threads 4 --threads-http 2 --flash-attn on --ctx-size 65536 --kv-unified --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap --temperature 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --model-draft ~/Models/mtp-gemma-4-12B-it.gguf --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

$ ./mtp-bench.py --url http://127.0.0.1:8888
  code_python        pred= 192 draft= 206 acc= 139 rate=0.675 tok/s=4.3
  code_cpp           pred= 192 draft= 238 acc= 131 rate=0.550 tok/s=3.7
  explain_concept    pred= 192 draft= 282 acc= 120 rate=0.425 tok/s=3.2
  summarize          pred= 192 draft= 199 acc= 140 rate=0.704 tok/s=4.4
  qa_factual         pred= 192 draft= 249 acc= 128 rate=0.514 tok/s=3.6
  translation        pred= 192 draft= 212 acc= 137 rate=0.646 tok/s=4.2
  creative_short     pred= 192 draft= 271 acc= 121 rate=0.447 tok/s=3.2
  stepwise_math      pred= 192 draft= 219 acc= 135 rate=0.616 tok/s=4.0
  long_code_review   pred= 192 draft= 254 acc= 126 rate=0.496 tok/s=2.8

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 2130,
  "total_draft_accepted": 1177,
  "aggregate_accept_rate": 0.5526,
  "wall_s_total": 537.0
}

NPU in this case is almost all the time loaded at ~83% all 3 cores.

Such a large boost is actual only for dense models. On MoE Qwen 3.6 35B-A3B I see only ~10% boost from MTP.

[danielferr85]

Yes! You are right! I forgot to mention that I only tested mtp for MoE models!

[akohlsmith]

@itayo1010 You need the recent changes in the upstream to make it work.

You can use my fork, which is based on the much newer version b9538 of the upstream, unless this repo gets all the recent changes from the upstream.

build      : b9555-b023bca90
model      : gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
modalities : text

your fork does not seem to work for me. I had to hack up the build because the web ui assets aren't available anymore, but when I try to load gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf with the MTP (or other gemma models) I get the fatal error llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4-assistant'

Nevermind, I was linking against the old rk-llama.cpp libraries. When I reset LD_LIBRARY_PATH correctly I have the same results as you.