Eval bug: Gemma-3n compatibility is partially broken

[IamHuskar]

Name and Version

bin/llama-cli --version
version: 7108 (4ae9216)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 for aarch64-linux-gn

Operating systems

Linux

GGML backends

CUDA, OpenCL

Hardware

rk3588 16GB memory

sudo cat /sys/kernel/debug/rknpu/version
RKNPU driver: v0.9.7

Models

https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF/tree/main

Problem description & steps to reproduce

bin/llama-server --verbose --mmproj ~/huggingface/Qwen3-VL-2B-Instruct-GGUF/mmproj-F16.gguf --host 0.0.0.0 -m ~/huggingface/Qwen3-VL-2B-Instruct-GGUF/Qwen3-VL-2B-Instruct-Q8_0.gguf

ask a question with image

First Bad Commit

No response

Relevant log output

main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
que    start_loop: processing new tasks
que    start_loop: update slots
srv  update_slots: all slots are idle
que    start_loop: waiting for new tasks
add_text: <|im_start|>user
wtf

add_text: <|vision_start|>
image_tokens->nx = 11
image_tokens->ny = 3
batch_f32 size = 1
add_text: <|vision_end|>
add_text: <|im_end|>
<|im_start|>user
what is this

add_text: <|vision_start|>
image_tokens->nx = 5
image_tokens->ny = 4
batch_f32 size = 1
add_text: <|vision_end|>
add_text: <|im_end|>
<|im_start|>user
123

add_text: <|vision_start|>
image_tokens->nx = 5
image_tokens->ny = 5
batch_f32 size = 1
add_text: <|vision_end|>
add_text: <|im_end|>
<|im_start|>assistant

srv  params_from_: Grammar: 
srv  params_from_: Grammar lazy: false
srv  params_from_: Chat format: Content-only
srv  add_waiting_: add task 0 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 0/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 0
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
slot        reset: id  3 | task -1 | 
slot launch_slot_: id  3 | task -1 | launching slot : {"id":3,"n_ctx":4096,"speculative":false,"is_processing":false}
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 1, front = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, task.n_tokens = 113
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 7, batch.n_tokens = 7, progress = 0.061947
srv  update_slots: decoding batch, n_tokens = 7
clear_adapter_lora: call
set_embeddings: value = 0
Segmentation fault (core dumped)

[IamHuskar]

./hfd.sh unsloth/gemma-3n-E2B-it-GGUF --include gemma-3n-E2B-it-UD-Q8_K_XL.gguf imatrix_unsloth.dat params templat

bin/llama-server --host 0.0.0.0 -m ~/huggingface/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-UD-Q8_K_XL.gguf

main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this)
build: 7108 (4ae9216b) with cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 for aarch64-linux-gnu
system info: n_threads = 8, n_threads_batch = 8, total_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

init: using 7 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/home/ubu/huggingface/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-UD-Q8_K_XL.gguf'
llama_model_loader: loaded meta data with 51 key-value pairs and 727 tensors from /home/ubu/huggingface/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-UD-Q8_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma3n
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma-3N-E2B-It
llama_model_loader: - kv   3:                           general.finetune str              = 3n-E2B-it
llama_model_loader: - kv   4:                           general.basename str              = Gemma-3N-E2B-It
llama_model_loader: - kv   5:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   6:                         general.size_label str              = 4.5B
llama_model_loader: - kv   7:                            general.license str              = gemma
llama_model_loader: - kv   8:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = Gemma 3n E2B It
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Google
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-3...
llama_model_loader: - kv  13:                               general.tags arr[str,6]       = ["automatic-speech-recognition", "uns...
llama_model_loader: - kv  14:                     gemma3n.context_length u32              = 32768
llama_model_loader: - kv  15:                   gemma3n.embedding_length u32              = 2048
llama_model_loader: - kv  16:                        gemma3n.block_count u32              = 30
llama_model_loader: - kv  17:                gemma3n.feed_forward_length arr[i32,30]      = [8192, 8192, 8192, 8192, 8192, 8192, ...
llama_model_loader: - kv  18:               gemma3n.attention.head_count u32              = 8
llama_model_loader: - kv  19:   gemma3n.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  20:               gemma3n.attention.key_length u32              = 256
llama_model_loader: - kv  21:             gemma3n.attention.value_length u32              = 256
llama_model_loader: - kv  22:                     gemma3n.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  23:           gemma3n.attention.sliding_window u32              = 512
llama_model_loader: - kv  24:            gemma3n.attention.head_count_kv u32              = 2
llama_model_loader: - kv  25:                   gemma3n.altup.active_idx u32              = 0
llama_model_loader: - kv  26:                   gemma3n.altup.num_inputs u32              = 4
llama_model_loader: - kv  27:   gemma3n.embedding_length_per_layer_input u32              = 256
llama_model_loader: - kv  28:         gemma3n.attention.shared_kv_layers u32              = 10
llama_model_loader: - kv  29:          gemma3n.activation_sparsity_scale arr[f32,30]      = [1.644854, 1.644854, 1.644854, 1.6448...
llama_model_loader: - kv  30:   gemma3n.attention.sliding_window_pattern arr[bool,30]     = [true, true, true, true, false, true,...
llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {{ bos_token }}\n{%- if messages[0]['r...
llama_model_loader: - kv  32:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  33:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  34:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  35:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  36:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  37:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  38:                tokenizer.ggml.eos_token_id u32              = 106
llama_model_loader: - kv  39:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  40:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  41:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  42:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  43:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  44:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  45:               general.quantization_version u32              = 2
llama_model_loader: - kv  46:                          general.file_type u32              = 7
llama_model_loader: - kv  47:                      quantize.imatrix.file str              = gemma-3n-E2B-it-GGUF/imatrix_unsloth.dat
llama_model_loader: - kv  48:                   quantize.imatrix.dataset str              = unsloth_calibration_gemma-3n-E2B-it.txt
llama_model_loader: - kv  49:             quantize.imatrix.entries_count u32              = 400
llama_model_loader: - kv  50:              quantize.imatrix.chunks_count u32              = 1326
llama_model_loader: - type  f32:  362 tensors
llama_model_loader: - type  f16:  150 tensors
llama_model_loader: - type q8_0:  215 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 6.95 GiB (13.40 BPW) 
load: printing all EOG tokens:
load:   - 106 ('<end_of_turn>')
load: special tokens cache size = 6414
load: token to piece cache size = 1.9446 MB
print_info: arch             = gemma3n
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_embd_inp       = 2048
print_info: n_layer          = 30
print_info: n_head           = 8
print_info: n_head_kv        = 2
print_info: n_rot            = 256
print_info: n_swa            = 512
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 256
print_info: n_embd_head_v    = 256
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 1.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = E2B
print_info: model params     = 4.46 B
print_info: general.name     = Gemma-3N-E2B-It
print_info: vocab type       = SPM
print_info: n_vocab          = 262144
print_info: n_merges         = 0
print_info: BOS token        = 2 '<bos>'
print_info: EOS token        = 106 '<end_of_turn>'
print_info: EOT token        = 106 '<end_of_turn>'
print_info: UNK token        = 3 '<unk>'
print_info: PAD token        = 0 '<pad>'
print_info: LF token         = 248 '<0x0A>'
print_info: EOG token        = 106 '<end_of_turn>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  7068.08 MiB
load_tensors:        RKNPU model buffer size =  3150.00 MiB
...............................
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     4.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 4096 cells
llama_kv_cache:        CPU KV buffer size =    32.00 MiB
llama_kv_cache: size =   32.00 MiB (  4096 cells,   4 layers,  4/1 seqs), K (f16):   16.00 MiB, V (f16):   16.00 MiB
llama_kv_cache_iswa: creating     SWA KV cache, size = 2560 cells
llama_kv_cache:        CPU KV buffer size =    80.00 MiB
llama_kv_cache: size =   80.00 MiB (  2560 cells,  16 layers,  4/1 seqs), K (f16):   40.00 MiB, V (f16):   40.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      RKNPU compute buffer size =   516.00 MiB
llama_context:        CPU compute buffer size =   512.00 MiB
llama_context: graph nodes  = 2733
llama_context: graph splits = 509
common_init_from_params: added <end_of_turn> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
E RKNN: [00:21:12.714] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:12.807] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:12.898] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:19.011] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:19.011] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:19.115] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:19.244] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:19.368] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:19.491] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:25.622] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:25.625] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:25.728] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:25.839] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:25.930] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:26.020] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:32.235] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:32.235] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:32.341] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:32.466] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:32.589] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:32.714] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:38.848] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:38.848] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:38.958] failed to submit!, op id: 0, op name: MatMul, flags: 0x5, task start: 0, task number: 1, run task counter: 0, int status: 0, If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
E RKNN: [00:21:39.077] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:39.180] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
E RKNN: [00:21:39.284] failed to convert fd(11) to handle, ret: -1, errno: 12, errstr: Cannot allocate memory
RKNN error -4 at /home/ubu/rk-llama.cpp/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp:375: set_io_mem B segment
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^C

[kinchahoy]

+1 I'd love to get image support working, even if the vision head runs on CPU only

[invisiofficial]

Everything is working for me on the newest version.

I am not really familiar with vision. Just downloaded Qwen3.5 2B Q8_0 and mmproj F16. The command was:

taskset -c 4-7 ./build/bin/llama-server -m ~/Projects/Qwen3.5-2B-Q8_0.gguf --mmproj ~/Projects/Qwen3.5-2B-mmproj-F16.gguf --host 0.0.0.0 --port 8080 -t 4 -c 8192 -n 2048 -ctk q8_0 -ctv q8_0 --no-cont-batching --no-repack --reasoning-budget 0 --prio 2 --poll 100 -cram 2048

Everything successfully loaded:

load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors:   CPU_Mapped model buffer size =  1911.66 MiB
load_tensors:        RKNPU model buffer size =  1793.00 MiB
...
sched_reserve:      RKNPU compute buffer size =   489.00 MiB
sched_reserve:        CPU compute buffer size =    70.28 MiB
...
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

load_hparams: projector:          qwen3vl_merger
load_hparams: n_embd:             1024
load_hparams: n_head:             16
load_hparams: n_ff:               4096
load_hparams: n_layer:            24
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     2048

--- vision hparams ---
load_hparams: image_size:         768
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels:   8192
load_hparams: image_max_pixels:   4194304

load_hparams: model size:         637.25 MiB
load_hparams: metadata size:      0.10 MiB
warmup: warmup with image size = 1472 x 1472
...
srv    load_model: loaded multimodal model, '/home/orangepi/Projects/Qwen3.5-2B-mmproj-F16.gguf'
srv    load_model: initializing slots, n_slots = 4
common_speculative_is_compat: the target context does not support partial sequence removal
srv    load_model: speculative decoding not supported by this context
slot   load_model: id  0 | task -1 | new slot, n_ctx = 8192
slot   load_model: id  1 | task -1 | new slot, n_ctx = 8192
slot   load_model: id  2 | task -1 | new slot, n_ctx = 8192
slot   load_model: id  3 | task -1 | new slot, n_ctx = 8192
srv    load_model: prompt cache is enabled, size limit: 2048 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
...
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle

Then I went to the OpenWebUI, connected to a board and chose a Qwen3.5 2B model. Started a new chat, attached an image and asked for explanation. Everything worked smoothly. Image encoding (?) was done using the CPU, PP and TG using NPU:

slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 2048.000 MiB, 8192 tokens, 2147483648 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 8192, n_keep = 0, task.n_tokens = 508
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 16, batch.n_tokens = 16, progress = 0.031496
slot update_slots: id  3 | task 0 | n_tokens = 16, memory_seq_rm [16, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 19715 ms
decoding image batch 1/1, n_tokens_batch = 484
find_slot: non-consecutive token position 16 after 15 for sequence 3 with 484 new tokens
find_slot: non-consecutive token position 16 after 15 for sequence 3 with 484 new tokens
image decoded (batch 1/1) in 5250 ms
srv  process_chun: image processed in 24966 ms
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 504, batch.n_tokens = 4, progress = 0.992126
find_slot: non-consecutive token position 41 after 16 for sequence 3 with 4 new tokens
find_slot: non-consecutive token position 41 after 16 for sequence 3 with 4 new tokens
slot update_slots: id  3 | task 0 | n_tokens = 504, memory_seq_rm [42, end)
reasoning-budget: activated, budget=0 tokens
reasoning-budget: budget=0, forcing immediately
slot init_sampler: id  3 | task 0 | init sampler, took 0.02 ms, tokens: text = 24, total = 508
slot update_slots: id  3 | task 0 | prompt processing done, n_tokens = 508, batch.n_tokens = 4
slot update_slots: id  3 | task 0 | created context checkpoint 1 of 32 (pos_min = 41, pos_max = 41, n_tokens = 504, size = 19.266 MiB)
reasoning-budget: forced sequence complete, done
srv  log_server_r: done request: POST /v1/chat/completions 192.168.0.32 200
slot print_timing: id  3 | task 0 |
prompt eval time =   26030.67 ms /   508 tokens (   51.24 ms per token,    19.52 tokens per second)
       eval time =   54297.43 ms /   584 tokens (   92.98 ms per token,    10.76 tokens per second)
      total time =   80328.10 ms /  1092 tokens
slot      release: id  3 | task 0 | stop processing: n_tokens = 1091, truncated = 0

The chat log:

> *Photo of a DD05CVSA board* Can you please tell me what is shown in this photo?

> This image is a technical diagram illustrating the **applications and connections** for a specific electronic component, a DC-DC power adapter. Here's a breakdown of what it shows:

### **The Central Component**
The main focus is a green circuit board labeled **"DD05CVSA"** and marked with a blue box containing that same text.
*   The diagram identifies this as a **5V-12V Step Up / Buck Converter** module. It is designed to convert voltage levels for LED lighting and battery-powered devices.

### **Application Scenarios (Left Side)**
On the left side of the diagram, two power sources are connected to the input (Vin) of the board:
*   **3.7V Lithium Battery:** Represented by a rectangle with `+` and `-` terminals. It has a green arrow labeled "Discharge Current 0-2A" indicating it can be used to power low-voltage devices like drones, phones, or LED strips.
*   **DC 4.5-8V Input:** An input arrow labeled "Charge current 0-1A" and "DC 4.5-8V" points from the solar panel. This indicates the board can also accept power directly from a DC power supply within this voltage range to charge batteries or charge LEDs.

### **Application Scenarios (Right Side)**
On the right side, two power consumers are connected to the output (Vout) of the board:
*   **Solar Panel:** Connected to the input side. This demonstrates using the board to charge a solar panel.
*   **LED Lighting Equipment:** Connected to the output side. This shows the board's primary use case: powering 5V-12V LED strips, downlights, or other LED lighting fixtures.

### **Key Features Highlighted**
The diagram emphasizes the board's versatility as a "Power Management Module" (PDM):
*   **Charge & Discharge:** It supports charging devices (like batteries) and lighting.
*   **Voltage Conversion:** It can step up voltage (increasing voltage) or step down voltage (decreasing voltage) to match the needs of the connected load or power source.
*   **Current Limits:**
    *   **Input:** Supports up to 1 Amp of charge current.
    *   **Output:** Supports up to 1.2 Amp of output current.

### **Summary**
In short, the photo illustrates a **5V-12V Step Up/Down Power Module (DD05CVSA)**. It shows how this single board can be used to convert power from a DC battery or solar panel into the 5V-12V range required to power various **LED lighting equipment** or charge **low-voltage lithium batteries**.

[romancone]

I ran several benchmarks

The list of working models that I already tested:

• gemma-3-1b-it
• gemma-3-4b-it
• gemma-4-E4B-it
• Qwen3.5-9B

Only gemma-3n-E4B-it produced a segmentation fault. I confirm the issue with the gemma-3n model

I noticed this message in the logs:

Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 1 to 0
Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 0 to 1

[happyme531]

I ran several benchmarks

The list of working models that I already tested:

* gemma-3-1b-it

* gemma-3-4b-it

* gemma-4-E4B-it

* Qwen3.5-9B

Only gemma-3n-E4B-it produced a segmentation fault. I confirm the issue with the gemma-3n model

I noticed this message in the logs:

Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 1 to 0
Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 0 to 1

rk-llama.cpp (rknpu2)> HYBRID_PATTERN="INT8_HARDMARD" taskset -c 4-7 ./build/bin/llama-cli -m ../gemma-3n-E4B-it-Q8_0.gguf

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8672-d90f5fea7
model      : gemma-3n-E4B-it-Q8_0.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> hello!

Hello there! 😊 

How can I help you today? I'm ready for anything - questions, stories, brainstorming, or just a friendly chat! Let me know what's on your mind.





[ Prompt: 12.6 t/s | Generation: 3.9 t/s ]

> 

Working fine for me, although INT8_STANDARD make the model produce unusable result, but won't crash.

[romancone]

@happyme531, your result seems to be for the CPU backend.
Try to test with the llama-bench instead

[invisiofficial]

I noticed this message in the logs:

Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 1 to 0
Apr 15 20:58:43 rock-5-itx kernel: RKNPU: switch iommu domain from 0 to 1

@romancone, this is normal behavior. The NPU driver is switching IOMMU domains for fitting big models.

@happyme531, your result seems to be for the CPU backend.

@happyme531, this is true. You made a typo in INT8_HADAMARD (INT8_HARDMARD). The backend encounters an unknown pipeline and rejects all tensors leaving them on the CPU.

[invisiofficial]

Can confirm the issue with the gemma-3n. It is super weird though... For me there is no SIGSEGV on E4B. It just does not print anything, but the generation seems to be working...

orangepi@orangepi5max:~/Projects/rk-llama.cpp$ taskset -c 4-7 ./build/bin/llama-cli -m ~/Projects/gemma-3n-E4B-it-Q8_0.gguf -tb 4 -t 3 --no-repack -c 1024 -n 512

Loading model...


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8672-d90f5fea7
model      : gemma-3n-E4B-it-Q8_0.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Hello, how are you?



[ Prompt: 6.7 t/s | Generation: 4.0 t/s ]

> What have you done...

[multimodal]

[ Prompt: 16.3 t/s | Generation: 4.2 t/s ]

Yea, it produces garbage.

orangepi@orangepi5max:~/Projects/rk-llama.cpp$ HYBRID_PATTERN="FP16_STANDARD" taskset -c 4-7 ./build/bin/llama-cli -m ~/Projects/gemma-3n-E4B-it-Q8_0.gguf -tb 4 -t 3 --no-repack -c 1024 -n 512

...

build      : b8672-d90f5fea7
model      : gemma-3n-E4B-it-Q8_0.gguf
modalities : text

...

> How old are you?

ovilcional中午‪ sóinspirina só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só

[ Prompt: 5.2 t/s | Generation: 2.5 t/s ]

And for some reason E2B Q8_0 is working fine, but on Q4_0 it prints the first message and then also breaks...

[happyme531]

Can confirm the issue with the gemma-3n. It is super weird though... For me there is no SIGSEGV on E4B. It just does not print anything, but the generation seems to be working...

orangepi@orangepi5max:~/Projects/rk-llama.cpp$ taskset -c 4-7 ./build/bin/llama-cli -m ~/Projects/gemma-3n-E4B-it-Q8_0.gguf -tb 4 -t 3 --no-repack -c 1024 -n 512

Loading model...


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8672-d90f5fea7
model      : gemma-3n-E4B-it-Q8_0.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Hello, how are you?



[ Prompt: 6.7 t/s | Generation: 4.0 t/s ]

> What have you done...

[multimodal]

[ Prompt: 16.3 t/s | Generation: 4.2 t/s ]

Yea, it produces garbage.

orangepi@orangepi5max:~/Projects/rk-llama.cpp$ HYBRID_PATTERN="FP16_STANDARD" taskset -c 4-7 ./build/bin/llama-cli -m ~/Projects/gemma-3n-E4B-it-Q8_0.gguf -tb 4 -t 3 --no-repack -c 1024 -n 512

...

build      : b8672-d90f5fea7
model      : gemma-3n-E4B-it-Q8_0.gguf
modalities : text

...

> How old are you?

ovilcional中午‪ sóinspirina só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só só

[ Prompt: 5.2 t/s | Generation: 2.5 t/s ]

And for some reason E2B Q8_0 is working fine, but on Q4_0 it prints the first message and then also breaks...

From 88f9cdb1572b4a8dde8b41c2bffd2321a33a470a Mon Sep 17 00:00:00 2001
From: hallo1 <2302004040@qq.com>
Date: Fri, 17 Apr 2026 23:21:05 +0800
Subject: [PATCH] rknpu: reject higher-rank mul_mat tensors

---
 ggml/src/ggml-rknpu2/ggml-rknpu2.cpp | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp b/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp
index 19c3e0797..4810e8e53 100644
--- a/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp
+++ b/ggml/src/ggml-rknpu2/ggml-rknpu2.cpp
@@ -1286,6 +1286,16 @@ static bool ggml_backend_rknpu_device_supports_op(ggml_backend_dev_t dev, const
             const struct ggml_tensor * src0 = op->src[0]; // Weights
             const struct ggml_tensor * src1 = op->src[1]; // Activations
 
+            // The current RKNPU matmul path only handles plain 2D matrices.
+            // Batched / higher-rank ggml_mul_mat tensors require semantics that
+            // depend on ne[2]/ne[3], but the implementation below only uses K/N/M
+            // from ne[0]/ne[1]. Reject them so they fall back to CPU.
+            if (src0->ne[2] != 1 || src0->ne[3] != 1 ||
+                src1->ne[2] != 1 || src1->ne[3] != 1 ||
+                op->ne[2]   != 1 || op->ne[3]   != 1) {
+                return false;
+            }
+
             // Searching for available hardware pipeline for this tensor
             const auto* pipeline = config.resolve_op_support(src0);
             if (!pipeline) {
-- 
2.39.2

This patch fixes this issue.