Eval bug: Segfault at the end of the cache (cache defragmentation?)

[cpg314]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
version: 4850 (ea002810)
built with cc (GCC) 14.2.1 20250128 for x86_64-pc-linux-gnu

Operating systems

Linux

GGML backends

CUDA, CPU

Hardware

NVIDIA RTX 3060 12 GB VRAM and an AMD Ryzen 9 7900

Models

Mistral Small 24B 2501 with Q4_K_M quantization and q4_0 KV cache quantization

Problem description & steps to reproduce

• A long message is passed to llama-server (started with -c 2048), producing a long response.
• The cache reaches the size of the context.
• There is a context shift
• There is a defrag
• The server segfaults
  (see logs below)

The segfault does NOT happen:

• with -c 1024 or with -c 2056
• when I disable defragmentation with -dt 0

First Bad Commit

No response

Relevant log output

slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 2047, n_cache_tokens = 2047, truncated = 1
srv  update_slots: decoding batch, n_tokens = 1
slot process_toke: id  0 | task 0 | n_decoded = 363, n_remaining = -1, next token:  4440 ' document'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 366
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 367, front = 0
slot update_slots: id  0 | task 0 | slot context shift, n_keep = 1, n_left = 2046, n_discard = 1023
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 1025, n_cache_tokens = 1025, truncated = 1
srv  update_slots: decoding batch, n_tokens = 1
llama_decode_impl: fragmentation: 0.37 - requesting defrag
slot process_toke: id  0 | task 0 | n_decoded = 364, n_remaining = -1, next token:  1395 ' is'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 367
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 368, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 1026, n_cache_tokens = 1026, truncated = 1
srv  update_slots: decoding batch, n_tokens = 1
fish: Job 1, './build/bin/llama-server -m /ho…' terminated by signal SIGSEGV (Address boundary error)

[cpg314]

Backtrace for this one:

__memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:461
461             VMOVU   -VEC_SIZE(%rsi, %rdx), %VMM(5)
(gdb) bt
#0  __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:461
#1  0x00007ffff78f5587 in ggml_compute_forward_dup_same_cont (params=0x7fff307d47b0, dst=0x555559d6a9e0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2980
#2  0x00007ffff78f9f51 in ggml_compute_forward_dup_bytes (params=0x7fff307d47b0, dst=0x555559d6a9e0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:3927
#3  0x00007ffff78fae97 in ggml_compute_forward_dup (params=0x7fff307d47b0, dst=0x555559d6a9e0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:4120
#4  0x00007ffff790db30 in ggml_compute_forward_cpy (params=0x7fff307d47b0, dst=0x555559d6a9e0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:8494
#5  0x00007ffff7920dd8 in ggml_compute_forward (params=0x7fff307d47b0, tensor=0x555559d6a9e0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:13038
#6  0x00007ffff7922a76 in ggml_graph_compute_thread (data=0x55555c2750b0)
    at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14046
#7  0x00007ffff792339f in ggml_graph_compute._omp_fn.0(void) () at /home/tmp/llamacpp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14321
#8  0x00007ffff7f05637 in gomp_thread_start (xdata=<optimized out>) at /usr/src/debug/gcc/gcc/libgomp/team.c:129
#9  0x00007ffff0ca370a in start_thread (arg=<optimized out>) at pthread_create.c:448
#10 0x00007ffff0d27aac in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

So it looks like the same as #12354

@ggerganov This is also fixed by #12310