hip: VEC flash-attn for D=512 (Gemma 4) on ROCm with quantized KV

[cclecle]

Overview

Extends the VEC flash-attention path to D=512 heads on HIP/ROCm, enabling Gemma 4 27B decode with quantized KV caches at long context.

Problem: the TILE path allocates an f16 temp buffer per FA call (~2 GB at 256K context with D=512) that the legacy pool retains permanently. With quantized KV this consumes more VRAM than the compression saves, causing OOM. The VEC path does inline dequant with no temp buffer, so it avoids the issue entirely.

D=512 was previously excluded from VEC because nthreads_KQ=2 exceeds the 256-VGPR limit on RDNA4 (wave32). Set nthreads_KQ=4 for D>=512 to halve Q register use. Decode only (ncols=1); prefill falls back to TILE.

Co-Authored-By: 15073640+cclecle@users.noreply.github.com

Additional information

Changes:

• fattn-vec.cuh: add ggml_cuda_flash_attn_ext_vec_case_d512<K,V>
  template (ncols=1 fixed), expose DECL_FATTN_VEC_CASE_D512 macro
• fattn.cu: add FATTN_VEC_CASE_D512 dispatch for K=q8_0 + common V types; extend can_use_vector_kernel to allow D=512 for q8_0 K decode
• fattn-vec-instance-q8_0-{f16,q8_0,bf16,turbo2/3/4_0}.cu: add DECL_FATTN_VEC_CASE_D512 instantiations in existing per-V-type TUs

Requirements

Tests:

• non regression on 8 models: Qwen / Gemma4 /Mistral and derivative variant
• Long context, up to 256k
• turbo3/turbo4 in different combinations
• 2 parallel requests

Environment:

• Ubuntu 26.04 / kernel 7.0.0-15
• 2x Radeon PRO AI R9700 / ROCm 7.2.1
• AMD R9 9950x / 128GB ram

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Claude used to generate all the code, but I tested it myself on 8 different models