Dflash issue on V100 build

[merbanan]

uv run --directory dflash python scripts/run.py --prompt "def fibonacci(n):"
[run] prompt 14 tokens, streaming up to 256 tokens, max_ctx=512
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=22 temp=1.00 chain_seed=1 fa_window=2048 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft]  loaded
[prompt] 14 tokens
[prefill] token-seg ubatch=256
[prefill] 14 tokens in 0.10 s, last_tok=0
[migrate] 242.89 ms
[dbg sib step 0] N=23 accept=1 walked_sib=0
  walk: 0
[step 0] committed=14 last_tok=0 tree_N=23 accept=1 next=-1

[timing] per-step averages over 0 steps (ms):
  draft_build    0.27
  draft_copyfeat 0.07
  draft_set      0.02
  draft_compute  10.16
  draft_bridge   0.01
  draft_logits   9.47
  snapshot_ssm   0.00
  verify_build   1.68
  verify_set     0.25
  verify_compute 93.74
  verify_logits  0.00
  accept         0.04
  restore_ssm    0.00
  replay_build   0.00
  replay_set     0.00
  replay_compute 0.00
  replay_logits  0.00
  mirror_sync    0.00
  ----- sum     115.71

[dflash] generated 0 tokens in 0.116 s  ->  0.00 tok/s
[dflash] 0 draft steps, accepted=0/0 (0.0% per step), avg commit/step=0.00
[dflash] output tail: 248045 846 198 727 73111 1393 1590 248046 198 248045 74455 198 248068 198 0 
!ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32501 MiB):
  Device 0: Tesla V100-PCIE-32GB, compute capability 7.0, VMM: yes, VRAM: 32501 MiB

[run] generated 1 tokens


uv run --directory dflash python scripts/bench_llm.py
[bench] target    = /media/per/work/tmp/lucebox-hub/dflash/models/Qwen3.6-27B-Q4_K_M.gguf
[bench] draft     = /media/per/work/tmp/lucebox-hub/dflash/models/draft/dflash-draft-3.6-q8_0.gguf
[bench] ar bin    = /media/per/work/tmp/lucebox-hub/dflash/build/test_generate
[bench] df bin    = /media/per/work/tmp/lucebox-hub/dflash/build/test_dflash
[bench] tokenizer = Qwen/Qwen3.5-27B
[bench] budget    = 22

[bench] ==== HumanEval (n=10, n_gen=256) ====
  [01/10] n_tok=  92  AR= 30.00  DFlash=   0.00  AL= 0.00

[merbanan]

Using 0c103a3 commit.

[merbanan]

0001-fix-dflash-pass-kv_pad_override-to-all-build_causal_.patch
0002-fix-dflash-update-generation-stats-before-early-exit.patch
0003-feat-dflash-add-startup-config-and-prefill-logit-ran.patch
0004-fix-llama.cpp-bump-submodule-with-Volta-MMA-dispatch.patch

[merbanan]

Claude said these patches fixed the benchmark test and the Volta support. 1/10 math benchmark passed so something got better then it ever was before.

Tagging @pebaryan as this might be of interest. I'm not 100% sure what all this does. I do get a 2.5x-3x speedup.

[davide221]

@merbanan thanks for working on a solution, can you make a PR with your fixes for the volta architecture?

[merbanan]

@davide221 I need to validate everything first and see if I can understand what everything does. I dont want to open a PR if I cant address possible issues. I understand far to little how everything is put together in this project.

[merbanan]

For the record, it seems that the budget value should be set to something lower then 16. 12 seems like a good value.

[merbanan]

With these patches and some more chat starts to working. Opencode still wont work reliably.

[davide221]

https://github.com/Luce-Org/lucebox-hub/blob/main/harness/clients/run_opencode.sh check this out

[pebaryan]

I am checking the patches now. will update later

[pebaryan]

I've been working through the same V100 issues. Here's what I found:
From @merbanan's patches on #259 (adapted for the current server/ restructure):

1. kv_pad_override : The attn_mask CPU buffer stride didn't match the GPU tensor stride. On V100 (cudaMalloc doesn't
   zero-initialize), rows q>=1 wrote at wrong offsets → flash-attention read garbage → NaN logits → every draft token rejected → 0 tok/s.
   Fixed by passing attn_mask->ne[0] as kv_pad_override at all build_causal_mask call sites. This is the core V100 fix.

2. Stats before early-exit : Three break paths (DDTree, fast-rollback, replay) exit the decode loop before updating
   n_generated/n_accept_sum/n_draft_steps, so hitting EOS on the first step reports 0 tok/s.

3. Startup diagnostics : Prints target/draft layer configs with WARNING on mismatch, plus prefill logit range to catch NaN/zero
   corruption early.

Additional issues I hit:

4. Single-GPU crash : When both target and draft share the same V100, passing the target's lm_head tensor directly into the draft
   graph's ggml allocator causes a silent crash inside ggml's gallocr. This is what prevented me from even running a single V100 at all.
   Fix: always use hidden bridge mode (draft_hidden_bridge = true), which does the lm_head projection in a separate step. (This codepath didn't exist in the old dflash/ structure that @merbanan's patches targeted.)

5. Volta MMA dispatch : On sm_70, ggml_cuda_fattn_mma_get_config_volta only has entries for DKQ 512 and 576. Qwen models use
   head_dim=128, which falls through to the Ampere config → NaN outputs. Fix: guard MMA_F16 kernel selection on Volta to only head_dim 512/576.

With all five fixes, Math500 runs all 10/10 samples cleanly on a single V100 at ~9.4 tok/s (AR baseline ~32 tok/s).

Branch with all fixes at pebaryan/lucebox-hub:v100-dflash-fixes if anyone wants to test.

[merbanan]

4. Fix: always use hidden bridge mode (draft_hidden_bridge = true), which does the lm_head projection in a separate step. (This codepath didn't exist in the old dflash/ structure that @merbanan's patches targeted.)

@pebaryan Things move so quickly it is hard to keep track :). I'll try your branch eventually.

[merbanan]

I tried building but I just got errors.