Skip to content

perf(qwen3.6): default the merged gdn_ar_fast_kernel ON#236

Open
jaso0n0818 wants to merge 1 commit into
gittensor-ai-lab:mainfrom
jaso0n0818:perf/gdn-fast-default-on
Open

perf(qwen3.6): default the merged gdn_ar_fast_kernel ON#236
jaso0n0818 wants to merge 1 commit into
gittensor-ai-lab:mainfrom
jaso0n0818:perf/gdn-fast-default-on

Conversation

@jaso0n0818

Copy link
Copy Markdown
Contributor

Summary

SPARKINFER_GDN_FAST (merged in #229) only activates gdn_ar_fast_kernel when the
env var is explicitly set to 1. The eval harness (bench/scripts/evaluate.sh,
evaluate_dual.sh) never sets feature env vars
-- it only passes
guard/baseline/difficulty scoring parameters (SPARKINFER_GUARD_*,
SPARKINFER_DIFFICULTY_*, SPARKINFER_EVAL_MODE, etc.) -- so the merged kernel has
never actually run during scoring. The naive gdn_ar_kernel
(<<<v_heads=32, head_dim>>>, ~81% of the SMs idle every token, two serial
head_dim-long dependent-load loops, as described in #229's own summary) has stayed
the decode path's largest single non-GEMV cost the whole time.

This flips the default to ON (opt-out via SPARKINFER_GDN_FAST=0), matching
every other feature flag in this runtime (use_pq, use_llama, use_q6mmvq,
use_fnq are all default-on / opt-out-via-0). No kernel code changes --
gdn_ar_fast_kernel is #229's kernel, unmodified; this is a one-line default flip
plus a comment explaining why.

Scope: kernels/csrc/cuda/fused/qwen36.cu, single function
(launch_qwen36_gdn_ar), single line changed.

Proof of speedup

  • Tested on RTX 5090 (sm_120)

Decode tok/s (bench/scripts/bench.sh, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, bs=1, same box):

context before (main default) after (this PR) delta
128 171.90 224.47 +30.6%
512 170.46 222.07 +30.3%
=== before (main, SPARKINFER_GDN_FAST unset -> naive gdn_ar_kernel) ===
model        : Qwen3.5/Qwen3.6-35B-A3B hybrid  (40 layers, 256 experts top-8)
VRAM used    : 25.3 GB
decode tg    : 171.90 tok/s  (5.8 ms/token, n=128, ctx=0, bs=1)
decode tg    : 170.46 tok/s  (5.9 ms/token, n=512, ctx=0, bs=1)

=== after (this PR, default now runs gdn_ar_fast_kernel) ===
model        : Qwen3.5/Qwen3.6-35B-A3B hybrid  (40 layers, 256 experts top-8)
VRAM used    : 25.3 GB
decode tg    : 224.47 tok/s  (4.5 ms/token, n=128, ctx=0, bs=1)
decode tg    : 222.07 tok/s  (4.5 ms/token, n=512, ctx=0, bs=1)

Correctness (teacher-forced, same held-out prompt, vs main default):

  • generate(): token-for-token identical across 48 generated tokens.
  • score: PPL 2.76957 -> 2.76956, ARGMATCH 73/101 -> 71/101 (0.7228 -> 0.7030,
    both far above bar). The tiny ARGMATCH delta is the same kind of fp32
    reduction-order noise any split/tiling change in this codebase produces (feat(qwen36): add optimized Gated-DeltaNet AR state update kernel wit… #229's
    own PR notes this is byte-exact math, different order); PPL is unchanged to 5
    significant figures.
  • ctest --test-dir build: 7/7 pass.

Why this wasn't caught by #229's own eval

#229 was rebased on top of #230 (shared-expert fix, 23->167 tok/s) before its final
eval run, and #230's fix alone is a 7x win -- enough to land eval:XL even with
gdn_ar_fast_kernel sitting inactive behind its opt-in flag the whole time. The
gap only shows up when you diff default-vs-explicit-flag on the current main,
which is what this PR does.

SPARKINFER_GDN_FAST (merged in gittensor-ai-lab#229) only activates gdn_ar_fast_kernel
when the env var is explicitly set to "1". The eval harness
(bench/scripts/evaluate.sh, evaluate_dual.sh) never sets feature env
vars -- it only passes guard/baseline/difficulty scoring parameters --
so the merged kernel has never actually run in scoring. The naive
gdn_ar_kernel (<<<v_heads=32, head_dim>>>, ~81% of the SMs idle, two
serial head_dim-long dependent loops) has stayed the decode path's
largest single non-GEMV cost the whole time.

This flips the default to ON (opt-out via SPARKINFER_GDN_FAST=0),
matching every other feature flag in this file/runtime (use_pq,
use_llama, use_q6mmvq, use_fnq are all default-on / opt-out-via-0).
No kernel code changes -- gdn_ar_fast_kernel is gittensor-ai-lab#229's kernel,
unmodified.
@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

⏳ Needs a benchmark to be evaluated

You ticked Tested on RTX 5090 but the decode before → after tok/s table is still empty / placeholder (or shows no gain). The on-device eval won't run until it shows a real improvement.

Fill it from the end-to-end decode bench (not an isolated-kernel microbench):

bench/scripts/bench.sh --download            # baseline (before)
bench/scripts/bench.sh --download            # your branch (after)

Then the bot greenlights it on the next poll and evaluates it on an RTX 5090.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels subsystem (emission weight 0.42) needs-benchmark Box ticked but decode before/after not filled with a real improvement — not evaluated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants