perf(qwen3.6): default the merged gdn_ar_fast_kernel ON#236
Open
jaso0n0818 wants to merge 1 commit into
Open
Conversation
SPARKINFER_GDN_FAST (merged in gittensor-ai-lab#229) only activates gdn_ar_fast_kernel when the env var is explicitly set to "1". The eval harness (bench/scripts/evaluate.sh, evaluate_dual.sh) never sets feature env vars -- it only passes guard/baseline/difficulty scoring parameters -- so the merged kernel has never actually run in scoring. The naive gdn_ar_kernel (<<<v_heads=32, head_dim>>>, ~81% of the SMs idle, two serial head_dim-long dependent loops) has stayed the decode path's largest single non-GEMV cost the whole time. This flips the default to ON (opt-out via SPARKINFER_GDN_FAST=0), matching every other feature flag in this file/runtime (use_pq, use_llama, use_q6mmvq, use_fnq are all default-on / opt-out-via-0). No kernel code changes -- gdn_ar_fast_kernel is gittensor-ai-lab#229's kernel, unmodified.
3 tasks
This was referenced Jul 6, 2026
Draft
Member
⏳ Needs a benchmark to be evaluatedYou ticked Tested on RTX 5090 but the decode before → after tok/s table is still empty / placeholder (or shows no gain). The on-device eval won't run until it shows a real improvement. Fill it from the end-to-end decode bench (not an isolated-kernel microbench): bench/scripts/bench.sh --download # baseline (before)
bench/scripts/bench.sh --download # your branch (after)Then the bot greenlights it on the next poll and evaluates it on an RTX 5090. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SPARKINFER_GDN_FAST(merged in #229) only activatesgdn_ar_fast_kernelwhen theenv var is explicitly set to
1. The eval harness (bench/scripts/evaluate.sh,evaluate_dual.sh) never sets feature env vars -- it only passesguard/baseline/difficulty scoring parameters (
SPARKINFER_GUARD_*,SPARKINFER_DIFFICULTY_*,SPARKINFER_EVAL_MODE, etc.) -- so the merged kernel hasnever actually run during scoring. The naive
gdn_ar_kernel(
<<<v_heads=32, head_dim>>>, ~81% of the SMs idle every token, two serialhead_dim-long dependent-load loops, as described in #229's own summary) has stayed
the decode path's largest single non-GEMV cost the whole time.
This flips the default to ON (opt-out via
SPARKINFER_GDN_FAST=0), matchingevery other feature flag in this runtime (
use_pq,use_llama,use_q6mmvq,use_fnqare all default-on / opt-out-via-0). No kernel code changes --gdn_ar_fast_kernelis #229's kernel, unmodified; this is a one-line default flipplus a comment explaining why.
Scope:
kernels/csrc/cuda/fused/qwen36.cu, single function(
launch_qwen36_gdn_ar), single line changed.Proof of speedup
sm_120)Decode tok/s (
bench/scripts/bench.sh, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, bs=1, same box):Correctness (teacher-forced, same held-out prompt, vs
maindefault):generate(): token-for-token identical across 48 generated tokens.score: PPL 2.76957 -> 2.76956, ARGMATCH 73/101 -> 71/101 (0.7228 -> 0.7030,both far above bar). The tiny ARGMATCH delta is the same kind of fp32
reduction-order noise any split/tiling change in this codebase produces (feat(qwen36): add optimized Gated-DeltaNet AR state update kernel wit… #229's
own PR notes this is byte-exact math, different order); PPL is unchanged to 5
significant figures.
ctest --test-dir build: 7/7 pass.Why this wasn't caught by #229's own eval
#229 was rebased on top of #230 (shared-expert fix, 23->167 tok/s) before its final
eval run, and #230's fix alone is a 7x win -- enough to land
eval:XLeven withgdn_ar_fast_kernelsitting inactive behind its opt-in flag the whole time. Thegap only shows up when you diff default-vs-explicit-flag on the current main,
which is what this PR does.