Skip to content

perf(qwen3.6): enable fast Gated-DeltaNet AR state kernel on the default decode path#238

Closed
fansilas wants to merge 1 commit into
gittensor-ai-lab:mainfrom
fansilas:perf-qwen36-gdn-fast-default
Closed

perf(qwen3.6): enable fast Gated-DeltaNet AR state kernel on the default decode path#238
fansilas wants to merge 1 commit into
gittensor-ai-lab:mainfrom
fansilas:perf-qwen36-gdn-fast-default

Conversation

@fansilas

@fansilas fansilas commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

The optimized Gated-DeltaNet AR state-update kernel (gdn_ar_fast_kernel in kernels/csrc/cuda/fused/qwen36.cu) is compiled in but gated behind SPARKINFER_GDN_FAST, which defaults OFF and is not set anywhere in the bench or eval path. So Qwen3.6 decode still runs the naive gdn_ar_kernel<<<v_heads=32, head_dim=128>>> on every one of the 30 linear layers, every token: 32 blocks on 170 SMs (about 81% of the GPU idle), two serial 128-iteration dependent-load loops over the recurrent state, and a write-then-reread of the fp32 state (roughly 4x its traffic).

This PR flips that one default so the fast kernel runs on the default decode path (warp-per-state-column, register-cached state slice for one read plus one write, transposed-for-coalescing state layout, filling the grid). SPARKINFER_GDN_FAST=0 still selects the naive kernel as a fallback. The transposed internal state is safe as a default: the recurrent state is zero-initialized and touched only by this kernel, the dispatch flag is static (all-or-nothing for the run, so prefill and decode agree), and the layout never escapes the kernel.

Proof of speedup

  • Tested on RTX 5090 (sm_120)

Decode tok/s (Qwen3.6-35B-A3B UD-Q4_K_M, single stream, bs=1, interleaved A/B on one build via the env toggle):

decode tok/s
before (main) 168.72
after (this PR) 219.97

Per scored context (before to after, median of interleaved pairs):

context before (naive) after (fast) gain
128 168.72 219.97 +30.4%
512 168.80 219.48 +30.0%
4k 162.65 209.44 +28.8%
# main (SPARKINFER_GDN_FAST unset -> naive kernel), ctx=128:
decode tg    : 168.72 tok/s  (5.9 ms/token, n=128, ctx=128, bs=1)
# this PR (fast kernel is the default), ctx=128:
decode tg    : 219.97 tok/s  (4.5 ms/token, n=128, ctx=128, bs=1)

Correctness

The fast kernel is the one added in the merged GDN AR work; it is not bit-identical to the naive kernel (the warp-tree reduction reorders the fp32 sum), so it is validated by output agreement rather than a byte compare. Teacher-forced on the eval corpus, fast vs naive argmax agree 97/100 and perplexity is identical (2.7512 vs 2.7530), i.e. within fp reassociation noise, so token-match and KL versus llama.cpp are unchanged from the naive path (which the correctness gate already passes). ctest is 7/7.

No-regression (Qwen3-30B guard)

The change lives entirely inside launch_qwen36_gdn_ar, which is only reached under w.linear_attn (the Qwen3.6 Gated-DeltaNet layers). Qwen3-30B-A3B has no linear-attention layers, so its decode path is byte-identical and unaffected.

@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

🐈 Flagged: copycat (retroactive sweep)

This PR is >=80% contained in the earlier #236 by jaso0n0818 — a verbatim copy by a different author. Duplicating another contributor's work is treated as gaming the SN74 emission mechanism. The scheduled bot will not evaluate this PR.

See .github/COPYCATS.md.

ai-hpc added a commit that referenced this pull request Jul 6, 2026
….json

- New eval/copycat_sweep.py: retroactive 4-layer detection for all open PRs
- copycats.json updated with sweep findings: #242 (nickmopen), #244 (cleanjunc),
  #238 (fansilas), #240 (fansilas)
- Labels + comments applied to all 4 PRs
@ai-hpc ai-hpc added copycat Re-submits an earlier PR's diff — not evaluated; 2 strikes = block flagged:gaming Eval-gaming / sybil — blocked, not evaluated or merged labels Jul 6, 2026
@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

Closed — copycat detection. See the comment above for details. The original author's work takes priority.

@ai-hpc ai-hpc closed this Jul 6, 2026
@fansilas

fansilas commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

@ai-hpc
Please review my PR manually. I'm really embarrassed.

@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

✅ Manual review — not a copycat

This was reviewed by a maintainer. The overlap with #236 is a coincidence, not a copycat — both PRs independently enabled the GDN fast kernel on the default decode path (a natural, small change). fansilas is a known contributor with multiple merged PRs and a clean history. No action taken — the label has been removed, the strike cleared from the log, and the PR reopened. Apologies for the false flag.

@ai-hpc ai-hpc reopened this Jul 6, 2026
@ai-hpc ai-hpc added copycat Re-submits an earlier PR's diff — not evaluated; 2 strikes = block and removed copycat Re-submits an earlier PR's diff — not evaluated; 2 strikes = block flagged:gaming Eval-gaming / sybil — blocked, not evaluated or merged labels Jul 6, 2026
@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

🐈 Flagged: copycat

This PR re-submits substantially the same diff as the earlier #236. Duplicating another contributor's work is treated as gaming the SN74 emission mechanism. The account has been blocked and this PR closed — zero tolerance, no warning. See .github/COPYCATS.md.

@ai-hpc ai-hpc added the flagged:gaming Eval-gaming / sybil — blocked, not evaluated or merged label Jul 6, 2026
@ai-hpc

ai-hpc commented Jul 6, 2026

Copy link
Copy Markdown
Member

🚩 Flagged: eval-gaming

This PR involves an account blocked for gaming the SN74 emission mechanism (sybil / coordinated duplicate farming): fansilas.

Per the project's no-gaming policy these accounts are blocked: the PR is not evaluated, scored, or merged. See .github/FLAGGED.md for the evidence and record.

@ai-hpc ai-hpc closed this Jul 6, 2026
@ai-hpc ai-hpc removed copycat Re-submits an earlier PR's diff — not evaluated; 2 strikes = block flagged:gaming Eval-gaming / sybil — blocked, not evaluated or merged labels Jul 6, 2026
inference2026 pushed a commit to inference2026/sparkinfer that referenced this pull request Jul 6, 2026
fansilas: maintainer reviewed gittensor-ai-lab#238 — legitimate coincidence, not a copycat.
  Both PRs previously merged (5 total), clean history. All labels removed,
  PRs reopened, copycat log purged.
nickmopen: appeal accepted — Q8_0 MMVQ is a standard dp4a pattern matching
  the repo's own Q4_K/Q6_K kernels. Warning lifted, strike cleared.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants