Skip to content

Add fused SwiGLU kernel benchmark #1

Merged
WaffleBits merged 1 commit into
mainfrom
codex/swiglu-kernel-20260614
Jun 14, 2026
Merged

Add fused SwiGLU kernel benchmark #1
WaffleBits merged 1 commit into
mainfrom
codex/swiglu-kernel-20260614

Conversation

@WaffleBits

Copy link
Copy Markdown
Owner

What changed

  • add a fused Triton SwiGLU kernel with FP32 PyTorch oracle validation
  • autotune block size and warp count per tensor size
  • benchmark RMSNorm and SwiGLU through the same cache-controlled CUDA-event harness
  • commit a 12-case RTX 5070 Ti report with raw samples and regression-compatible case keys
  • document measured results and the boundary between logical bandwidth and hardware counters

Why

Current inference performance roles consistently ask for custom kernels, correctness-first optimization, low-precision numerics, launch tuning, and quantitative evidence. This broadens the lab beyond a single reduction kernel without inventing scale claims.

Validation

  • python -m pytest
  • python -m ruff check .
  • python -m compileall -q src tests
  • full RTX 5070 Ti cache-cold run: 100 warmups, 500 samples, FP16/BF16, three shapes, both kernels
  • all 12 GPU correctness cases passed

@WaffleBits WaffleBits added the enhancement New feature or request label Jun 14, 2026
@WaffleBits
WaffleBits marked this pull request as ready for review June 14, 2026 01:21
@WaffleBits
WaffleBits merged commit 81e05cf into main Jun 14, 2026
1 check passed
@WaffleBits
WaffleBits deleted the codex/swiglu-kernel-20260614 branch June 14, 2026 01:21
@WaffleBits WaffleBits changed the title [codex] Add fused SwiGLU kernel benchmark Add fused SwiGLU kernel benchmark Jul 16, 2026
WaffleBits added a commit that referenced this pull request Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant