Add measured LLM decode kernel suite

.github/workflows/ci.yml

@@ -12,8 +12,8 @@ jobs:
   cpu-contracts:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
       - run: python -m pip install --upgrade pip
@@ -22,4 +22,3 @@ jobs:
       - run: pytest
       - run: python -m compileall -q src tests
       - run: triton-kernel-lab --help
-

README.md

@@ -2,15 +2,18 @@
 
 Correctness-first GPU kernel work for inference performance engineering.
 
-The repository contains fused RMSNorm and SwiGLU kernels written in Triton,
+The repository contains RMSNorm, SwiGLU, decode QK dot-product, paged KV gather,
+fused residual-plus-RMSNorm, and signed INT4 GEMV kernels written in Triton,
 high-precision PyTorch oracles, PyTorch eager and `torch.compile` baselines, raw
-latency samples, effective-bandwidth modeling, machine-readable reports, and a
-baseline regression gate.
+latency samples, roofline modeling, machine-readable reports, and a baseline
+regression gate.
 
 ## What It Demonstrates
 
 - Triton GPU kernel development with FP32 reduction, fused normalization, and
   fused gated activation.
+- Decode-oriented kernels for attention scoring, non-contiguous KV movement,
+  residual normalization, and packed signed INT4 weight-only projection.
 - Shape-aware SwiGLU launch autotuning across block sizes and warp counts.
 - Correctness validation across shapes and low-precision dtypes before timing,
   including validation of the `torch.compile` baseline against the FP32 oracle.
@@ -67,9 +70,38 @@ samples are in
 [artifacts/rtx-5070-ti-rmsnorm-swiglu.json](artifacts/rtx-5070-ti-rmsnorm-swiglu.json).
 These results are specific to this hardware and software stack.
 
-The reported bandwidth remains a logical traffic model. Nsight Compute was not
-available in this environment, so the repository does not claim measured DRAM
-transactions, cache hit rates, occupancy, or tensor-core utilization.
+## Decode Kernel Results
+
+Serial cache-cold measurements on the same RTX 5070 Ti stack:
+
+| Kernel | Shape | Dtype | Triton p50 | `torch.compile` p50 | Speedup | Logical BW |
+|---|---:|---:|---:|---:|---:|---:|
+| QK dot | 512 x 128 | FP16 | 0.0055 ms | 0.0127 ms | 2.32x | 48.4 GB/s |
+| QK dot | 2,048 x 128 | FP16 | 0.0078 ms | 0.0141 ms | 1.81x | 135.9 GB/s |
+| Paged KV gather | 512 x 128 | FP16 | 0.0057 ms | 0.0163 ms | 2.85x | 46.4 GB/s |
+| Selected attention | 64 x 128 | BF16 | 0.0064 ms | 0.0185 ms | 2.90x | 5.3 GB/s |
+| Selected attention | 256 x 128 | BF16 | 0.0115 ms | 0.0185 ms | 1.60x | 11.5 GB/s |
+| Residual RMSNorm | 4 x 1,536 | BF16 | 0.0051 ms | 0.0121 ms | 2.37x | 10.3 GB/s |
+| INT4 GEMV | 1,536 x 1,536 | BF16 | 0.0077 ms | 0.0208 ms | 2.70x | 154.6 GB/s |
+| INT4 GEMV | 8,960 x 1,536 | BF16 | 0.0205 ms | 0.0449 ms | 2.19x | 339.4 GB/s |
+
+The checked roofline report uses the RTX 5070 Ti specification of 896 GB/s and
+labels all ten decode cases memory-bound or data-movement-only. Its best
+logical bandwidth fraction is 37.9% for the 8,960 x 1,536 INT4 GEMV. This is a
+specification-based projection, not a physical-traffic measurement.
+
+Nsight Compute 2026.2 is installed and the repository includes a filtered
+capture command. The current host returns `ERR_NVGPUCTRPERM` because Windows
+performance-counter access is disabled. No measured DRAM transactions, cache
+hit rates, occupancy, or execution-pipeline utilization are claimed.
+
+```bash
+triton-kernel-nsight \
+  --kernel int4-gemv \
+  --shape 8960x1536 \
+  --dtype bfloat16 \
+  --output artifacts/nsight/int4-gemv
+```
 
 ## Regression Gate
 
@@ -112,6 +144,5 @@ python -m compileall -q src tests
 ## Scope
 
 This is a focused kernel and measurement lab, not a claim of production-scale
-GPU infrastructure. Planned extensions are hardware-counter capture with
-Nsight Compute, roofline analysis, and additional low-precision inference
-kernels.
+GPU infrastructure. The remaining hardware-counter work depends on manually
+enabling NVIDIA counter access on the Windows host.

artifacts/rtx-5070-ti-decode-roofline.json

@@ -0,0 +1,201 @@
+{
+  "schema_version": 1,
+  "source_artifacts": [
+    "artifacts\\rtx-5070-ti-qk-dot.json",
+    "artifacts\\rtx-5070-ti-paged-gather.json",
+    "artifacts\\rtx-5070-ti-selective-attention.json",
+    "artifacts\\rtx-5070-ti-residual-rmsnorm.json",
+    "artifacts\\rtx-5070-ti-int4-gemv.json"
+  ],
+  "assumptions": {
+    "peak_memory_bandwidth_gbps": 896.0,
+    "peak_compute_tflops": 43.9,
+    "traffic": "logical payload bytes, not measured DRAM transactions",
+    "compute": "logical FLOPs, excluding address and conversion instructions"
+  },
+  "cases": [
+    {
+      "kernel": "qk-dot",
+      "shape": [
+        512,
+        128
+      ],
+      "dtype": "float16",
+      "p50_ms": 0.005456000100821257,
+      "logical_bytes": 264192,
+      "logical_flops": 131072,
+      "achieved_logical_bandwidth_gbps": 48.422286495235376,
+      "bandwidth_fraction_of_spec": 0.05404273046343234,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.49612403100775193,
+      "achieved_gflops": 24.0234599666284,
+      "memory_ceiling_gflops": 444.52713178294573,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.05404273046343233
+    },
+    {
+      "kernel": "qk-dot",
+      "shape": [
+        2048,
+        128
+      ],
+      "dtype": "float16",
+      "p50_ms": 0.007775999838486314,
+      "logical_bytes": 1056768,
+      "logical_flops": 524288,
+      "achieved_logical_bandwidth_gbps": 135.9012373906777,
+      "bandwidth_fraction_of_spec": 0.1516754881592385,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.49612403100775193,
+      "achieved_gflops": 67.42386971320444,
+      "memory_ceiling_gflops": 444.52713178294573,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.1516754881592385
+    },
+    {
+      "kernel": "paged-gather",
+      "shape": [
+        128,
+        128
+      ],
+      "dtype": "float16",
+      "p50_ms": 0.004991999827325344,
+      "logical_bytes": 66048,
+      "logical_flops": 0,
+      "achieved_logical_bandwidth_gbps": 13.230769688425202,
+      "bandwidth_fraction_of_spec": 0.01476648402726027,
+      "classification": "data-movement-only",
+      "arithmetic_intensity_flops_per_byte": 0.0
+    },
+    {
+      "kernel": "paged-gather",
+      "shape": [
+        512,
+        128
+      ],
+      "dtype": "float16",
+      "p50_ms": 0.005696000065654516,
+      "logical_bytes": 264192,
+      "logical_flops": 0,
+      "achieved_logical_bandwidth_gbps": 46.38202193729122,
+      "bandwidth_fraction_of_spec": 0.051765649483583955,
+      "classification": "data-movement-only",
+      "arithmetic_intensity_flops_per_byte": 0.0
+    },
+    {
+      "kernel": "selective-attention",
+      "shape": [
+        64,
+        128
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.0063680000603199005,
+      "logical_bytes": 33536,
+      "logical_flops": 33088,
+      "achieved_logical_bandwidth_gbps": 5.266331608406942,
+      "bandwidth_fraction_of_spec": 0.005877602241525605,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.9866412213740458,
+      "achieved_gflops": 5.195979850279367,
+      "memory_ceiling_gflops": 884.030534351145,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.005877602241525604
+    },
+    {
+      "kernel": "selective-attention",
+      "shape": [
+        256,
+        128
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.01152000017464161,
+      "logical_bytes": 132608,
+      "logical_flops": 132352,
+      "achieved_logical_bandwidth_gbps": 11.51111093660426,
+      "bandwidth_fraction_of_spec": 0.01284722202746011,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.9980694980694981,
+      "achieved_gflops": 11.488888714718923,
+      "memory_ceiling_gflops": 894.2702702702703,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.01284722202746011
+    },
+    {
+      "kernel": "residual-rmsnorm",
+      "shape": [
+        1,
+        1536
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.00531200016848743,
+      "logical_bytes": 15360,
+      "logical_flops": 9216,
+      "achieved_logical_bandwidth_gbps": 2.8915661733447755,
+      "bandwidth_fraction_of_spec": 0.003227194389893723,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.6,
+      "achieved_gflops": 1.7349397040068655,
+      "memory_ceiling_gflops": 537.6,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.003227194389893723
+    },
+    {
+      "kernel": "residual-rmsnorm",
+      "shape": [
+        4,
+        1536
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.00508800009265542,
+      "logical_bytes": 52224,
+      "logical_flops": 36864,
+      "achieved_logical_bandwidth_gbps": 10.264150756480108,
+      "bandwidth_fraction_of_spec": 0.011455525397857264,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 0.7058823529411765,
+      "achieved_gflops": 7.245282886927136,
+      "memory_ceiling_gflops": 632.4705882352941,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.011455525397857264
+    },
+    {
+      "kernel": "int4-gemv",
+      "shape": [
+        1536,
+        1536
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.007727999938651919,
+      "logical_bytes": 1195008,
+      "logical_flops": 4718592,
+      "achieved_logical_bandwidth_gbps": 154.63354160021626,
+      "bandwidth_fraction_of_spec": 0.17258207767881278,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 3.948586118251928,
+      "achieved_gflops": 610.5838557787458,
+      "memory_ceiling_gflops": 3537.9331619537274,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.17258207767881276
+    },
+    {
+      "kernel": "int4-gemv",
+      "shape": [
+        8960,
+        1536
+      ],
+      "dtype": "bfloat16",
+      "p50_ms": 0.020495999604463577,
+      "logical_bytes": 6956032,
+      "logical_flops": 27525120,
+      "achieved_logical_bandwidth_gbps": 339.38486213110235,
+      "bandwidth_fraction_of_spec": 0.37877774791417673,
+      "classification": "memory-bound",
+      "arithmetic_intensity_flops_per_byte": 3.9570145738259974,
+      "achieved_gflops": 1342.950845588699,
+      "memory_ceiling_gflops": 3545.4850581480937,
+      "compute_ceiling_gflops": 43900.0,
+      "roofline_efficiency": 0.37877774791417673
+    }
+  ],
+  "claim_boundary": "This is a specification-based roofline projection. Nsight Compute counters are required for physical memory-traffic and utilization claims."
+}