Notes/qwen3 30b a3b fp8 turbo campaign

agent/workspace/qwen3_fp8_tw_gfx950_20261010/README.md

@@ -0,0 +1,147 @@
+# Qwen3-30B-A3B FP8 tensorwise on 8x MI355X: Turbo kernel campaign (handoff)
+
+Task: [prompt.md](prompt.md). Log of every round: [logs/optimize.md](logs/optimize.md). Every end-to-end run: [e2e/results.md](e2e/results.md).
+
+Status (2026-10-10): the 40,000 tokens/s/GPU target is reached. The baseline was 29,474 tokens/s/GPU and the best stack runs at 40,283 (mean of 4 runs). One open correctness issue: an intermittent NaN forward loss (see open items).
+
+## End-to-end
+
+Qwen3-30B-A3B FP8 tensorwise pretrain on 8x MI355X: EP8, MBS 8, GBS 512, seq 4096, even routing, 20 iterations, mean of iterations 11-20. tokens/s/GPU = 262144 / (s/iter). Each row adds one change on top of the row above.
+
+| run | ms/iter | TFLOP/s/GPU | tokens/s/GPU | mem | loss@20 |
+|---|---|---|---|---|---|
+| baseline: submitted config, Turbo e2d9f1d7 | 8894.0 | 670.6 | 29,474 | 277 GB | 11.33393 |
+| + `use_turbo_fused_act_with_probs` | 7915.6 | 753.5 | 33,117 | 223 GB | 11.33435 |
+| Turbo 1103b2df (base of the Turbo branches) | 7925.4 | 752.5 | 33,077 | 222 GB | 11.33427 |
+| + flat tensorwise FP8 quant kernel | 7870.9 | 757.7 | 33,305 | 222 GB | 11.33415 |
+| + per-token `get_dispatch_layout` | 7774.2 | 767.2 | 33,720 | 222 GB | 11.33431 |
+| + HIP permute in `DeepEPTokenDispatcher` | 7643.5 | 780.3 | 34,296 | 222 GB | 11.33423 |
+| + `turbo_deepep_num_cu 160` | 7353.4 | 811.1 | 35,649 | 223 GB | 11.33456 |
+| + fused qk RMSNorm + RoPE, head_dim 128 | 6987.1 | 853.6 | 37,518 | 224 GB | 11.33373 |
+| + `turbo_fused_grouped_gemm` (2 runs: 6820.2, 6815.1) | 6817.7 | 874.9 | 38,451 | 224.5 GB | 11.33412 |
+| + `turbo_fp8_permute` (4 runs: 6442.4, 6515.0, 6523.8, 6549.1) | 6507.6 | 916.7 | 40,283 | 223.3 GB | 11.33404 |
+
+Run-to-run loss noise is about 4e-4 at iteration 20.
+
+## Settings for the last row
+
+On the Primus branch `perf/qwen3-30b-a3b-fp8-turbo`, `examples/megatron/configs/MI355X/qwen3_30B_A3B-FP8-pretrain.yaml` turns all of these on:
+- `use_turbo_fused_act_with_probs: true`
+- `turbo_deepep_num_cu: 160`
+- `turbo_fused_grouped_gemm: true`
+- `turbo_fp8_permute: true`
+- top-level `env: PRIMUS_FUSED_QK_RMSNORM_ROPE: "1"`
+
+On that branch, the config alone gave 6410.6 and 6459.6 ms/iter (40,892 / 40,582 tokens/s/GPU). One more config-only run hit the intermittent NaN (see open items).
+
+Code it needs:
+- Turbo: all five Turbo branches below. `#554` (HIP permute default) is on main; `#555` is rebased onto it.
+- Primus: `perf/qwen3-30b-a3b-fp8-turbo`.
+
+See [e2e/e2e_run_cfg.sh](e2e/e2e_run_cfg.sh) (config only) and [e2e/env_turbo_opt_cfg.sh](e2e/env_turbo_opt_cfg.sh). The older [e2e/e2e_run.sh](e2e/e2e_run.sh) passes the switches on the command line. All paths are for the old machine.
+
+## Branches and PRs
+
+Turbo (`AMD-AGI/Primus-Turbo`):
+
+| branch | head | PR | notes |
+|---|---|---|---|
+| `fix/moe/permute-padding-rows` | f1a59fb9 | #553 merged | TURBO permute padding rows with worst-case buffers + pad |
+| `perf/moe/permute-default-hip` | 30b6cc7c | #554 merged | body: [pr_bodies/perf-moe-permute-default-hip.md](pr_bodies/perf-moe-permute-default-hip.md) |
+| `perf/moe/fp8-permute-tensorwise` | cf630aad | #555 merged | rebased onto main after #554 squash; body: [pr_bodies/perf-moe-fp8-permute-tensorwise.md](pr_bodies/perf-moe-fp8-permute-tensorwise.md) |
+| `perf/quantization/tensorwise-fp8-qwen3` | 6351abf4 | #558 open | independent; body: [pr_bodies/perf-quantization-tensorwise-fp8-qwen3.md](pr_bodies/perf-quantization-tensorwise-fp8-qwen3.md) |
+| `perf/deep_ep/dispatch-layout-per-token` | 0931ba1e | #557 open | independent; body: [pr_bodies/perf-deep-ep-dispatch-layout-per-token.md](pr_bodies/perf-deep-ep-dispatch-layout-per-token.md) |
+| `perf/flydsl/qk-rmsnorm-rope-hd128` | 1b920afe | #556 open (D=64 no reg) | independent; needed by the Primus qk-norm patch; body: [pr_bodies/perf-flydsl-qk-rmsnorm-rope-hd128.md](pr_bodies/perf-flydsl-qk-rmsnorm-rope-hd128.md) |
+| `fix/fp8-attn-strided-layout` | c825b13e | opening | FP8 attention with sbhd/bhsd storage; body: [pr_bodies/fix-fp8-attn-strided-layout.md](pr_bodies/fix-fp8-attn-strided-layout.md) |
+
+`perf/moe/fp8-permute-tensorwise` is now one commit on main (`cf630aad`); rebased after #554's squash merge.
+
+Primus (`AMD-AGI/Primus`):
+
+| branch | head | PR | notes |
+|---|---|---|---|
+| `perf/qwen3-30b-a3b-fp8-turbo` | bef41e8a | opening | the one Primus PR, on main 0a68e5cd; body: [pr_bodies/primus-perf-qwen3-30b-a3b-fp8-turbo.md](pr_bodies/primus-perf-qwen3-30b-a3b-fp8-turbo.md) |
+
+It holds five commits: the FP8 attention fix, the tensorwise config, the head_dim 128 qk-norm patch, `turbo_fp8_permute` (with new unit tests), and the config turning every switch on. It replaces `perf/qwen3-30b-a3b-tuning`, `perf/qwen3-qk-rmsnorm-rope-hd128` and `perf/megatron/turbo-fp8-permute`. Their force-"even" routing fix is already on main (#1243).
+
+## Rebuilding the stack on a new machine
+
+All of these merges are clean as of 2026-10-10.
+
+Container: every build, test and run used the image `tasimage/primus:pr-1236` (container `xiaoming-dev`, started with `--network=host --ipc=host --privileged`, `/dev/kfd`, `/dev/dri` and `/dev/infiniband`, home directory mounted). Run everything with `ulimit -c 0`, because core dumps fill the disk.
+
+Turbo:
+
+```bash
+git fetch origin && git checkout -b integ/qwen3-fp8 origin/main
+for b in perf/quantization/tensorwise-fp8-qwen3 perf/deep_ep/dispatch-layout-per-token \
+         perf/flydsl/qk-rmsnorm-rope-hd128 perf/moe/fp8-permute-tensorwise; do
+  git merge --no-edit origin/$b || break
+done
+GPU_ARCHS=gfx950 MAX_JOBS=64 python setup.py build_ext --inplace
+```
+
+Primus:
+
+```bash
+git fetch origin && git checkout -b integ/qwen3-fp8 origin/perf/qwen3-30b-a3b-fp8-turbo
+git submodule update --init third_party/Megatron-LM
+```
+
+Then edit the paths in `e2e/env_turbo_opt_cfg.sh` and `e2e/e2e_run_cfg.sh` for the new machine, and run, for example:
+
+```bash
+e2e/e2e_run_cfg.sh cfg1
+```
+
+Summarize the runs with `e2e/parse_e2e.py <run names>`, which reads `/tmp/q3_e2e_<name>.log`.
+
+Run Python from the campaign directory with `PYTHONPATH=<turbo checkout>`. Otherwise the pip-installed Turbo is imported.
+
+## Findings and rejected options
+
+- **MegaMoE** (bf16 and mxfp8) runs out of memory at MBS 8. When the worst-case dispatch pool allocation fails, 261 GB (bf16, failing on 3 GiB) or 264 GB (mxfp8, failing on 1.5 GiB) is already allocated. Using it would need a MegaMoE change, a smaller MBS, or recompute.
+- **`turbo_deepep_num_cu`**:
+  - 192 gave a NaN forward loss on rank 2 at iteration 5 in its only run. 160 also hit it once (see open items), so this is not specific to 192.
+  - 256 is slower (7028 ms).
+  - 160 is best.
+- **Sync-free MoE stage 2** costs +300 ms over stage 1. `num_worst_tokens` makes every post-dispatch step scan 262144 receive rows instead of 32768 tokens.
+- **`turbo_grouped_gemm_without_padding`** needs `moe_router_padding_for_quantization False`, which sync-free stage 1 forces to True.
+- **Attention**: TE CK is the fastest. The Turbo AITER forward is 2x slower (causal kernel without tile skipping), so keep `use_turbo_attention` false.
+- **#455** made TRITON the generic `moe_permute` default, and it is slower than HIP at every measured shape. `perf/moe/permute-default-hip` switches the dispatcher back.
+- **MXFP4 tests**: 20 MXFP4 quantization tests also fail on the original build (`x.is_contiguous()` assertion in MXFP4 dequant). Unrelated to this work.
+- **TURBO permute requires routed tokens to come first.** DeepEP guarantees this. The worst-case + pad padding-row bug is fixed in #553. TURBO unpermute still leaves rows past the routed count unwritten; this is harmless because DeepEP combine reads only received rows.
+
+## Open items
+
+- **Intermittent NaN forward loss (mitigated).** Run `cfg1` of the final config died at iteration 3 with `found NaN in local forward loss calculation` on rank 4. Rate: 1/13 at `turbo_deepep_num_cu 160`, 1/1 at 192; none at 80 (those also predated the fused paths).
+  - Idle DeepEP dispatch/combine stress at 80/160/192 (200 iters, exact checks) passed under both fences; concurrent-GEMM stress at 160 also stayed numerically clean until a launch-resource abort.
+  - Mitigation landed: Primus config sets `PRIMUS_TURBO_DEEPEP_DISABLE_CHEAP_FENCE=1` (full release/acquire fence; ~2–3%/step historically). Turbo branch `fix/deep_ep/cheap-fence-high-cu` defaults to the full fence when `num_sms > 80` unless the env forces otherwise (#539 only validated ≤80 CUs).
+  - e2e confirmation: `fence1`–`fence4` queue under `/tmp/q3_e2e_fence*.log`.
+- **Unexplained e2e jitter with FP8 permute.** The first run took 6442 ms; three later runs took 6515-6549 ms with more per-iteration jitter. These causes are ruled out:
+  - the Primus change: a same-code rerun gave 6515 ms;
+  - the padding fix: permute kernel times are unchanged;
+  - the node: the `turbo_fused_grouped_gemm` baseline reran at 6815 ms.
+  The cause is not identified.
+- **Recompute**: `turbo_fp8_permute` rejects selective recompute of `moe_act`, because the checkpointed fused-MLP path is not validated with `QuantizedTensor` inputs and gradients.
+- **Untested on the final build**: the 12 MXFP8 tests in `tests/pytorch/ops/test_grouped_mlp_fp8.py`. The MXFP8 path is untouched, and FlyDSL compilation takes more than 30 minutes.
+- **Remaining hot spots** (R3 profile, rank 0, one step, taken before the fused MLP and the FP8 permute):
+
+  | kernel group | ms |
+  |---|---|
+  | attention | 1583 |
+  | grouped GEMM | 1268 |
+  | hipBLASLt | 1003 |
+  | DeepEP | 876 |
+  | quant chain | 739 |
+  | permute / unpermute | 346 |
+  | SwiGLU | 262 |
+
+## Files
+
+- `bench_*.py`: per-item microbenchmarks at Qwen3 shapes. `bench_fp8_permute.py --time` gives the kernel breakdown of the FP8 permute.
+- `trace_*.py`: kernel-time attribution from torch profiler traces.
+- `debug_*.py` and `dbg_*.py`: one-off reproducers.
+- `quant_variants/`: tensorwise quant kernel variants, source only.
+- `e2e/`: run, queue and A/B scripts, the env files, `parse_e2e.py`, and `amend_msg.py`, which writes the cumulative table into a commit message.
+- `pr_bodies/`: PR descriptions, ready to paste.

agent/workspace/qwen3_fp8_tw_gfx950_20261010/bench_attention_qwen3.py

@@ -0,0 +1,118 @@
+"""Qwen3-30B-A3B attention (sbhd, GQA 32/4, hd128, causal, bf16): TE CK vs Turbo backends.
+
+Env: SHAPE=S,B,HQ,HKV,D (default 4096,8,32,4,128), BACKENDS=te,flydsl,aiter
+"""
+
+import os
+
+os.environ.setdefault("NVTE_FLASH_ATTN", "0")
+os.environ.setdefault("NVTE_FUSED_ATTN", "1")
+os.environ.setdefault("NVTE_CK_USES_BWD_V3", "1")
+
+import torch  # noqa: E402  (torch before TE: LLVM option clash otherwise)
+
+S, B, HQ, HKV, D = (int(v) for v in os.environ.get("SHAPE", "4096,8,32,4,128").split(","))
+BACKENDS = os.environ.get("BACKENDS", "te,flydsl,aiter").split(",")
+ITERS = int(os.environ.get("ITERS", "20"))
+dev = "cuda"
+
+
+def timed(fn, iters=ITERS, warmup=5):
+    for _ in range(warmup):
+        fn()
+    torch.cuda.synchronize()
+    ts = []
+    for _ in range(iters):
+        s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
+        s.record()
+        fn()
+        e.record()
+        torch.cuda.synchronize()
+        ts.append(s.elapsed_time(e) * 1e3)
+    return sorted(ts)[len(ts) // 2]
+
+
+def make_te():
+    import transformer_engine.pytorch as te
+
+    attn = te.DotProductAttention(
+        HQ, D, num_gqa_groups=HKV, attention_dropout=0.0, attn_mask_type="causal", qkv_format="sbhd"
+    )
+
+    def run(q, k, v):
+        return attn(q, k, v).view(S, B, HQ, D)
+
+    return run
+
+
+def make_turbo(backend_name):
+    from primus_turbo.pytorch.core.backend import BackendType, GlobalBackendManager
+    from primus_turbo.pytorch.ops import flash_attn_func
+
+    backend = getattr(BackendType, backend_name.upper())
+
+    def run(q, k, v):
+        GlobalBackendManager.set_attn_backend(backend)
+        o = flash_attn_func(
+            q.permute(1, 0, 2, 3), k.permute(1, 0, 2, 3), v.permute(1, 0, 2, 3), causal=True
+        )
+        GlobalBackendManager.set_attn_backend(None)
+        return o.permute(1, 0, 2, 3)
+
+    return run
+
+
+torch.manual_seed(0)
+q0 = torch.randn(S, B, HQ, D, device=dev, dtype=torch.bfloat16)
+k0 = torch.randn(S, B, HKV, D, device=dev, dtype=torch.bfloat16)
+v0 = torch.randn(S, B, HKV, D, device=dev, dtype=torch.bfloat16)
+do = torch.randn(S, B, HQ, D, device=dev, dtype=torch.bfloat16)
+
+fwd_flops = 4 * B * HQ * S * S * D / 2
+results, outs = {}, {}
+for name in BACKENDS:
+    run = make_te() if name == "te" else make_turbo(name)
+    q, k, v = (t.clone().requires_grad_() for t in (q0, k0, v0))
+    try:
+        o = run(q, k, v)
+        o.backward(do)
+    except Exception as exc:  # noqa: BLE001
+        print(f"{name}: unsupported ({type(exc).__name__}: {str(exc)[:200]})")
+        continue
+    outs[name] = (o.detach().float(), q.grad.float(), k.grad.float(), v.grad.float())
+
+    def fwd():
+        with torch.no_grad():
+            run(q0, k0, v0)
+
+    def fwdbwd():
+        q.grad = k.grad = v.grad = None
+        run(q, k, v).backward(do)
+
+    t_f, t_fb = timed(fwd), timed(fwdbwd)
+    results[name] = (t_f, t_fb)
+    if os.environ.get("PROFILE") == "1":
+        with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA]) as prof:
+            for _ in range(5):
+                fwdbwd()
+            torch.cuda.synchronize()
+        print(f"--- {name} kernels (5 x fwd+bwd)")
+        print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=12, max_name_column_width=70))
+
+print(f"shape S={S} B={B} HQ={HQ} HKV={HKV} D={D} causal bf16")
+for name, (t_f, t_fb) in results.items():
+    t_b = t_fb - t_f
+    print(
+        f"{name:8s} fwd {t_f:8.1f} us ({fwd_flops / t_f / 1e6:6.0f} TF/s)  bwd {t_b:8.1f} us "
+        f"({2.5 * fwd_flops / t_b / 1e6:6.0f} TF/s)  fwd+bwd {t_fb:8.1f} us"
+    )
+if "te" in outs:
+    ref = outs["te"]
+    for name, got in outs.items():
+        if name == "te":
+            continue
+        diffs = " ".join(
+            f"{tag} max {(a - b).abs().max().item():.3g}"
+            for tag, a, b in zip(("o", "dq", "dk", "dv"), ref, got)
+        )
+        print(f"{name} vs te: {diffs}")

agent/workspace/qwen3_fp8_tw_gfx950_20261010/bench_deepep_qwen3.py

@@ -0,0 +1,98 @@
+"""Item 3 microbench: Turbo DeepEP intranode at Qwen3-30B-A3B EP8 (32768 tokens/rank, hidden 2048,
+128 experts, top-8). Default dispatch/combine configs (what training uses); sweeps num_sms and
+compares BF16 vs FP8 (per-128 scales) dispatch, under random top-k and Primus "even" routing.
+Each phase is timed separately with CUDA events (median over ITERS, max over ranks); the
+non-cached dispatch includes its CPU wait on the notify kernel, the cached one does not.
+"""
+
+import os
+import statistics
+import sys
+
+import torch
+import torch.distributed as dist
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../../../benchmark/ops/training"))
+from deep_ep.utils import init_dist  # noqa: E402
+
+from primus_turbo.pytorch import deep_ep  # noqa: E402
+
+T, H, E, K = 32768, 2048, 128, 8
+SMS = [int(s) for s in os.environ.get("SMS", "64,80,96").split(",")]
+ITERS = 20
+
+
+def topk_for(routing, rank):
+    if routing == "even":  # PrimusTopKRouter._force_even_routing
+        return (torch.arange(T * K, device="cuda").view(T, K) % E).to(torch.int64)
+    g = torch.Generator(device="cuda").manual_seed(rank)
+    scores = torch.randn((T, E), device="cuda", generator=g).abs() + 1
+    return torch.topk(scores, K, dim=-1, sorted=False)[1]
+
+
+def timed(fn):
+    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
+    torch.cuda.synchronize()
+    s.record()
+    out = fn()
+    e.record()
+    torch.cuda.synchronize()
+    return out, s.elapsed_time(e) * 1e3
+
+
+def worker(local_rank, num_local_ranks):
+    rank, num_ranks, group = init_dist(local_rank, num_local_ranks)
+    buf = deep_ep.Buffer(group, int(2e9), 0, low_latency_mode=False, explicitly_destroy=True)
+    x = torch.randn(T, H, device="cuda", dtype=torch.bfloat16)
+    x_fp8 = x.to(torch.float8_e4m3fn)
+    x_scales = torch.ones(T, H // 128, device="cuda", dtype=torch.float32)
+    rows = []
+    for routing in ("random", "even"):
+        topk_idx = topk_for(routing, rank)
+        topk_w = torch.rand(T, K, device="cuda", dtype=torch.float32)
+        for sms in SMS:
+            deep_ep.Buffer.set_num_sms(sms)
+            for prec in ("bf16", "fp8"):
+                inp = x if prec == "bf16" else (x_fp8, x_scales)
+                ts = {"layout": [], "dispatch": [], "dispatch_cached": [], "combine": []}
+                for it in range(ITERS + 3):
+                    dist.barrier()
+                    (ntr, _, nte, itr, _), t_l = timed(lambda: buf.get_dispatch_layout(topk_idx, E))
+                    dist.barrier()
+                    (recv, _, rw, _, handle, _), t_d = timed(
+                        lambda: buf.dispatch(
+                            inp,
+                            num_tokens_per_rank=ntr,
+                            is_token_in_rank=itr,
+                            num_tokens_per_expert=nte,
+                            topk_idx=topk_idx,
+                            topk_weights=topk_w,
+                        )
+                    )
+                    dist.barrier()
+                    _, t_dc = timed(lambda: buf.dispatch(inp, handle=handle))
+                    r = recv[0] if isinstance(recv, tuple) else recv
+                    t_c = 0.0
+                    if prec == "bf16":
+                        dist.barrier()
+                        _, t_c = timed(lambda: buf.combine(r, handle, topk_weights=rw))
+                    if it >= 3:
+                        for k, v in zip(ts, (t_l, t_d, t_dc, t_c)):
+                            ts[k].append(v)
+                med = torch.tensor([statistics.median(v) for v in ts.values()], device="cuda")
+                dist.all_reduce(med, op=dist.ReduceOp.MAX)
+                rows.append((routing, sms, prec, r.shape[0], med.tolist()))
+    if rank == 0:
+        print("routing sms prec recv_tokens | layout dispatch dispatch_cached combine (us, max over ranks)")
+        for routing, sms, prec, nrecv, m in rows:
+            print(
+                f"{routing:6s} sms={sms:3d} {prec:4s} recv={nrecv:6d} | "
+                f"layout {m[0]:6.0f}  dispatch {m[1]:6.0f}  cached {m[2]:6.0f}  combine {m[3]:6.0f}"
+            )
+    buf.destroy()
+    dist.barrier()
+    dist.destroy_process_group()
+
+
+if __name__ == "__main__":
+    torch.multiprocessing.spawn(worker, args=(8,), nprocs=8)