Skip to content

Commit b65426a

Browse files
committed
docs(norm): reconcile the C1 norm page, docstrings and gated tests with the stacked C1 branch
Carried over from the former merge of the C1 branch into this one: the decoder-norm page again states the registration and the check_operator command, the gated tests use the contract's reduction tolerances through _forward_tol, and the reference docstrings name the gated page. Same final tree as before the re-stack. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
1 parent b1e2efd commit b65426a

3 files changed

Lines changed: 14 additions & 16 deletions

File tree

‎docs/operators/qwen3-next-rms-norm.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,11 @@ y = rmsnorm_cuda(x, weight, eps=1e-6, weight_offset=1.0)
4949

5050
## Dispatch Behavior
5151

52-
Not registered on this branch: there is no `qwen3_next_rms_norm` gtest spec and no
53-
registry entry yet. Both arrive with the gated-norm PR. Until then, construct the
54-
ops directly as in "Entry Point". The CUDA op validates the compiled symbols in
55-
`__init__`, so on a build without the extension construction raises instead of
56-
handing out an op that fails at call time.
52+
Registered as the `qwen3_next_rms_norm` gtest operator. On CUDA the registry
53+
prefers `Qwen3NextRMSNormCudaOp`; every other platform resolves to the PyTorch
54+
reference. The CUDA op validates the compiled symbols in `__init__`, so on a build
55+
without the extension construction raises and the registry falls through to the
56+
reference rather than handing out an op that fails at call time.
5757

5858
## Accuracy
5959

@@ -102,16 +102,16 @@ element and no extra memory traffic.
102102

103103
```bash
104104
python -m pytest tests/test_qwen3_next_norm.py -v
105+
python scripts/check_operator.py --op qwen3_next_rms_norm --candidate cuda \
106+
--device cuda --dtype bf16 --check-grad
105107
```
106108

107109
## Known Limitations
108110

109111
- CUDA only; no ROCm, Ascend or Triton backend.
110-
- Not registered as a gtest operator or in the registry on this branch (see
111-
"Dispatch Behavior").
112112
- Not bitwise against vLLM (see Accuracy). An L2 claim needs a single source of
113113
truth for the forward on both sides, per RFC #428 §0 item 1.
114114
- The gated pair (`Qwen3NextRMSNormGatedOp`, `Qwen3NextRMSNormGatedHFOp`) is
115-
documented with its CUDA kernel in the gated-norm PR, not on this page.
115+
documented with its CUDA kernel on [Gated RMSNorm](qwen3-next-rms-norm-gated.md).
116116
- Measured on sm_100 (B200). Per RFC #428 §2.2 no claim carries across
117117
H100/H200/B100/B200.

‎rl_engine/kernels/ops/pytorch/norm/qwen3_next_rms_norm.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@
2222
All three reuse :func:`shape_invariant_rstd`, a fixed-order reduction. They
2323
reproduce the weight convention and cast order, not vLLM's reduction tree, and are
2424
not bitwise equal to any vLLM path probed so far. Claim levels, measurements and
25-
limitations are in ``docs/operators/qwen3-next-rms-norm.md``.
25+
limitations are in ``docs/operators/qwen3-next-rms-norm.md`` and, for the gated
26+
pair, ``docs/operators/qwen3-next-rms-norm-gated.md``.
2627
"""
2728

2829
from __future__ import annotations
@@ -61,7 +62,7 @@ class Qwen3NextRMSNormGatedOp:
6162
``out = (x * rstd * weight) * silu(gate)``, with every multiply in fp32 and
6263
a single cast on the way out. This is the convention vLLM's ``RMSNormGated``
6364
uses with ``norm_before_gate=True``. Which gated convention is the strict
64-
default is still open; see the operator page.
65+
default is still open; see ``docs/operators/qwen3-next-rms-norm-gated.md``.
6566
6667
Not a subclass of the plain op: it takes an extra tensor and its epilogue
6768
differs, so it is not a drop-in substitute for one.

‎tests/test_qwen3_next_norm.py‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -475,9 +475,6 @@ def test_cuda_zero_centred_backward_is_offset_aware():
475475
not _HAS_CUDA_GATED, reason="gated RMSNorm CUDA extension is not available"
476476
)
477477

478-
# tolerance_contract.json, judgments/forward_accuracy/by_op_class/reduction/bfloat16
479-
_BF16_ATOL, _BF16_RTOL = 2e-2, 1.6e-2
480-
481478

482479
def _gated_cuda_inputs(seed=0, rows=512, hidden=_HEAD_V_DIM, dtype=torch.bfloat16):
483480
g = torch.Generator(device="cuda").manual_seed(seed)
@@ -494,7 +491,7 @@ def test_cuda_gated_matches_golden_within_contract():
494491
x, w, gate = _gated_cuda_inputs()
495492
got = Qwen3NextRMSNormGatedCudaOp().forward(x, w, gate, eps=_EPS)
496493
ref = Qwen3NextRMSNormGatedOp().forward_fp32(x, w, gate, eps=_EPS)
497-
torch.testing.assert_close(got.float(), ref, atol=_BF16_ATOL, rtol=_BF16_RTOL)
494+
torch.testing.assert_close(got.float(), ref, **_forward_tol(torch.bfloat16))
498495

499496

500497
@requires_cuda_gated
@@ -551,7 +548,7 @@ def test_cuda_gated_unit_weight_is_plain_norm_times_silu():
551548
ones = torch.ones(_HEAD_V_DIM, device="cuda", dtype=x.dtype)
552549
got = Qwen3NextRMSNormGatedCudaOp().forward(x, ones, gate, eps=_EPS).float()
553550
ref = rmsnorm_cuda(x, ones, eps=_EPS).float() * F.silu(gate.float())
554-
torch.testing.assert_close(got, ref, atol=_BF16_ATOL, rtol=_BF16_RTOL)
551+
torch.testing.assert_close(got, ref, **_forward_tol(torch.bfloat16))
555552

556553

557554
@requires_cuda_gated
@@ -561,7 +558,7 @@ def test_cuda_gated_sigmoid_activation():
561558
x, w, gate = _gated_cuda_inputs()
562559
got = rmsnorm_gated_cuda(x, w, gate, eps=_EPS, activation="sigmoid").float()
563560
ref = rmsnorm_cuda(x, w, eps=_EPS).float() * torch.sigmoid(gate.float())
564-
torch.testing.assert_close(got, ref, atol=_BF16_ATOL, rtol=_BF16_RTOL)
561+
torch.testing.assert_close(got, ref, **_forward_tol(torch.bfloat16))
565562

566563

567564
@requires_cuda_gated

0 commit comments

Comments
 (0)