Support BF16 activations in the Numba RNN-T losses - #15996
Open
MahmoudAshraf97 wants to merge 6 commits into
Open
Support BF16 activations in the Numba RNN-T losses#15996MahmoudAshraf97 wants to merge 6 commits into
MahmoudAshraf97 wants to merge 6 commits into
Conversation
The three Numba grad kernels wrote `grad` to the output buffer, read it straight back, clamped it, and wrote it again. Clamp the value while it is still an FP32 register instead: this drops a global read plus a redundant write per vocabulary element per (b, t, u), and stops narrow output dtypes from being rounded twice. FP32 loss and gradients are bit-identical before and after. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
Numba-CUDA cannot implicitly unify BF16 with the FP32 dynamic-programming state, so BF16 activations make kernel type inference diverge and the launch fails with RecursionError. Widen activation reads at the point of use in logp() and logp_duration(), which keeps `acts` narrow in memory and does the arithmetic in FP32. Also allocate the CTA reduction scratch as the module FP32 dtype rather than the activation dtype, so reductions no longer accumulate in the input precision. Standard RNN-T now runs under BF16. Multi-blank and TDT still fail earlier, in the workspace allocation. FP32 results are unchanged. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
compute_grad_kernel, compute_multiblank_grad_kernel and compute_tdt_grad_kernel open-coded `denom[col] + acts[...]` and `duration_acts[...]` instead of calling logp() and logp_duration(). Use the helpers, which removes the duplicated index arithmetic and picks up their FP32 promotion. No uncast activation read remains in the Numba RNN-T kernels. FP32 results are unchanged. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
The workspace backs FP32 dynamic-programming state -- softmax denominator, alphas, betas, log-likelihoods -- but multiblank_rnnt_loss_gpu and tdt_loss_gpu sized it with the activation dtype. With BF16 activations the denominator buffer reached the reduction kernel as a 2-byte opaque dtype and the launch failed with NumbaNotImplementedError. rnnt_loss_gpu already hardcodes FP32; make the other two consistent. get_workspace_size returns an element count, so this changes only the buffer dtype, not its length. Multi-blank and TDT now run under BF16. FP32 results are unchanged. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
Covers standard, multi-blank and TDT across fastemit and clamp settings, asserting each compiles under BF16 and agrees with FP32. The regression these guard is a kernel compilation failure rather than numerical drift, so the assertion that matters most is that the launch succeeds at all. Gated only on Numba CUDA support: BF16 needs no NUMBA_CUDA_USE_NVIDIA_BINDING at the Numba level -- that variable gates whether NeMo routes narrow activations to the loss, not whether the kernels can consume them -- so these run in ordinary CI rather than silently skipping. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
The `### ... ###` banners trip flake8's E266. Linting runs over the files a change touches, so these pre-existing violations otherwise fail CI for any branch that edits this module. Comment text is unchanged. Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Numba CUDA cannot unify BF16 with the FP32 dynamic-programming state the RNN-T kernels
use. Passing BF16 activations fails at kernel compile time:
RecursionErrorfrom typeinference for standard RNN-T,
NumbaNotImplementedErrorfor multi-blank and TDT.I came across this when benchmarking BF16 in numba when implementing a tiled rnnt loss, it will be added in a followup PR, it achieves 2x speedup and 10x less memory usage
Collection: asr
Changelog
logp,logp_durationandthe reduction kernels. Activations stay narrow in memory and the arithmetic runs in FP32.
logphelpers rather than repeating theindex arithmetic inline.
denominator, alphas, betas and log-likelihoods, but was sized from the activation
dtype.
rnnt_loss_gpualready allocated FP32.output buffer, reading it back, clamping, and writing again.
### ... ###banner comments as#innemo/collections/asr/parts/numba/rnnt_loss/rnnt.py. Those E266 violations predate thisbranch, but linting runs over changed files and fails the job, so any PR editing that
module has to clear them first.
Results
Joint and loss region, forward and backward, B=32 T=400 U=128 V=1024, RTX 5090:
21% faster, 24% less memory. FP32 loss and gradients are bit-identical before and after.
Tests
TestRNNTLossNumbaBFloat16adds 12 cases covering standard, multi-blank and TDT acrossfastemit and clamp settings. Each asserts the kernels compile under BF16 and agree with
FP32. The regression these guard is a compile failure rather than numerical drift.
This also fixes two pre-existing FP16 failures in
test_reduce.py, which now passes 4/4.The five FP16 failures in
test_gpu_rnnt_kernel.pypredate this branch and are unchanged.Usage
BF16 reaches the kernels through
NUMBA_CUDA_USE_NVIDIA_BINDING=1, which already setsforce_float32=Falseforwarprnnt_numba.GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information