[tdene]

What does this PR do ?

Updates default Megatron Inference settings.

Effects:

• grpo-nanov3-30BA3B-4n4g-megatron_generation-noncolocated-mxfp8-rollouts has total_step_time drop from 23.6 to 17.5, while generation time drops from 15.6 to 9.3.

• grpo-nanov3-30BA3B-4n4g-megatron_generation-noncolocated-mxfp8-rollouts-packed-refit has total_step_time drop from 24.5 to 18.3, while generation time drops from 15.6 to 9.2.

• grpo-qwen2.5-math-1.5b-instruct-1n8g-megatron_generation-single-controller-async-lag4 drops exposed generation time from 2.45 to 1.68, and various other megatron_generation tests see similar improvements on Blackwell.

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

The base branch was changed.

[yuki-97]

Thanks for retuning the Megatron inference defaults and for retiring the stale buffer_guaranteed_fraction / unified_memory_level keys behind a startup error that names the replacement. The PR switches num_cuda_graphs to autotune, enables prefix caching by default, moves the nanov3 MXFP8 recipes to FlashInfer, and derives the fp32 router for inference-optimized MoE.

Great to see the before/after step and generation times for the two nanov3 MXFP8 recipes and the lag4 Qwen recipe posted in the description; the effect of the new defaults can be read off directly.

Scope: default/recipe/config changes, the new token-capacity check and the removed-key rejection, at a50dcd6

Checked:

• MXFP8 grouped-GEMM validation removal: MCore's resolve_mxfp8_backend returns the same storage for torch, flashinfer and vllm, so dropping the NeMo-RL-side guard (and its omitted-key special case) is consistent with upstream.
• Prefix caching default: MCore's DynamicInferenceEngine.resume() bumps _weight_epoch and re-salts the prefix cache, so enabling it does not reuse KV computed under pre-refit weights.
• Removed-key rejection in outdated_config_checks.py:141: generalizing the single offload check into one table keeps the error text per key, and no YAML, doc or test in the repo still carries buffer_guaranteed_fraction or unified_memory_level, so no in-tree config can trip it.
• Token-capacity check in megatron_worker.py:707: the decode-row bound matches the formula MCore's text generation controller uses to engage the bounded MXFP8 path, so the new error fires exactly when the capacity could never apply.

Remaining comments are on the router dtype in setup.py:1395, which is overridden to fp32 silently and should raise instead when it is not fp32.

[yuki-97]

changes LGTM, will need to wait #4802 get into main first, otherwise CI will fail.

[yuki-97]

/ok to test a511e2b

The base branch was changed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[tdene]

/ok to test c3bd883

[yuki-97]

changed to run with CI:Lfast, so that we can cover e2e runs.

[tdene]

/ok to test dccc2a7