Skip to content

Add support for converting RMSNorm when using transformer-engine #20804

@cyanic-selkie

Description

@cyanic-selkie

Description & Motivation

Recently, models (e.g., Qwen3/Llama) have replaced the LayerNorm layers with the RMSNorm layers because they perform identically but are slightly more efficient.

To support it, a new branch almost identical to the LayerNorm would have to be added to the conversion function.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureIs an improvement or enhancementneeds triageWaiting to be triaged by maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions