Add an AutoModel Path for EMTTS In Addition to Local NemotronH file - #16036
Add an AutoModel Path for EMTTS In Addition to Local NemotronH file#16036blisc wants to merge 2 commits into
Conversation
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: Jason <jasoli@nvidia.com>
| """ | ||
|
|
||
| @staticmethod | ||
| def _to_container(cfg_value): |
There was a problem hiding this comment.
IMO would be cleaner to have some of these utilities as private free functions at the bottom of the file rather than class-level static methods. My rule of thumb: if it's generic (eg config/dict manipulation), make it a function; if it's tightly coupled with the class, make it a method.
| logging.info("NeMo AutoModel kwargs: %s", automodel_kwargs) | ||
| # Some Automodel NemotronV3 builds leave mixer tensors from scratch init uninitialized after | ||
| # from_config(); reset those tensors before running Automodel's regular init/rescaling path. | ||
| with torch.device('cpu'): |
There was a problem hiding this comment.
If you're going to train Nano-sized or larger models with distributed tensor sharding, you'll need to initialize the model on "meta" device and move to GPU directly later to avoid a spike in GPU memory or super long CPU init time. You can consult SALMAutomodel.configure_model() for reference how to put it together with PTL trainer
| @staticmethod | ||
| @torch.no_grad() | ||
| def _initialize_automodel_scratch_parameters(module: nn.Module): | ||
| """Reset NemotronV3 tensors that some Automodel scratch-init paths leave uninitialized.""" |
There was a problem hiding this comment.
CC @akoumpa @hemildesai looks like some parameters in Automodel Nemotron3 implementation are not being initialized when training from scratch.
2 participants
What does this PR do ?
Add an AutoModel Path for EMTTS In Addition to Local NemotronH file
Collection: tts
Changelog
PR Type: