Deep technical topics for advanced users.
- Parallelism strategies: DP, TP, PP, SP, CP, EP, FSDP explained
- Parallelism configuration: per-backend parallelism setup and batch size relationships
- Collective operations: NCCL/RCCL operations and their role in each parallelism strategy
- Performance tuning: HipBLASLt, Primus-Turbo, FP8, MoE optimization
- MoE training deep-dive: bottlenecks and Primus-Turbo optimizations for Mixture-of-Experts models
- MegaMoE fused MoE layer: FlyDSL-based fused MoE layer for EP-only bf16 training, setup and reproduction
- Data preparation: tokenization, data formats, mock data
- Checkpoint management: formats, save/load, distributed checkpointing
- Multi-node networking: InfiniBand, RoCE, AINIC configuration
- AINIC bundle versions: rebuilding a training image against a different AINIC bundle
- Profiling and observability: Torch profiler, TraceLens, memory snapshots, projection, pp_vis
- Logging and experiment tracking: TensorBoard, WandB, MLflow setup per backend
- Fault tolerance and elastic training: graceful exit, auto-resume, in-process restart, torchft
- Determinism and reproducibility: deterministic mode, seeds, trade-offs
- Diffusion models: Flux diffusion architecture, data pipeline, and FP8 / MXFP4 training
- Hybrid models: Hylo hybrid hybrid recurrent-attention (Mamba/KDA/GDN + MLA) models, FLA-parity recipes, and checkpoint conversion
- Native SFT and LoRA: Megatron-native SFT/LoRA runbook (BF16 / FP8 / FP4), no Megatron-Bridge dependency
- Native SFT LoRA on MI455X: 1-GPU gfx1250 recipes (Llama-3.2-1B + 70B/72B/235B proxies)