DeeperSpeed 3.0 is based off upstream DeepSpeed 0.16.5 and is intended to closely match the upstream along with:
Any additional fixes we find that are yet to be accepted into DeepSpeed
Branches for optimal performance on EleutherAI's compute providers (e.g. stability, coreweave)