[feature request] FSDP2 wen?

[jramapuram]

CUDA backend is 🔥

Are there any plans to get FSDP2 (per param arbitrary mesh sharding) in?

[awni]

Thanks! We don't have concrete plans for this yet. Some activities in this direction are underway. CC @nastya236 you could chat with about it.

[zcbenz]

Closing since we don't have any plan on this.

[nastya236]

Actually I added fsdp couple of weeks ago :) It is almost closed by #3194 and #3130. But we don't shard model parameters yet, only optimizer state, it should be clear from the docs.