Suspected numerical underflow on Nvidia RTX 6000 Ada

[g-luo]

Documenting a reported hardware issue in case folks experience this!

Observed Behavior

• On Nvidia RTX 6000 Ada, when hosting the image generator and VLM on two different GPUs and running the smiling example via ./run_dual_process.sh, the loss oscillates and does not converge.
• However, the run does converge if the image generator and VLM are hosted the same GPU.
• For different hardware, there is no issue observed. Specifically, on Nvidia RTX 3090, 4090, A6000, and A100, the models can be hosted on two different GPUs.

Recommendation

• If you see this behavior, for now I would suggest either (1) switching to a supported GPU or (2) trying to fit the two models onto the same GPU.
• It seems like the issue comes from passing very small gradient values from one GPU to another, and how different hardware handles this.

[g-luo]

cc @reachomk

[qiisun]

Thanks a lot, but I found some problem with in my NVIDIA-A6000 when trying "flux-schnell + idefics2 + man-smile" case. Here is my results.

I will update my results when I fix it.