Quasi Global Momentum (QGM) combined with Stochastic Gradient Push (SGP)

This implementation modifies the facebook's SGP codebase to include quasi global momentum (QGM) updates for heterogenous data distribution. The modified algorithm works for both directed and unidirected graphs with time varying structures in a given decelentralized setup.

References

1. Lin, T., Karimireddy, S.P., Stich, S. & Jaggi, M.. (2021). Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6654-6665 Available here.
2. Assran, M., Loizou, N., Ballas, N. & Rabbat, M.. (2019). Stochastic Gradient Push for Distributed Deep Learning. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:344-353 Available here.