FedEC enhancing the performance of Federated Optimization, thanks for Karhou Tam for publish a library framework of Federated Learning, I have used some function from your copyright library, especially MOON architecture and kind of how to synthesis dataset for deploy model training.
Federated learning is a key method for addressing data privacy and security in distributed AI training. However, non-IID data among local clients often causes issues like client drift, leading to slow and unstable model convergence. Current studies typically use one-sided methods that optimize either the client or server side with single algorithms like FedAvg. In contrast, we introduce FedEC, a two-sided strategy that combines distinct methods: an elastic aggregation algorithm for optimizing the global model on the server and contrastive learning techniques on the client side to reduce divergence between local and global models. This complementary approach fosters mutual reinforcement between client and server, allowing FedEC to better tackle non-IID data challenges. Results from experiments conducted on some benchmark datasets with different settings show that FedEC offers more efficient training and outperforms previous one-sided algorithms, underscoring the effectiveness of this two-sided approach in federated learning.
The proposed FedEC is a two-sided method that integrates two complementary approaches: elastic aggregation on the server side and model-contrastive federated learning on the client side. The two-sided method is an approach that simultaneously uses two optimization algorithms on both the server and client sides, instead of employing a single algorithm on only one side. Elastic aggregation demonstrates its effectiveness by enhancing aggregation in federated learning using sensitivity information from client models, while MOON is designed to improve the accuracy of the global model by leveraging information from both local and server models. Both methods have been successful in mitigating client parameter drift when dealing with non-IID data. Therefore, FedEC takes advantage of the benefits and strengths of both approaches to create a more robust model with higher accuracy while ensuring the convergence of federated learning. However, it is important to note that combining these two methods may increase computational costs, as MOON requires an additional loss function during local training, and Elastic Aggregation requires sensitivity calculations based on the transmitted information. Nonetheless, given the benefits it offers, we believe that this is a worthwhile investment.
By combining the strengths of the elastic aggregation and model-contrastive federated learning algorithms, the process operates in two parallel steps: local optimization using MOON and aggregation optimization using Elastic. This dual approach is key to enhancing overall performance of the algorithm. First, the local objective supported by the model-contrastive at client-level impacts each client by using the Eq.2 to update and optimize based on individual data. Subsequently, elastic aggregation is employed instead of the standard averaging method used in FedAvg. Clients compute sensitivity coefficients to evaluate and send them back to the server for use in the elastic aggregation process. Thus, elastic aggregation requires not only the transmission of the local model gradients but also an additional set of parameters representing the sensitivities.
In this paper, we developed optimization techniques in Federated Learning by proposing FedEC, a combination of Elastic Aggregation and Model-contrastive Federated Learning. FedEC consistently outperformed baselines across all datasets and distributions. On MNIST (α=0.1), FedEC achieved 92.96% accuracy, surpassing baselines by 1.41% to 4.78%. For CIFAR-10, FedEC reached 65.35% (α=0.1) and 67.42% (α=100), outperforming the best baseline by 4.66% and 0.66%, respectively. On CIFAR-100, FedEC attained 27.15% (α=0.1) and 30.45% (α=100), surpassing baselines by up to 1.96%. However, limitations such as computational load, communication cost, and insufficient use of state-of-the-art optimization techniques impacted convergence. In future research, we will explore knowledge distillation techniques, personalized federated learning, and more comprehensive testing with diverse models and datasets. Additionally, we will continue analyzing parameter sensitivity and propose light adjustments to further improve performance in solving machine learning and AI challenges.