Speeding up Heterogeneous Federated Learning with Sequentially Trained
  Superclients
  Federated Learning (FL) allows training machine learning models in
privacy-constrained scenarios by enabling the cooperation of edge devices
without requiring local data sharing. This approach raises several challenges
due to the different statistical distribution of the local datasets and the
clients' computational heterogeneity. In particular, the presence of highly
non-i.i.d. data severely impairs both the performance of the trained neural
network and its convergence rate, increasing the number of communication rounds
requested to reach a performance comparable to that of the centralized
scenario. As a solution, we propose FedSeq, a novel framework leveraging the
sequential training of subgroups of heterogeneous clients, i.e. superclients,
to emulate the centralized paradigm in a privacy-compliant way. Given a fixed
budget of communication rounds, we show that FedSeq outperforms or match
several state-of-the-art federated algorithms in terms of final performance and
speed of convergence. Finally, our method can be easily integrated with other
approaches available in the literature. Empirical results show that combining
existing algorithms with FedSeq further improves its final performance and
convergence speed. We test our method on CIFAR-10 and CIFAR-100 and prove its
effectiveness in both i.i.d. and non-i.i.d. scenarios.
