Clustered Sampling: Low-Variance and Improved Representativity for
  Clients Selection in Federated Learning
  This work addresses the problem of optimizing communications between server
and clients in federated learning (FL). Current sampling approaches in FL are
either biased, or non optimal in terms of server-clients communications and
training stability. To overcome this issue, we introduce \textit{clustered
sampling} for clients selection. We prove that clustered sampling leads to
better clients representatitivity and to reduced variance of the clients
stochastic aggregation weights in FL. Compatibly with our theory, we provide
two different clustering approaches enabling clients aggregation based on 1)
sample size, and 2) models similarity. Through a series of experiments in
non-iid and unbalanced scenarios, we demonstrate that model aggregation through
clustered sampling consistently leads to better training convergence and
variability when compared to standard sampling approaches. Our approach does
not require any additional operation on the clients side, and can be seamlessly
integrated in standard FL implementations. Finally, clustered sampling is
compatible with existing methods and technologies for privacy enhancement, and
for communication reduction through model compression.
