QuPeD: Quantized Personalization via Distillation with Applications to
  Federated Learning
  Traditionally, federated learning (FL) aims to train a single global model
while collaboratively using multiple clients and a server. Two natural
challenges that FL algorithms face are heterogeneity in data across clients and
collaboration of clients with {\em diverse resources}. In this work, we
introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeD
that facilitates collective (personalized model compression) training via
\textit{knowledge distillation} (KD) among clients who have access to
heterogeneous data and resources. For personalization, we allow clients to
learn \textit{compressed personalized models} with different quantization
parameters and model dimensions/structures. Towards this, first we propose an
algorithm for learning quantized models through a relaxed optimization problem,
where quantization values are also optimized over. When each client
participating in the (federated) learning process has different requirements
for the compressed model (both in model dimension and precision), we formulate
a compressed personalization framework by introducing knowledge distillation
loss for local client objectives collaborating through a global model. We
develop an alternating proximal gradient update for solving this compressed
personalization problem, and analyze its convergence properties. Numerically,
we validate that QuPeD outperforms competing personalized FL methods, FedAvg,
and local training of clients in various heterogeneous settings.
