Ternary Compression for Communication-Efficient Federated Learning
  Learning over massive data stored in different locations is essential in many
real-world applications. However, sharing data is full of challenges due to the
increasing demands of privacy and security with the growing use of smart mobile
devices and IoT devices. Federated learning provides a potential solution to
privacy-preserving and secure machine learning, by means of jointly training a
global model without uploading data distributed on multiple devices to a
central server. However, most existing work on federated learning adopts
machine learning models with full-precision weights, and almost all these
models contain a large number of redundant parameters that do not need to be
transmitted to the server, consuming an excessive amount of communication
costs. To address this issue, we propose a federated trained ternary
quantization (FTTQ) algorithm, which optimizes the quantized networks on the
clients through a self-learning quantization factor. Theoretical proofs of the
convergence of quantization factors, unbiasedness of FTTQ, as well as a reduced
weight divergence are given. On the basis of FTTQ, we propose a ternary
federated averaging protocol (T-FedAvg) to reduce the upstream and downstream
communication of federated learning systems. Empirical experiments are
conducted to train widely used deep learning models on publicly available
datasets, and our results demonstrate that the proposed T-FedAvg is effective
in reducing communication costs and can even achieve slightly better
performance on non-IID data in contrast to the canonical federated learning
algorithms.
