Private and Communication-Efficient Edge Learning: A Sparse Differential
  Gaussian-Masking Distributed SGD Approach
  With rise of machine learning (ML) and the proliferation of smart mobile
devices, recent years have witnessed a surge of interest in performing ML in
wireless edge networks. In this paper, we consider the problem of jointly
improving data privacy and communication efficiency of distributed edge
learning, both of which are critical performance metrics in wireless edge
network computing. Toward this end, we propose a new decentralized stochastic
gradient method with sparse differential Gaussian-masked stochastic gradients
(SDM-DSGD) for non-convex distributed edge learning. Our main contributions are
three-fold: i) We theoretically establish the privacy and communication
efficiency performance guarantee of our SDM-DSGD method, which outperforms all
existing works; ii) We show that SDM-DSGD improves the fundamental
training-privacy trade-off by {\em two orders of magnitude} compared with the
state-of-the-art. iii) We reveal theoretical insights and offer practical
design guidelines for the interactions between privacy preservation and
communication efficiency, two conflicting performance goals. We conduct
extensive experiments with a variety of learning models on MNIST and CIFAR-10
datasets to verify our theoretical findings. Collectively, our results
contribute to the theory and algorithm design for distributed edge learning.
