FedSel: Federated SGD under Local Differential Privacy with Top-k
  Dimension Selection
  As massive data are produced from small gadgets, federated learning on mobile
devices has become an emerging trend. In the federated setting, Stochastic
Gradient Descent (SGD) has been widely used in federated learning for various
machine learning models. To prevent privacy leakages from gradients that are
calculated on users' sensitive data, local differential privacy (LDP) has been
considered as a privacy guarantee in federated SGD recently. However, the
existing solutions have a dimension dependency problem: the injected noise is
substantially proportional to the dimension $d$. In this work, we propose a
two-stage framework FedSel for federated SGD under LDP to relieve this problem.
Our key idea is that not all dimensions are equally important so that we
privately select Top-k dimensions according to their contributions in each
iteration of federated SGD. Specifically, we propose three private dimension
selection mechanisms and adapt the gradient accumulation technique to stabilize
the learning process with noisy updates. We also theoretically analyze privacy,
accuracy and time complexity of FedSel, which outperforms the state-of-the-art
solutions. Experiments on real-world and synthetic datasets verify the
effectiveness and efficiency of our framework.
