Federated Learning with Sparsified Model Perturbation: Improving
  Accuracy under Client-Level Differential Privacy
  Federated learning (FL) that enables distributed clients to collaboratively
learn a shared statistical model while keeping their training data locally has
received great attention recently and can improve privacy and communication
efficiency in comparison with traditional centralized machine learning
paradigm. However, sensitive information about the training data can still be
inferred from model updates shared in FL. Differential privacy (DP) is the
state-of-the-art technique to defend against those attacks. The key challenge
to achieve DP in FL lies in the adverse impact of DP noise on model accuracy,
particularly for deep learning models with large numbers of model parameters.
This paper develops a novel differentially-private FL scheme named Fed-SMP that
provides client-level DP guarantee while maintaining high model accuracy. To
mitigate the impact of privacy protection on model accuracy, Fed-SMP leverages
a new technique called Sparsified Model Perturbation (SMP), where local models
are sparsified first before being perturbed with additive Gaussian noise. Two
sparsification strategies are considered in Fed-SMP: random sparsification and
top-$k$ sparsification. We also apply R{\'e}nyi differential privacy to
providing a tight analysis for the end-to-end DP guarantee of Fed-SMP and prove
the convergence of Fed-SMP with general loss functions. Extensive experiments
on real-world datasets are conducted to demonstrate the effectiveness of
Fed-SMP in largely improving model accuracy with the same level of DP guarantee
and saving communication cost simultaneously.
