Shuffled Model of Federated Learning: Privacy, Communication and
  Accuracy Trade-offs
  We consider a distributed empirical risk minimization (ERM) optimization
problem with communication efficiency and privacy requirements, motivated by
the federated learning (FL) framework. Unique challenges to the traditional ERM
problem in the context of FL include (i) need to provide privacy guarantees on
clients' data, (ii) compress the communication between clients and the server,
since clients might have low-bandwidth links, (iii) work with a dynamic client
population at each round of communication between the server and the clients,
as a small fraction of clients are sampled at each round. To address these
challenges we develop (optimal) communication-efficient schemes for private
mean estimation for several $\ell_p$ spaces, enabling efficient gradient
aggregation for each iteration of the optimization solution of the ERM. We also
provide lower and upper bounds for mean estimation with privacy and
communication constraints for arbitrary $\ell_p$ spaces. To get the overall
communication, privacy, and optimization performance operation point, we
combine this with privacy amplification opportunities inherent to this setup.
Our solution takes advantage of the inherent privacy amplification provided by
client sampling and data sampling at each client (through Stochastic Gradient
Descent) as well as the recently developed privacy framework using
anonymization, which effectively presents to the server responses that are
randomly shuffled with respect to the clients. Putting these together, we
demonstrate that one can get the same privacy, optimization-performance
operating point developed in recent methods that use full-precision
communication, but at a much lower communication cost, i.e., effectively
getting communication efficiency for "free".
