Order Optimal One-Shot Federated Learning for non-Convex Loss Functions
  We consider the problem of federated learning in a one-shot setting in which
there are $m$ machines, each observing $n$ samples function from an unknown
distribution on non-convex loss functions. Let $F:[-1,1]^d\to\mathbb{R}$ be the
expected loss function with respect to this unknown distribution. The goal is
to find an estimate of the minimizer of $F$. Based on its observations, each
machine generates a signal of bounded length $B$ and sends it to a server. The
sever collects signals of all machines and outputs an estimate of the minimizer
of $F$. We propose a distributed learning algorithm, called Multi-Resolution
Estimator for Non-Convex loss function (MRE-NC), whose expected error is
bounded by $\max\big(1/\sqrt{n}(mB)^{1/d}, 1/\sqrt{mn}\big)$, up to
polylogarithmic factors. We also provide a matching lower bound on the
performance of any algorithm, showing that MRE-NC is order optimal in terms of
$n$ and $m$. Experiments on synthetic and real data show the effectiveness of
MRE-NC in distributed learning of model's parameters for non-convex loss
functions.
