Decentralised Learning with Random Features and Distributed Gradient
  Descent
  We investigate the generalisation performance of Distributed Gradient Descent
with Implicit Regularisation and Random Features in the homogenous setting
where a network of agents are given data sampled independently from the same
unknown distribution. Along with reducing the memory footprint, Random Features
are particularly convenient in this setting as they provide a common
parameterisation across agents that allows to overcome previous difficulties in
implementing Decentralised Kernel Regression. Under standard source and
capacity assumptions, we establish high probability bounds on the predictive
performance for each agent as a function of the step size, number of
iterations, inverse spectral gap of the communication matrix and number of
Random Features. By tuning these parameters, we obtain statistical rates that
are minimax optimal with respect to the total number of samples in the network.
The algorithm provides a linear improvement over single machine Gradient
Descent in memory cost and, when agents hold enough data with respect to the
network size and inverse spectral gap, a linear speed-up in computational
runtime for any network topology. We present simulations that show how the
number of Random Features, iterations and samples impact predictive
performance.
