Fast Convergence for Stochastic and Distributed Gradient Descent in the
  Interpolation Limit
  Modern supervised learning techniques, particularly those using deep nets,
involve fitting high dimensional labelled data sets with functions containing
very large numbers of parameters. Much of this work is empirical. Interesting
phenomena have been observed that require theoretical explanations; however the
non-convexity of the loss functions complicates the analysis. Recently it has
been proposed that the success of these techniques rests partly in the
effectiveness of the simple stochastic gradient descent algorithm in the so
called interpolation limit in which all labels are fit perfectly. This analysis
is made possible since the SGD algorithm reduces to a stochastic linear system
near the interpolating minimum of the loss function. Here we exploit this
insight by presenting and analyzing a new distributed algorithm for gradient
descent, also in the interpolating limit. The distributed SGD algorithm
presented in the paper corresponds to gradient descent applied to a simple
penalized distributed loss function, $L({\bf w}_1,...,{\bf w}_n) = \Sigma_i
l_i({\bf w}_i) + \mu \sum_{<i,j>}|{\bf w}_i-{\bf w}_j|^2$. Here each node holds
only one sample, and its own parameter vector. The notation $<i,j>$ denotes
edges of a connected graph defining the links between nodes. It is shown that
this distributed algorithm converges linearly (ie the error reduces
exponentially with iteration number), with a rate
$1-\frac{\eta}{n}\lambda_{min}(H)<R<1$ where $\lambda_{min}(H)$ is the smallest
nonzero eigenvalue of the sample covariance or the Hessian H. In contrast with
previous usage of similar penalty functions to enforce consensus between nodes,
in the interpolating limit it is not required to take the penalty parameter to
infinity for consensus to occur. The analysis further reinforces the utility of
the interpolation limit in the theoretical treatment of modern machine learning
algorithms.
