A Convergence Analysis of Nesterov's Accelerated Gradient Method in
  Training Deep Linear Neural Networks
  Momentum methods, including heavy-ball~(HB) and Nesterov's accelerated
gradient~(NAG), are widely used in training neural networks for their fast
convergence. However, there is a lack of theoretical guarantees for their
convergence and acceleration since the optimization landscape of the neural
network is non-convex. Nowadays, some works make progress towards understanding
the convergence of momentum methods in an over-parameterized regime, where the
number of the parameters exceeds that of the training instances. Nonetheless,
current results mainly focus on the two-layer neural network, which are far
from explaining the remarkable success of the momentum methods in training deep
neural networks. Motivated by this, we investigate the convergence of NAG with
constant learning rate and momentum parameter in training two architectures of
deep linear networks: deep fully-connected linear neural networks and deep
linear ResNets. Based on the over-parameterization regime, we first analyze the
residual dynamics induced by the training trajectory of NAG for a deep
fully-connected linear neural network under the random Gaussian initialization.
Our results show that NAG can converge to the global minimum at a $(1 -
\mathcal{O}(1/\sqrt{\kappa}))^t$ rate, where $t$ is the iteration number and
$\kappa > 1$ is a constant depending on the condition number of the feature
matrix. Compared to the $(1 - \mathcal{O}(1/{\kappa}))^t$ rate of GD, NAG
achieves an acceleration over GD. To the best of our knowledge, this is the
first theoretical guarantee for the convergence of NAG to the global minimum in
training deep neural networks. Furthermore, we extend our analysis to deep
linear ResNets and derive a similar convergence result.
