A More Stable Accelerated Gradient Method Inspired by Continuous-Time
  Perspective
  Nesterov's accelerated gradient method (NAG) is widely used in problems with
machine learning background including deep learning, and is corresponding to a
continuous-time differential equation. From this connection, the property of
the differential equation and its numerical approximation can be investigated
to improve the accelerated gradient method. In this work we present a new
improvement of NAG in terms of stability inspired by numerical analysis. We
give the precise order of NAG as a numerical approximation of its
continuous-time limit and then present a new method with higher order. We show
theoretically that our new method is more stable than NAG for large step size.
Experiments of matrix completion and handwriting digit recognition demonstrate
that the stability of our new method is better. Furthermore, better stability
leads to higher computational speed in experiments.
