Direct Runge-Kutta Discretization Achieves Acceleration
  We study gradient-based optimization methods obtained by directly
discretizing a second-order ordinary differential equation (ODE) related to the
continuous limit of Nesterov's accelerated gradient method. When the function
is smooth enough, we show that acceleration can be achieved by a stable
discretization of this ODE using standard Runge-Kutta integrators.
Specifically, we prove that under Lipschitz-gradient, convexity and
order-$(s+2)$ differentiability assumptions, the sequence of iterates generated
by discretizing the proposed second-order ODE converges to the optimal solution
at a rate of $\mathcal{O}({N^{-2\frac{s}{s+1}}})$, where $s$ is the order of
the Runge-Kutta numerical integrator. Furthermore, we introduce a new local
flatness condition on the objective, under which rates even faster than
$\mathcal{O}(N^{-2})$ can be achieved with low-order integrators and only
gradient information. Notably, this flatness condition is satisfied by several
standard loss functions used in machine learning. We provide numerical
experiments that verify the theoretical rates predicted by our results.
