Heavy Ball Neural Ordinary Differential Equations
  We propose heavy ball neural ordinary differential equations (HBNODEs),
leveraging the continuous limit of the classical momentum accelerated gradient
descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have
two properties that imply practical advantages over NODEs: (i) The adjoint
state of an HBNODE also satisfies an HBNODE, accelerating both forward and
backward ODE solvers, thus significantly reducing the number of function
evaluations (NFEs) and improving the utility of the trained models. (ii) The
spectrum of HBNODEs is well structured, enabling effective learning of
long-term dependencies from complex sequential data. We verify the advantages
of HBNODEs over NODEs on benchmark tasks, including image classification,
learning complex dynamics, and sequential modeling. Our method requires
remarkably fewer forward and backward NFEs, is more accurate, and learns
long-term dependencies more effectively than the other ODE-based neural network
models. Code is available at \url{https://github.com/hedixia/HeavyBallNODE}.
