HeunNet: Extending ResNet using Heun's Methods
  There is an analogy between the ResNet (Residual Network) architecture for
deep neural networks and an Euler solver for an ODE. The transformation
performed by each layer resembles an Euler step in solving an ODE. We consider
the Heun Method, which involves a single predictor-corrector cycle, and
complete the analogy, building a predictor-corrector variant of ResNet, which
we call a HeunNet. Just as Heun's method is more accurate than Euler's,
experiments show that HeunNet achieves high accuracy with low computational
(both training and test) time compared to both vanilla recurrent neural
networks and other ResNet variants.
