Convolutional Neural Networks combined with Runge-Kutta Methods
  A convolutional neural network for image classification can be constructed
mathematically since it can be regarded as a multi-period dynamical system. In
this paper, a novel approach is proposed to construct network models from the
dynamical systems view. Since a pre-activation residual network can be deemed
an approximation of a time-dependent dynamical system using the forward Euler
method, higher order Runge-Kutta methods (RK methods) can be utilized to build
network models in order to achieve higher accuracy. The model constructed in
such a way is referred to as the Runge-Kutta Convolutional Neural Network
(RKNet). RK methods also provide an interpretation of Dense Convolutional
Networks (DenseNets) and Convolutional Neural Networks with Alternately Updated
Clique (CliqueNets) from the dynamical systems view. The proposed methods are
evaluated on benchmark datasets: CIFAR-10/100, SVHN and ImageNet. The
experimental results are consistent with the theoretical properties of RK
methods and support the dynamical systems interpretation. Moreover, the
experimental results show that the RKNets are superior to the state-of-the-art
network models on CIFAR-10 and on par on CIFAR-100, SVHN and ImageNet.
