Accuracy and Architecture Studies of Residual Neural Network solving
  Ordinary Differential Equations
  In this paper we consider utilizing a residual neural network (ResNet) to
solve ordinary differential equations. Stochastic gradient descent method is
applied to obtain the optimal parameter set of weights and biases of the
network. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite
difference methods to generate three sets of targets training the ResNet and
carry out the target study. The well trained ResNet behaves just as its
counterpart of the corresponding one-step finite difference method. In
particular, we carry out (1) the architecture study in terms of number of
hidden layers and neurons per layer to find the optimal ResNet structure; (2)
the target study to verify the ResNet solver behaves as accurate as its finite
difference method counterpart; (3) solution trajectory simulation. Even the
ResNet solver looks like and is implemented in a way similar to forward Euler
scheme, its accuracy can be as high as any one step method. A sequence of
numerical examples are presented to demonstrate the performance of the ResNet
solver.
