Layer-Parallel Training of Deep Residual Neural Networks
  Residual neural networks (ResNets) are a promising class of deep neural
networks that have shown excellent performance for a number of learning tasks,
e.g., image classification and recognition. Mathematically, ResNet
architectures can be interpreted as forward Euler discretizations of a
nonlinear initial value problem whose time-dependent control variables
represent the weights of the neural network. Hence, training a ResNet can be
cast as an optimal control problem of the associated dynamical system. For
similar time-dependent optimal control problems arising in engineering
applications, parallel-in-time methods have shown notable improvements in
scalability. This paper demonstrates the use of those techniques for efficient
and effective training of ResNets. The proposed algorithms replace the
classical (sequential) forward and backward propagation through the network
layers by a parallel nonlinear multigrid iteration applied to the layer domain.
This adds a new dimension of parallelism across layers that is attractive when
training very deep networks. From this basic idea, we derive multiple
layer-parallel methods. The most efficient version employs a simultaneous
optimization approach where updates to the network parameters are based on
inexact gradient information in order to speed up the training process. Using
numerical examples from supervised classification, we demonstrate that the new
approach achieves similar training performance to traditional methods, but
enables layer-parallelism and thus provides speedup over layer-serial methods
through greater concurrency.
