Accelerating Training in Artificial Neural Networks with Dynamic Mode
  Decomposition
  Training of deep neural networks (DNNs) frequently involves optimizing
several millions or even billions of parameters. Even with modern computing
architectures, the computational expense of DNN training can inhibit, for
instance, network architecture design optimization, hyper-parameter studies,
and integration into scientific research cycles. The key factor limiting
performance is that both the feed-forward evaluation and the back-propagation
rule are needed for each weight during optimization in the update rule. In this
work, we propose a method to decouple the evaluation of the update rule at each
weight. At first, Proper Orthogonal Decomposition (POD) is used to identify a
current estimate of the principal directions of evolution of weights per layer
during training based on the evolution observed with a few backpropagation
steps. Then, Dynamic Mode Decomposition (DMD) is used to learn the dynamics of
the evolution of the weights in each layer according to these principal
directions. The DMD model is used to evaluate an approximate converged state
when training the ANN. Afterward, some number of backpropagation steps are
performed, starting from the DMD estimates, leading to an update to the
principal directions and DMD model. This iterative process is repeated until
convergence. By fine-tuning the number of backpropagation steps used for each
DMD model estimation, a significant reduction in the number of operations
required to train the neural networks can be achieved. In this paper, the DMD
acceleration method will be explained in detail, along with the theoretical
justification for the acceleration provided by DMD. This method is illustrated
using a regression problem of key interest for the scientific machine learning
community: the prediction of a pollutant concentration field in a diffusion,
advection, reaction problem.
