Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged
  Learning
  Recent work has focused on data-driven learning of the evolution of unknown
systems via deep neural networks (DNNs), with the goal of conducting long time
prediction of the evolution of the unknown system. Training a DNN with low
generalization error is a particularly important task in this case as error is
accumulated over time. Because of the inherent randomness in DNN training,
chiefly in stochastic optimization, there is uncertainty in the resulting
prediction, and therefore in the generalization error. Hence, the
generalization error can be viewed as a random variable with some probability
distribution. Well-trained DNNs, particularly those with many hyperparameters,
typically result in probability distributions for generalization error with low
bias but high variance. High variance causes variability and unpredictably in
the results of a trained DNN. This paper presents a computational technique
which decreases the variance of the generalization error, thereby improving the
reliability of the DNN model to generalize consistently. In the proposed
ensemble averaging method, multiple models are independently trained and model
predictions are averaged at each time step. A mathematical foundation for the
method is presented, including results regarding the distribution of the local
truncation error. In addition, three time-dependent differential equation
problems are considered as numerical examples, demonstrating the effectiveness
of the method to decrease variance of DNN predictions generally.
