Error Autocorrelation Objective Function for Improved System Modeling
  Deep learning models are trained to minimize the error between the model's
output and the actual values. The typical cost function, the Mean Squared Error
(MSE), arises from maximizing the log-likelihood of additive independent,
identically distributed Gaussian noise. However, minimizing MSE fails to
minimize the residuals' cross-correlations, leading to over-fitting and poor
extrapolation of the model outside the training set (generalization). In this
paper, we introduce a "whitening" cost function, the Ljung-Box statistic, which
not only minimizes the error but also minimizes the correlations between
errors, ensuring that the fits enforce compatibility with an independent and
identically distributed (i.i.d) gaussian noise model. The results show
significant improvement in generalization for recurrent neural networks (RNNs)
(1d) and image autoencoders (2d). Specifically, we look at both temporal
correlations for system-id in simulated and actual mechanical systems. We also
look at spatial correlation in vision autoencoders to demonstrate that the
whitening objective functions lead to much better extrapolation--a property
very desirable for reliable control systems.
