Industrial Forecasting with Exponentially Smoothed Recurrent Neural
  Networks
  Time series modeling has entered an era of unprecedented growth in the size
and complexity of data which require new modeling approaches. While many new
general purpose machine learning approaches have emerged, they remain poorly
understand and irreconcilable with more traditional statistical modeling
approaches. We present a general class of exponential smoothed recurrent neural
networks (RNNs) which are well suited to modeling non-stationary dynamical
systems arising in industrial applications. In particular, we analyze their
capacity to characterize the non-linear partial autocorrelation structure of
time series and directly capture dynamic effects such as seasonality and
trends. Application of exponentially smoothed RNNs to forecasting electricity
load, weather data, and stock prices highlight the efficacy of exponential
smoothing of the hidden state for multi-step time series forecasting. The
results also suggest that popular, but more complicated neural network
architectures originally designed for speech processing, such as LSTMs and
GRUs, are likely over-engineered for industrial forecasting and light-weight
exponentially smoothed architectures, trained in a fraction of the time,
capture the salient features while being superior and more robust than simple
RNNs and ARIMA models. Additionally uncertainty quantification of the
exponential smoothed recurrent neural networks, provided by Bayesian
estimation, is shown to provide improved coverage.
