An Efficient and Effective Second-Order Training Algorithm for
  LSTM-based Adaptive Learning
  We study adaptive (or online) nonlinear regression with
Long-Short-Term-Memory (LSTM) based networks, i.e., LSTM-based adaptive
learning. In this context, we introduce an efficient Extended Kalman filter
(EKF) based second-order training algorithm. Our algorithm is truly online,
i.e., it does not assume any underlying data generating process and future
information, except that the target sequence is bounded. Through an extensive
set of experiments, we demonstrate significant performance gains achieved by
our algorithm with respect to the state-of-the-art methods. Here, we mainly
show that our algorithm consistently provides 10 to 45\% improvement in the
accuracy compared to the widely-used adaptive methods Adam, RMSprop, and DEKF,
and comparable performance to EKF with a 10 to 15 times reduction in the
run-time.
