Learning Long-Term Dependencies in Irregularly-Sampled Time Series
  Recurrent neural networks (RNNs) with continuous-time hidden states are a
natural fit for modeling irregularly-sampled time series. These models,
however, face difficulties when the input data possess long-term dependencies.
We prove that similar to standard RNNs, the underlying reason for this issue is
the vanishing or exploding of the gradient during training. This phenomenon is
expressed by the ordinary differential equation (ODE) representation of the
hidden state, regardless of the ODE solver's choice. We provide a solution by
designing a new algorithm based on the long short-term memory (LSTM) that
separates its memory from its time-continuous state. This way, we encode a
continuous-time dynamical flow within the RNN, allowing it to respond to inputs
arriving at arbitrary time-lags while ensuring a constant error propagation
through the memory path. We call these RNN models ODE-LSTMs. We experimentally
show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly
sampled data with long-term dependencies. All code and data is available at
https://github.com/mlech26l/ode-lstms.
