On the Implicit Bias of Gradient Descent for Temporal Extrapolation
  When using recurrent neural networks (RNNs) it is common practice to apply
trained models to sequences longer than those seen in training. This
"extrapolating" usage deviates from the traditional statistical learning setup
where guarantees are provided under the assumption that train and test
distributions are identical. Here we set out to understand when RNNs can
extrapolate, focusing on a simple case where the data generating distribution
is memoryless. We first show that even with infinite training data, there exist
RNN models that interpolate perfectly (i.e., they fit the training data) yet
extrapolate poorly to longer sequences. We then show that if gradient descent
is used for training, learning will converge to perfect extrapolation under
certain assumptions on initialization. Our results complement recent studies on
the implicit bias of gradient descent, showing that it plays a key role in
extrapolation when learning temporal prediction models.
