The Importance of the Current Input in Sequence Modeling
  The last advances in sequence modeling are mainly based on deep learning
approaches. The current state of the art involves the use of variations of the
standard LSTM architecture, combined with several tricks that improve the final
prediction rates of the trained neural networks. However, in some cases, these
adaptations might be too much tuned to the particular problems being addressed.
In this article, we show that a very simple idea, to add a direct connection
between the input and the output, skipping the recurrent module, leads to an
increase of the prediction accuracy in sequence modeling problems related to
natural language processing. Experiments carried out on different problems show
that the addition of this kind of connection to a recurrent network always
improves the results, regardless of the architecture and training-specific
details. When this idea is introduced into the models that lead the field, the
resulting networks achieve a new state-of-the-art perplexity in language
modeling problems.
