Noisin: Unbiased Regularization for Recurrent Neural Networks
  Recurrent neural networks (RNNs) are powerful models of sequential data. They
have been successfully used in domains such as text and speech. However, RNNs
are susceptible to overfitting; regularization is important. In this paper we
develop Noisin, a new method for regularizing RNNs. Noisin injects random noise
into the hidden states of the RNN and then maximizes the corresponding marginal
likelihood of the data. We show how Noisin applies to any RNN and we study many
different types of noise. Noisin is unbiased--it preserves the underlying RNN
on average. We characterize how Noisin regularizes its RNN both theoretically
and empirically. On language modeling benchmarks, Noisin improves over dropout
by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We
also compared the state-of-the-art language model of Yang et al. 2017, both
with and without Noisin. On the Penn Treebank, the method with Noisin more
quickly reaches state-of-the-art performance.
