Multiplicative Models for Recurrent Language Modeling
  Recently, there has been interest in multiplicative recurrent neural networks
for language modeling. Indeed, simple Recurrent Neural Networks (RNNs)
encounter difficulties recovering from past mistakes when generating sequences
due to high correlation between hidden states. These challenges can be
mitigated by integrating second-order terms in the hidden-state update. One
such model, multiplicative Long Short-Term Memory (mLSTM) is particularly
interesting in its original formulation because of the sharing of its
second-order term, referred to as the intermediate state. We explore these
architectural improvements by introducing new models and testing them on
character-level language modeling tasks. This allows us to establish the
relevance of shared parametrization in recurrent language modeling.
