On Multiplicative Integration with Recurrent Neural Networks
  We introduce a general and simple structural design called Multiplicative
Integration (MI) to improve recurrent neural networks (RNNs). MI changes the
way in which information from difference sources flows and is integrated in the
computational building block of an RNN, while introducing almost no extra
parameters. The new structure can be easily embedded into many popular RNN
models, including LSTMs and GRUs. We empirically analyze its learning behaviour
and conduct evaluations on several tasks using different RNN models. Our
experimental results demonstrate that Multiplicative Integration can provide a
substantial performance boost over many of the existing RNN models.
