Adaptive Recurrent Neural Network Based on Mixture Layer
  Although Recurrent Neural Network (RNN) has been a powerful tool for modeling
sequential data, its performance is inadequate when processing sequences with
multiple patterns. In this paper, we address this challenge by introducing a
novel mixture layer and constructing an adaptive RNN. The mixture layer
augmented RNN (termed as M-RNN) partitions patterns in training sequences into
several clusters and stores the principle patterns as prototype vectors of
components in a mixture model. By leveraging the mixture layer, the proposed
method can adaptively update states according to the similarities between
encoded inputs and prototype vectors, leading to a stronger capacity in
assimilating sequences with multiple patterns. Moreover, our approach can be
further extended by taking advantage of prior knowledge about data. Experiments
on both synthetic and real datasets demonstrate the effectiveness of the
proposed method.
