Learning to Adaptively Scale Recurrent Neural Networks
  Recent advancements in recurrent neural network (RNN) research have
demonstrated the superiority of utilizing multiscale structures in learning
temporal representations of time series. Currently, most of multiscale RNNs use
fixed scales, which do not comply with the nature of dynamical temporal
patterns among sequences. In this paper, we propose Adaptively Scaled Recurrent
Neural Networks (ASRNN), a simple but efficient way to handle this problem.
Instead of using predefined scales, ASRNNs are able to learn and adjust scales
based on different temporal contexts, making them more flexible in modeling
multiscale patterns. Compared with other multiscale RNNs, ASRNNs are bestowed
upon dynamical scaling capabilities with much simpler structures, and are easy
to be integrated with various RNN cells. The experiments on multiple sequence
modeling tasks indicate ASRNNs can efficiently adapt scales based on different
sequence contexts and yield better performances than baselines without
dynamical scaling abilities.
