Recurrent Attention Unit
  Recurrent Neural Network (RNN) has been successfully applied in many sequence
learning problems. Such as handwriting recognition, image description, natural
language processing and video motion analysis. After years of development,
researchers have improved the internal structure of the RNN and introduced many
variants. Among others, Gated Recurrent Unit (GRU) is one of the most widely
used RNN model. However, GRU lacks the capability of adaptively paying
attention to certain regions or locations, so that it may cause information
redundancy or loss during leaning. In this paper, we propose a RNN model,
called Recurrent Attention Unit (RAU), which seamlessly integrates the
attention mechanism into the interior of GRU by adding an attention gate. The
attention gate can enhance GRU's ability to remember long-term memory and help
memory cells quickly discard unimportant content. RAU is capable of extracting
information from the sequential data by adaptively selecting a sequence of
regions or locations and pay more attention to the selected regions during
learning. Extensive experiments on image classification, sentiment
classification and language modeling show that RAU consistently outperforms GRU
and other baseline methods.
