Rotational Unit of Memory
  The concepts of unitary evolution matrices and associative memory have
boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art
performance in a variety of sequential tasks. However, RNN still have a limited
capacity to manipulate long-term memory. To bypass this weakness the most
successful applications of RNN use external techniques such as attention
mechanisms. In this paper we propose a novel RNN model that unifies the
state-of-the-art approaches: Rotational Unit of Memory (RUM). The core of RUM
is its rotational operation, which is, naturally, a unitary matrix, providing
architectures with the power to learn long-term dependencies by overcoming the
vanishing and exploding gradients problem. Moreover, the rotational unit also
serves as associative memory. We evaluate our model on synthetic memorization,
question answering and language modeling tasks. RUM learns the Copying Memory
task completely and improves the state-of-the-art result in the Recall task.
RUM's performance in the bAbI Question Answering task is comparable to that of
models with attention mechanism. We also improve the state-of-the-art result to
1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB)
task, which is to signify the applications of RUM to real-world sequential
data. The universality of our construction, at the core of RNN, establishes RUM
as a promising approach to language modeling, speech recognition and machine
translation.
