Full-Capacity Unitary Recurrent Neural Networks
  Recurrent neural networks are powerful models for processing sequential data,
but they are generally plagued by vanishing and exploding gradient problems.
Unitary recurrent neural networks (uRNNs), which use unitary recurrence
matrices, have recently been proposed as a means to avoid these issues.
However, in previous experiments, the recurrence matrices were restricted to be
a product of parameterized unitary matrices, and an open question remains: when
does such a parameterization fail to represent all unitary matrices, and how
does this restricted representational capacity limit what can be learned? To
address this question, we propose full-capacity uRNNs that optimize their
recurrence matrix over all unitary matrices, leading to significantly improved
performance over uRNNs that use a restricted-capacity recurrence matrix. Our
contribution consists of two main components. First, we provide a theoretical
argument to determine if a unitary parameterization has restricted capacity.
Using this argument, we show that a recently proposed unitary parameterization
has restricted capacity for hidden state dimension greater than 7. Second, we
show how a complete, full-capacity unitary recurrence matrix can be optimized
over the differentiable manifold of unitary matrices. The resulting
multiplicative gradient step is very simple and does not require gradient
clipping or learning rate adaptation. We confirm the utility of our claims by
empirically evaluating our new full-capacity uRNNs on both synthetic and
natural data, achieving superior performance compared to both LSTMs and the
original restricted-capacity uRNNs.
