On orthogonality and learning recurrent networks with long term
  dependencies
  It is well known that it is challenging to train deep neural networks and
recurrent neural networks for tasks that exhibit long term dependencies. The
vanishing or exploding gradient problem is a well known issue associated with
these challenges. One approach to addressing vanishing and exploding gradients
is to use either soft or hard constraints on weight matrices so as to encourage
or enforce orthogonality. Orthogonal matrices preserve gradient norm during
backpropagation and may therefore be a desirable property. This paper explores
issues with optimization convergence, speed and gradient stability when
encouraging or enforcing orthogonality. To perform this analysis, we propose a
weight matrix factorization and parameterization strategy through which we can
bound matrix norms and therein control the degree of expansivity induced during
backpropagation. We find that hard constraints on orthogonality can negatively
affect the speed of convergence and model performance.
