Orthogonal Weight Normalization: Solution to Optimization over Multiple
  Dependent Stiefel Manifolds in Deep Neural Networks
  Orthogonal matrix has shown advantages in training Recurrent Neural Networks
(RNNs), but such matrix is limited to be square for the hidden-to-hidden
transformation in RNNs. In this paper, we generalize such square orthogonal
matrix to orthogonal rectangular matrix and formulating this problem in
feed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent
Stiefel Manifolds (OMDSM). We show that the rectangular orthogonal matrix can
stabilize the distribution of network activations and regularize FNNs. We also
propose a novel orthogonal weight normalization method to solve OMDSM.
Particularly, it constructs orthogonal transformation over proxy parameters to
ensure the weight matrix is orthogonal and back-propagates gradient information
through the transformation during training. To guarantee stability, we minimize
the distortions between proxy parameters and canonical weights over all
tractable orthogonal transformations. In addition, we design an orthogonal
linear module (OLM) to learn orthogonal filter banks in practice, which can be
used as an alternative to standard linear module. Extensive experiments
demonstrate that by simply substituting OLM for standard linear module without
revising any experimental protocols, our method largely improves the
performance of the state-of-the-art networks, including Inception and residual
networks on CIFAR and ImageNet datasets. In particular, we have reduced the
test error of wide residual network on CIFAR-100 from 20.04% to 18.61% with
such simple substitution. Our code is available online for result reproduction.
