Approximated Orthonormal Normalisation in Training Neural Networks
  Generalisation of a deep neural network (DNN) is one major concern when
employing the deep learning approach for solving practical problems. In this
paper we propose a new technique, named approximated orthonormal normalisation
(AON), to improve the generalisation capacity of a DNN model. Considering a
weight matrix W from a particular neural layer in the model, our objective is
to design a function h(W) such that its row vectors are approximately
orthogonal to each other while allowing the DNN model to fit the training data
sufficiently accurate. By doing so, it would avoid co-adaptation among neurons
of the same layer to be able to improve network-generalisation capacity.
Specifically, at each iteration, we first approximate (WW^T)^(-1/2) using its
Taylor expansion before multiplying the matrix W. After that, the matrix
product is then normalised by applying the spectral normalisation (SN)
technique to obtain h(W). Conceptually speaking, AON is designed to turn
orthonormal regularisation into orthonormal normalisation to avoid manual
balancing the original and penalty functions. Experimental results show that
AON yields promising validation performance compared to orthonormal
regularisation.
