Reducing Overfitting in Deep Networks by Decorrelating Representations
  One major challenge in training Deep Neural Networks is preventing
overfitting. Many techniques such as data augmentation and novel regularizers
such as Dropout have been proposed to prevent overfitting without requiring a
massive amount of training data. In this work, we propose a new regularizer
called DeCov which leads to significantly reduced overfitting (as indicated by
the difference between train and val performance), and better generalization.
Our regularizer encourages diverse or non-redundant representations in Deep
Neural Networks by minimizing the cross-covariance of hidden activations. This
simple intuition has been explored in a number of past works but surprisingly
has never been applied as a regularizer in supervised learning. Experiments
across a range of datasets and network architectures show that this loss always
reduces overfitting while almost always maintaining or increasing
generalization performance and often improving performance over Dropout.
