Weight Expansion: A New Perspective on Dropout and Generalization
  While dropout is known to be a successful regularization technique, insights
into the mechanisms that lead to this success are still lacking. We introduce
the concept of \emph{weight expansion}, an increase in the signed volume of a
parallelotope spanned by the column or row vectors of the weight covariance
matrix, and show that weight expansion is an effective means of increasing the
generalization in a PAC-Bayesian setting. We provide a theoretical argument
that dropout leads to weight expansion and extensive empirical support for the
correlation between dropout and weight expansion. To support our hypothesis
that weight expansion can be regarded as an \emph{indicator} of the enhanced
generalization capability endowed by dropout, and not just as a mere
by-product, we have studied other methods that achieve weight expansion (resp.\
contraction), and found that they generally lead to an increased (resp.\
decreased) generalization ability. This suggests that dropout is an attractive
regularizer, because it is a computationally cheap method for obtaining weight
expansion. This insight justifies the role of dropout as a regularizer, while
paving the way for identifying regularizers that promise improved
generalization through weight expansion.
