Exploiting the Full Capacity of Deep Neural Networks while Avoiding
  Overfitting by Targeted Sparsity Regularization
  Overfitting is one of the most common problems when training deep neural
networks on comparatively small datasets. Here, we demonstrate that neural
network activation sparsity is a reliable indicator for overfitting which we
utilize to propose novel targeted sparsity visualization and regularization
strategies. Based on these strategies we are able to understand and counteract
overfitting caused by activation sparsity and filter correlation in a targeted
layer-by-layer manner. Our results demonstrate that targeted sparsity
regularization can efficiently be used to regularize well-known datasets and
architectures with a significant increase in image classification performance
while outperforming both dropout and batch normalization. Ultimately, our study
reveals novel insights into the contradicting concepts of activation sparsity
and network capacity by demonstrating that targeted sparsity regularization
enables salient and discriminative feature learning while exploiting the full
capacity of deep models without suffering from overfitting, even when trained
excessively.
