AL2: Progressive Activation Loss for Learning General Representations in
  Classification Neural Networks
  The large capacity of neural networks enables them to learn complex
functions. To avoid overfitting, networks however require a lot of training
data that can be expensive and time-consuming to collect. A common practical
approach to attenuate overfitting is the use of network regularization
techniques. We propose a novel regularization method that progressively
penalizes the magnitude of activations during training. The combined activation
signals produced by all neurons in a given layer form the representation of the
input image in that feature space. We propose to regularize this representation
in the last feature layer before classification layers. Our method's effect on
generalization is analyzed with label randomization tests and cumulative
ablations. Experimental results show the advantages of our approach in
comparison with commonly-used regularizers on standard benchmark datasets.
