Impact of classification difficulty on the weight matrices spectra in
  Deep Learning and application to early-stopping
  Much research effort has been devoted to explaining the success of deep
learning. Random Matrix Theory (RMT) provides an emerging way to this end:
spectral analysis of large random matrices involved in a trained deep neural
network (DNN) such as weight matrices or Hessian matrices with respect to the
stochastic gradient descent algorithm. To have more comprehensive understanding
of weight matrices spectra, we conduct extensive experiments on weight matrices
in different modules, e.g., layers, networks and data sets. Following the
previous work of \cite{martin2018implicit}, we classify the spectra in the
terminal stage into three main types: Light Tail (LT), Bulk Transition period
(BT) and Heavy Tail(HT). These different types, especially HT, implicitly
indicate some regularization in the DNNs. A main contribution from the paper is
that we identify the difficulty of the classification problem as a driving
factor for the appearance of heavy tail in weight matrices spectra. Higher the
classification difficulty, higher the chance for HT to appear. Moreover, the
classification difficulty can be affected by the signal-to-noise ratio of the
dataset, or by the complexity of the classification problem (complex features,
large number of classes) as well. Leveraging on this finding, we further
propose a spectral criterion to detect the appearance of heavy tails and use it
to early stop the training process without testing data. Such early stopped
DNNs have the merit of avoiding overfitting and unnecessary extra training
while preserving a much comparable generalization ability. These findings from
the paper are validated in several NNs, using Gaussian synthetic data and real
data sets (MNIST and CIFAR10).
