Implicit Self-Regularization in Deep Neural Networks: Evidence from
  Random Matrix Theory and Implications for Learning
  Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep
Neural Networks (DNNs), including both production quality, pre-trained models
such as AlexNet and Inception, and smaller models trained from scratch, such as
LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly
indicate that the DNN training process itself implicitly implements a form of
Self-Regularization. The empirical spectral density (ESD) of DNN layer matrices
displays signatures of traditionally-regularized statistical models, even in
the absence of exogenously specifying traditional forms of explicit
regularization. Building on relatively recent results in RMT, most notably its
extension to Universality classes of Heavy-Tailed matrices, we develop a theory
to identify 5+1 Phases of Training, corresponding to increasing amounts of
Implicit Self-Regularization. These phases can be observed during the training
process as well as in the final learned DNNs. For smaller and/or older DNNs,
this Implicit Self-Regularization is like traditional Tikhonov regularization,
in that there is a "size scale" separating signal from noise. For
state-of-the-art DNNs, however, we identify a novel form of Heavy-Tailed
Self-Regularization, similar to the self-organization seen in the statistical
physics of disordered systems. This results from correlations arising at all
size scales, which arises implicitly due to the training process itself. This
implicit Self-Regularization can depend strongly on the many knobs of the
training process. By exploiting the generalization gap phenomena, we
demonstrate that we can cause a small model to exhibit all 5+1 phases of
training simply by changing the batch size. This demonstrates that---all else
being equal---DNN optimization with larger batch sizes leads to less-well
implicitly-regularized models, and it provides an explanation for the
generalization gap phenomena.
