On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep
  Neural Networks
  The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is
often considered to be Gaussian in the large data regime by assuming that the
\emph{classical} central limit theorem (CLT) kicks in. This assumption is often
made for mathematical convenience, since it enables SGD to be analyzed as a
stochastic differential equation (SDE) driven by a Brownian motion. We argue
that the Gaussianity assumption might fail to hold in deep learning settings
and hence render the Brownian motion-based analyses inappropriate. Inspired by
non-Gaussian natural phenomena, we consider the GN in a more general context
and invoke the \emph{generalized} CLT, which suggests that the GN converges to
a \emph{heavy-tailed} $\alpha$-stable random vector, where \emph{tail-index}
$\alpha$ determines the heavy-tailedness of the distribution. Accordingly, we
propose to analyze SGD as a discretization of an SDE driven by a L\'{e}vy
motion. Such SDEs can incur `jumps', which force the SDE and its discretization
\emph{transition} from narrow minima to wider minima, as proven by existing
metastability theory and the extensions that we proved recently. In this study,
under the $\alpha$-stable GN assumption, we further establish an explicit
connection between the convergence rate of SGD to a local minimum and the
tail-index $\alpha$. To validate the $\alpha$-stable assumption, we conduct
experiments on common deep learning scenarios and show that in all settings,
the GN is highly non-Gaussian and admits heavy-tails. We investigate the tail
behavior in varying network architectures and sizes, loss functions, and
datasets. Our results open up a different perspective and shed more light on
the belief that SGD prefers wide minima.
