Anomalous diffusion dynamics of learning in deep neural networks
  Learning in deep neural networks (DNNs) is implemented through minimizing a
highly non-convex loss function, typically by a stochastic gradient descent
(SGD) method. This learning process can effectively find good wide minima
without being trapped in poor local ones. We present a novel account of how
such effective deep learning emerges through the interactions of the SGD and
the geometrical structure of the loss landscape. Rather than being a normal
diffusion process (i.e. Brownian motion) as often assumed, we find that the SGD
exhibits rich, complex dynamics when navigating through the loss landscape;
initially, the SGD exhibits anomalous superdiffusion, which attenuates
gradually and changes to subdiffusion at long times when the solution is
reached. Such learning dynamics happen ubiquitously in different DNNs such as
ResNet and VGG-like networks and are insensitive to batch size and learning
rate. The anomalous superdiffusion process during the initial learning phase
indicates that the motion of SGD along the loss landscape possesses
intermittent, big jumps; this non-equilibrium property enables the SGD to
escape from sharp local minima. By adapting the methods developed for studying
energy landscapes in complex physical systems, we find that such superdiffusive
learning dynamics are due to the interactions of the SGD and the fractal-like
structure of the loss landscape. We further develop a simple model to
demonstrate the mechanistic role of the fractal loss landscape in enabling the
SGD to effectively find global minima. Our results thus reveal the
effectiveness of deep learning from a novel perspective and have implications
for designing efficient deep neural networks.
