How neural networks find generalizable solutions: Self-tuned annealing
  in deep learning
  Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm
in deep learning, little is known about how SGD finds generalizable solutions
in the high-dimensional weight space. By analyzing the learning dynamics and
loss function landscape, we discover a robust inverse relation between the
weight variance and the landscape flatness (inverse of curvature) for all
SGD-based learning algorithms. To explain the inverse variance-flatness
relation, we develop a random landscape theory, which shows that the SGD noise
strength (effective temperature) depends inversely on the landscape flatness.
Our study indicates that SGD attains a self-tuned landscape-dependent annealing
strategy to find generalizable solutions at the flat minima of the landscape.
Finally, we demonstrate how these new theoretical insights lead to more
efficient algorithms, e.g., for avoiding catastrophic forgetting.
