On Convergence of Training Loss Without Reaching Stationary Points
  It is a well-known fact that nonconvex optimization is computationally
intractable in the worst case. As a result, theoretical analysis of
optimization algorithms such as gradient descent often focuses on local
convergence to stationary points where the gradient norm is zero or negligible.
In this work, we examine the disconnect between the existing theoretical
analysis of gradient-based algorithms and actual practice. Specifically, we
provide numerical evidence that in large-scale neural network training, such as
in ImageNet, ResNet, and WT103 + TransformerXL models, the Neural Network
weight variables do not converge to stationary points where the gradient of the
loss function vanishes. Remarkably, however, we observe that while weights do
not converge to stationary points, the value of the loss function converges.
Inspired by this observation, we propose a new perspective based on ergodic
theory of dynamical systems. We prove convergence of the distribution of weight
values to an approximate invariant measure (without smoothness assumptions)
that explains this phenomenon. We further discuss how this perspective can
better align the theory with empirical observations.
