CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics
  with Simulated Annealing
  Deep learning applications require global optimization of non-convex
objective functions, which have multiple local minima. The same problem is
often found in physical simulations and may be resolved by the methods of
Langevin dynamics with Simulated Annealing, which is a well-established
approach for minimization of many-particle potentials. This analogy provides
useful insights for non-convex stochastic optimization in machine learning.
Here we find that integration of the discretized Langevin equation gives a
coordinate updating rule equivalent to the famous Momentum optimization
algorithm. As a main result, we show that a gradual decrease of the momentum
coefficient from the initial value close to unity until zero is equivalent to
application of Simulated Annealing or slow cooling, in physical terms. Making
use of this novel approach, we propose CoolMomentum -- a new stochastic
optimization method. Applying Coolmomentum to optimization of Resnet-20 on
Cifar-10 dataset and Efficientnet-B0 on Imagenet, we demonstrate that it is
able to achieve high accuracies.
