Stochastic gradient descent with random learning rate
  We propose to optimize neural networks with a uniformly-distributed random
learning rate. The associated stochastic gradient descent algorithm can be
approximated by continuous stochastic equations and analyzed within the
Fokker-Planck formalism. In the small learning rate regime, the training
process is characterized by an effective temperature which depends on the
average learning rate, the mini-batch size and the momentum of the optimization
algorithm. By comparing the random learning rate protocol with cyclic and
constant protocols, we suggest that the random choice is generically the best
strategy in the small learning rate regime, yielding better regularization
without extra computational cost. We provide supporting evidence through
experiments on both shallow, fully-connected and deep, convolutional neural
networks for image classification on the MNIST and CIFAR10 datasets.
