Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic
  Gradient Descent using Stochastic Learning Rates
  Large-scale optimization problems require algorithms both effective and
efficient. One such popular and proven algorithm is Stochastic Gradient Descent
which uses first-order gradient information to solve these problems. This paper
studies almost-sure convergence rates of the Stochastic Gradient Descent method
when instead of deterministic, its learning rate becomes stochastic. In
particular, its learning rate is equipped with a multiplicative stochasticity,
producing a stochastic learning rate scheme. Theoretical results show
accelerated almost-sure convergence rates of Stochastic Gradient Descent in a
nonconvex setting when using an appropriate stochastic learning rate, compared
to a deterministic-learning-rate scheme. The theoretical results are verified
empirically.
