Stochastic Optimization with Non-stationary Noise
  We investigate stochastic optimization problems under relaxed assumptions on
the distribution of noise that are motivated by empirical observations in
neural network training. Standard results on optimal convergence rates for
stochastic optimization assume either there exists a uniform bound on the
moments of the gradient noise, or that the noise decays as the algorithm
progresses. These assumptions do not match the empirical behavior of
optimization algorithms used in neural network training where the noise level
in stochastic gradients could even increase with time. We address this behavior
by studying convergence rates of stochastic gradient methods subject to
changing second moment (or variance) of the stochastic oracle as the iterations
progress. When the variation in the noise is known, we show that it is always
beneficial to adapt the step-size and exploit the noise variability. When the
noise statistics are unknown, we obtain similar improvements by developing an
online estimator of the noise level, thereby recovering close variants of
RMSProp. Consequently, our results reveal an important scenario where adaptive
stepsize methods outperform SGD.
