Predictive Local Smoothness for Stochastic Gradient Methods
  Stochastic gradient methods are dominant in nonconvex optimization especially
for deep models but have low asymptotical convergence due to the fixed
smoothness. To address this problem, we propose a simple yet effective method
for improving stochastic gradient methods named predictive local smoothness
(PLS). First, we create a convergence condition to build a learning rate which
varies adaptively with local smoothness. Second, the local smoothness can be
predicted by the latest gradients. Third, we use the adaptive learning rate to
update the stochastic gradients for exploring linear convergence rates. By
applying the PLS method, we implement new variants of three popular algorithms:
PLS-stochastic gradient descent (PLS-SGD), PLS-accelerated SGD (PLS-AccSGD),
and PLS-AMSGrad. Moreover, we provide much simpler proofs to ensure their
linear convergence. Empirical results show that the variants have better
performance gains than the popular algorithms, such as, faster convergence and
alleviating explosion and vanish of gradients.
