On the Convergence of mSGD and AdaGrad for Stochastic Optimization
  As one of the most fundamental stochastic optimization algorithms, stochastic
gradient descent (SGD) has been intensively developed and extensively applied
in machine learning in the past decade. There have been some modified SGD-type
algorithms, which outperform the SGD in many competitions and applications in
terms of convergence rate and accuracy, such as momentum-based SGD (mSGD) and
adaptive gradient algorithm (AdaGrad). Despite these empirical successes, the
theoretical properties of these algorithms have not been well established due
to technical difficulties. With this motivation, we focus on convergence
analysis of mSGD and AdaGrad for any smooth (possibly non-convex) loss
functions in stochastic optimization. First, we prove that the iterates of mSGD
are asymptotically convergent to a connected set of stationary points with
probability one, which is more general than existing works on subsequence
convergence or convergence of time averages. Moreover, we prove that the loss
function of mSGD decays at a certain rate faster than that of SGD. In addition,
we prove the iterates of AdaGrad are asymptotically convergent to a connected
set of stationary points with probability one. Also, this result extends the
results from the literature on subsequence convergence and the convergence of
time averages. Despite the generality of the above convergence results, we have
relaxed some assumptions of gradient noises, convexity of loss functions, as
well as boundedness of iterates.
