Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth
  Non-Convex Optimization
  Stochastic gradient methods with momentum are widely used in applications and
at the core of optimization subroutines in many popular machine learning
libraries. However, their sample complexities have not been obtained for
problems beyond those that are convex or smooth. This paper establishes the
convergence rate of a stochastic subgradient method with a momentum term of
Polyak type for a broad class of non-smooth, non-convex, and constrained
optimization problems. Our key innovation is the construction of a special
Lyapunov function for which the proven complexity can be achieved without any
tuning of the momentum parameter. For smooth problems, we extend the known
complexity bound to the constrained case and demonstrate how the unconstrained
case can be analyzed under weaker assumptions than the state-of-the-art.
Numerical results confirm our theoretical developments.
