Convergence and Complexity of Stochastic Subgradient Methods with
  Dependent Data for Nonconvex Optimization
  We show that under a general dependent data sampling scheme, the classical
stochastic projected and proximal subgradient methods for weakly convex
functions have worst-case rate of convergence $\tilde{O}(n^{-1/4})$ and
complexity $\tilde{O}(\varepsilon^{-4})$ for achieving an $\varepsilon$-near
stationary point in terms of the norm of the gradient of Moreau envelope. While
classical convergence guarantee requires i.i.d. data sampling from the target
distribution, we only require a mild mixing condition of the conditional
distribution, which holds for a wide class of Markov chain sampling algorithms.
This improves the existing complexity for the specific case of constrained
smooth nonconvex optimization with dependent data from
$\tilde{O}(\varepsilon^{-8})$ to $\tilde{O}(\varepsilon^{-4})$ with a
significantly simpler analysis. We illustrate the generality of our approach by
deriving convergence results with dependent data for adaptive stochastic
subgradient algorithm AdaGrad and stochastic subgradient algorithm with heavy
ball momentum. As an application, we obtain first online nonnegative matrix
factorization algorithms for dependent data based on stochastic projected
gradient methods with adaptive step sizes with optimal rate of convergence
guarantee.
