A Hybrid Stochastic Optimization Framework for Stochastic Composite
  Nonconvex Optimization
  We introduce a new approach to develop stochastic optimization algorithms for
a class of stochastic composite and possibly nonconvex optimization problems.
The main idea is to combine two stochastic estimators to create a new hybrid
one. We first introduce our hybrid estimator and then investigate its
fundamental properties to form a foundational theory for algorithmic
development. Next, we apply our theory to develop several variants of
stochastic gradient methods to solve both expectation and finite-sum composite
optimization problems. Our first algorithm can be viewed as a variant of
proximal stochastic gradient methods with a single-loop, but can achieve
$\mathcal{O}(\sigma^3\varepsilon^{-1} + \sigma \varepsilon^{-3})$-oracle
complexity bound, matching the best-known ones from state-of-the-art
double-loop algorithms in the literature, where $\sigma > 0$ is the variance
and $\varepsilon$ is a desired accuracy. Then, we consider two different
variants of our method: adaptive step-size and restarting schemes that have
similar theoretical guarantees as in our first algorithm. We also study two
mini-batch variants of the proposed methods. In all cases, we achieve the
best-known complexity bounds under standard assumptions. We test our methods on
several numerical examples with real datasets and compare them with
state-of-the-arts. Our numerical experiments show that the new methods are
comparable and, in many cases, outperform their competitors.
