Improved Learning Rates for Stochastic Optimization: Two Theoretical
  Viewpoints
  Generalization performance of stochastic optimization stands a central place
in learning theory. In this paper, we investigate the excess risk performance
and towards improved learning rates for two popular approaches of stochastic
optimization: empirical risk minimization (ERM) and stochastic gradient descent
(SGD). Although there exists plentiful generalization analysis of ERM and SGD
for supervised learning, current theoretical understandings of ERM and SGD
either have stronger assumptions in convex learning, e.g., strong convexity, or
show slow rates and less studied in nonconvex learning. Motivated by these
problems, we aim to provide improved rates under milder assumptions in convex
learning and derive faster rates in nonconvex learning. It is notable that our
analysis span two popular theoretical viewpoints: \emph{stability} and
\emph{uniform convergence}. Specifically, in stability regime, we present high
probability learning rates of order $\mathcal{O} (1/n)$ w.r.t. the sample size
$n$ for ERM and SGD with milder assumptions in convex learning and similar high
probability rates of order $\mathcal{O} (1/n)$ in nonconvex learning, rather
than in expectation. Furthermore, this type of learning rate is improved to
faster order $\mathcal{O} (1/n^2)$ in uniform convergence regime. To our best
knowledge, for ERM and SGD, the learning rates presented in this paper are all
state-of-the-art.
