MixedGrad: An O(1/T) Convergence Rate Algorithm for Stochastic Smooth
  Optimization
  It is well known that the optimal convergence rate for stochastic
optimization of smooth functions is $O(1/\sqrt{T})$, which is same as
stochastic optimization of Lipschitz continuous convex functions. This is in
contrast to optimizing smooth functions using full gradients, which yields a
convergence rate of $O(1/T^2)$. In this work, we consider a new setup for
optimizing smooth functions, termed as {\bf Mixed Optimization}, which allows
to access both a stochastic oracle and a full gradient oracle. Our goal is to
significantly improve the convergence rate of stochastic optimization of smooth
functions by having an additional small number of accesses to the full gradient
oracle. We show that, with an $O(\ln T)$ calls to the full gradient oracle and
an $O(T)$ calls to the stochastic oracle, the proposed mixed optimization
algorithm is able to achieve an optimization error of $O(1/T)$.
