Statistical Query Algorithms for Mean Vector Estimation and Stochastic
  Convex Optimization
  Stochastic convex optimization, where the objective is the expectation of a
random convex function, is an important and widely used method with numerous
applications in machine learning, statistics, operations research and other
areas. We study the complexity of stochastic convex optimization given only
statistical query (SQ) access to the objective function. We show that
well-known and popular first-order iterative methods can be implemented using
only statistical queries. For many cases of interest we derive nearly matching
upper and lower bounds on the estimation (sample) complexity including linear
optimization in the most general setting. We then present several consequences
for machine learning, differential privacy and proving concrete lower bounds on
the power of convex optimization based methods.
  The key ingredient of our work is SQ algorithms and lower bounds for
estimating the mean vector of a distribution over vectors supported on a convex
body in $\mathbb{R}^d$. This natural problem has not been previously studied
and we show that our solutions can be used to get substantially improved SQ
versions of Perceptron and other online algorithms for learning halfspaces.
