Sum-of-Squares Lower Bounds for Sparse PCA
  This paper establishes a statistical versus computational trade-off for
solving a basic high-dimensional machine learning problem via a basic convex
relaxation method. Specifically, we consider the {\em Sparse Principal
Component Analysis} (Sparse PCA) problem, and the family of {\em
Sum-of-Squares} (SoS, aka Lasserre/Parillo) convex relaxations. It was well
known that in large dimension $p$, a planted $k$-sparse unit vector can be {\em
in principle} detected using only $n \approx k\log p$ (Gaussian or Bernoulli)
samples, but all {\em efficient} (polynomial time) algorithms known require $n
\approx k^2$ samples. It was also known that this quadratic gap cannot be
improved by the the most basic {\em semi-definite} (SDP, aka spectral)
relaxation, equivalent to a degree-2 SoS algorithms. Here we prove that also
degree-4 SoS algorithms cannot improve this quadratic gap. This average-case
lower bound adds to the small collection of hardness results in machine
learning for this powerful family of convex relaxation algorithms. Moreover,
our design of moments (or "pseudo-expectations") for this lower bound is quite
different than previous lower bounds. Establishing lower bounds for higher
degree SoS algorithms for remains a challenging problem.
