Structures of Spurious Local Minima in $k$-means
  $k$-means clustering is a fundamental problem in unsupervised learning. The
problem concerns finding a partition of the data points into $k$ clusters such
that the within-cluster variation is minimized. Despite its importance and wide
applicability, a theoretical understanding of the $k$-means problem has not
been completely satisfactory. Existing algorithms with theoretical performance
guarantees often rely on sophisticated (sometimes artificial) algorithmic
techniques and restricted assumptions on the data. The main challenge lies in
the non-convex nature of the problem; in particular, there exist additional
local solutions other than the global optimum. Moreover, the simplest and most
popular algorithm for $k$-means, namely Lloyd's algorithm, generally converges
to such spurious local solutions both in theory and in practice.
  In this paper, we approach the $k$-means problem from a new perspective, by
investigating the structures of these spurious local solutions under a
probabilistic generative model with $k$ ground truth clusters. As soon as
$k=3$, spurious local minima provably exist, even for well-separated and
balanced clusters. One such local minimum puts two centers at one true cluster,
and the third center in the middle of the other two true clusters. For general
$k$, one local minimum puts multiple centers at a true cluster, and one center
in the middle of multiple true clusters. Perhaps surprisingly, we prove that
this is essentially the only type of spurious local minima under a separation
condition. Our results pertain to the $k$-means formulation for mixtures of
Gaussians or bounded distributions. Our theoretical results corroborate
existing empirical observations and provide justification for several improved
algorithms for $k$-means clustering.
