Analysis of the Optimization Landscapes for Overcomplete Representation
  Learning
  We study nonconvex optimization landscapes for learning overcomplete
representations, including learning (i) sparsely used overcomplete dictionaries
and (ii) convolutional dictionaries, where these unsupervised learning problems
find many applications in high-dimensional data analysis. Despite the empirical
success of simple nonconvex algorithms, theoretical justifications of why these
methods work so well are far from satisfactory. In this work, we show these
problems can be formulated as $\ell^4$-norm optimization problems with
spherical constraint, and study the geometric properties of their nonconvex
optimization landscapes. For both problems, we show the nonconvex objectives
have benign (global) geometric structures, in the sense that every local
minimizer is close to one of the target solutions and every saddle point
exhibits negative curvature. This discovery enables the development of
guaranteed global optimization methods using simple initializations. For both
problems, we show the nonconvex objectives have benign geometric structures --
every local minimizer is close to one of the target solutions and every saddle
point exhibits negative curvature -- either in the entire space or within a
sufficiently large region. This discovery ensures local search algorithms (such
as Riemannian gradient descent) with simple initializations approximately find
the target solutions. Finally, numerical experiments justify our theoretical
discoveries.
