Sparse Optimization on General Atomic Sets: Greedy and Forward-Backward
  Algorithms
  We consider the problem of sparse atomic optimization, where the notion of
"sparsity" is generalized to meaning some linear combination of few atoms. The
definition of atomic set is very broad; popular examples include the standard
basis, low-rank matrices, overcomplete dictionaries, permutation matrices,
orthogonal matrices, etc. The model of sparse atomic optimization therefore
includes problems coming from many fields, including statistics, signal
processing, machine learning, computer vision and so on. Specifically, we
consider the problem of maximizing a restricted strongly convex (or concave),
smooth function restricted to a sparse linear combination of atoms. We extend
recent work that establish linear convergence rates of greedy algorithms on
restricted strongly concave, smooth functions on sparse vectors to the realm of
general atomic sets, where the convergence rate involves a novel quantity: the
"sparse atomic condition number". This leads to the strongest known
multiplicative approximation guarantees for various flavors of greedy
algorithms for sparse atomic optimization; in particular, we show that in many
settings of interest the greedy algorithm can attain strong approximation
guarantees while maintaining sparsity. Furthermore, we introduce a scheme for
forward-backward algorithms that achieves the same approximation guarantees.
Secondly, we define an alternate notion of weak submodularity, which we show is
tightly related to the more familiar version that has been used to prove
earlier linear convergence rates. We prove analogous multiplicative
approximation guarantees using this alternate weak submodularity, and establish
its distinct identity and applications.
