Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse
  Optimization and Noisy Matrix Decomposition
  We propose an efficient ADMM method with guarantees for high-dimensional
problems. We provide explicit bounds for the sparse optimization problem and
the noisy matrix decomposition problem. For sparse optimization, we establish
that the modified ADMM method has an optimal convergence rate of
$\mathcal{O}(s\log d/T)$, where $s$ is the sparsity level, $d$ is the data
dimension and $T$ is the number of steps. This matches with the minimax lower
bounds for sparse estimation. For matrix decomposition into sparse and low rank
components, we provide the first guarantees for any online method, and prove a
convergence rate of $\tilde{\mathcal{O}}((s+r)\beta^2(p) /T) +
\mathcal{O}(1/p)$ for a $p\times p$ matrix, where $s$ is the sparsity level,
$r$ is the rank and $\Theta(\sqrt{p})\leq \beta(p)\leq \Theta(p)$. Our
guarantees match the minimax lower bound with respect to $s,r$ and $T$. In
addition, we match the minimax lower bound with respect to the matrix dimension
$p$, i.e. $\beta(p)=\Theta(\sqrt{p})$, for many important statistical models
including the independent noise model, the linear Bayesian network and the
latent Gaussian graphical model under some conditions. Our ADMM method is based
on epoch-based annealing and consists of inexpensive steps which involve
projections on to simple norm balls. Experiments show that for both sparse
optimization and matrix decomposition problems, our algorithm outperforms the
state-of-the-art methods. In particular, we reach higher accuracy with same
time complexity.
