Robustness in sparse linear models: relative efficiency based on robust
  approximate message passing
  Understanding efficiency in high dimensional linear models is a longstanding
problem of interest. Classical work with smaller dimensional problems dating
back to Huber and Bickel has illustrated the benefits of efficient loss
functions. When the number of parameters $p$ is of the same order as the sample
size $n$, $p \approx n$, an efficiency pattern different from the one of Huber
was recently established. In this work, we consider the effects of model
selection on the estimation efficiency of penalized methods. In particular, we
explore whether sparsity, results in new efficiency patterns when $p > n$. In
the interest of deriving the asymptotic mean squared error for regularized
M-estimators, we use the powerful framework of approximate message passing. We
propose a novel, robust and sparse approximate message passing algorithm
(RAMP), that is adaptive to the error distribution. Our algorithm includes many
non-quadratic and non-differentiable loss functions. We derive its asymptotic
mean squared error and show its convergence, while allowing $p, n, s \to
\infty$, with $n/p \in (0,1)$ and $n/s \in (1,\infty)$. We identify new
patterns of relative efficiency regarding a number of penalized $M$ estimators,
when $p$ is much larger than $n$. We show that the classical information bound
is no longer reachable, even for light--tailed error distributions. We show
that the penalized least absolute deviation estimator dominates the penalized
least square estimator, in cases of heavy--tailed distributions. We observe
this pattern for all choices of the number of non-zero parameters $s$, both $s
\leq n$ and $s \approx n$. In non-penalized problems where $s =p \approx n$,
the opposite regime holds. Therefore, we discover that the presence of model
selection significantly changes the efficiency patterns.
