High Dimensional Robust $M$-Estimation: Arbitrary Corruption and Heavy
  Tails
  We consider the problem of sparsity-constrained $M$-estimation when both
explanatory and response variables have heavy tails (bounded 4-th moments), or
a fraction of arbitrary corruptions. We focus on the $k$-sparse,
high-dimensional regime where the number of variables $d$ and the sample size
$n$ are related through $n \sim k \log d$. We define a natural condition we
call the Robust Descent Condition (RDC), and show that if a gradient estimator
satisfies the RDC, then Robust Hard Thresholding (IHT using this gradient
estimator), is guaranteed to obtain good statistical rates. The contribution of
this paper is in showing that this RDC is a flexible enough concept to recover
known results, and obtain new robustness results. Specifically, new results
include: (a) For $k$-sparse high-dimensional linear- and logistic-regression
with heavy tail (bounded 4-th moment) explanatory and response variables, a
linear-time-computable median-of-means gradient estimator satisfies the RDC,
and hence Robust Hard Thresholding is minimax optimal; (b) When instead of
heavy tails we have $O(1/\sqrt{k}\log(nd))$-fraction of arbitrary corruptions
in explanatory and response variables, a near linear-time computable trimmed
gradient estimator satisfies the RDC, and hence Robust Hard Thresholding is
minimax optimal. We demonstrate the effectiveness of our approach in sparse
linear, logistic regression, and sparse precision matrix estimation on
synthetic and real-world US equities data.
