Query Complexity of Least Absolute Deviation Regression via Robust
  Uniform Convergence
  Consider a regression problem where the learner is given a large collection
of $d$-dimensional data points, but can only query a small subset of the
real-valued labels. How many queries are needed to obtain a $1+\epsilon$
relative error approximation of the optimum? While this problem has been
extensively studied for least squares regression, little is known for other
losses. An important example is least absolute deviation regression ($\ell_1$
regression) which enjoys superior robustness to outliers compared to least
squares. We develop a new framework for analyzing importance sampling methods
in regression problems, which enables us to show that the query complexity of
least absolute deviation regression is $\Theta(d/\epsilon^2)$ up to logarithmic
factors. We further extend our techniques to show the first bounds on the query
complexity for any $\ell_p$ loss with $p\in(1,2)$. As a key novelty in our
analysis, we introduce the notion of robust uniform convergence, which is a new
approximation guarantee for the empirical loss. While it is inspired by uniform
convergence in statistical learning, our approach additionally incorporates a
correction term to avoid unnecessary variance due to outliers. This can be
viewed as a new connection between statistical learning theory and variance
reduction techniques in stochastic optimization, which should be of independent
interest.
