LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark
  Suite for Lasso
  While Weighted Lasso sparse regression has appealing statistical guarantees
that would entail a major real-world impact in finance, genomics, and brain
imaging applications, it is typically scarcely adopted due to its complex
high-dimensional space composed by thousands of hyperparameters. On the other
hand, the latest progress with high-dimensional hyperparameter optimization
(HD-HPO) methods for black-box functions demonstrates that high-dimensional
applications can indeed be efficiently optimized. Despite this initial success,
HD-HPO approaches are mostly applied to synthetic problems with a moderate
number of dimensions, which limits its impact in scientific and engineering
applications. We propose LassoBench, the first benchmark suite tailored for
Weighted Lasso regression. LassoBench consists of benchmarks for both
well-controlled synthetic setups (number of samples, noise level, ambient and
effective dimensionalities, and multiple fidelities) and real-world datasets,
which enables the use of many flavors of HPO algorithms to be studied and
extended to the high-dimensional Lasso setting. We evaluate 6 state-of-the-art
HPO methods and 3 Lasso baselines, and demonstrate that Bayesian optimization
and evolutionary strategies can improve over the methods commonly used for
sparse regression while highlighting limitations of these frameworks in very
high-dimensional and noisy settings.
