HyperJump: Accelerating HyperBand via Risk Modelling
  In the literature on hyper-parameter tuning, a number of recent solutions
rely on low-fidelity observations (e.g., training with sub-sampled datasets or
for short periods of time) to extrapolate good configurations to use when
performing full training. Among these, HyperBand is arguably one of the most
popular solutions, due to its efficiency and theoretically provable robustness.
In this work, we introduce HyperJump, a new approach that builds on HyperBand's
robust search strategy and complements it with novel model-based risk analysis
techniques that accelerate the search by \textit{jumping} the evaluation of low
risk configurations, i.e., configurations that are likely to be discarded by
HyperBand. We evaluate HyperJump on a suite of hyper-parameter optimization
problems and show that it provides over one-order of magnitude speed-ups, both
in sequential and parallel deployments, on a variety of deep learning,
kernel-based learning, and neural architectural search problems when compared
to HyperBand and to several state-of-the-art optimizers.
