Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate
  models
  Bayesian Optimization is a popular tool for tuning algorithms in automatic
machine learning (AutoML) systems. Current state-of-the-art methods leverage
Random Forests or Gaussian processes to build a surrogate model that predicts
algorithm performance given a certain set of hyperparameter settings. In this
paper, we propose a new surrogate model based on gradient boosting, where we
use quantile regression to provide optimistic estimates of the performance of
an unobserved hyperparameter setting, and combine this with a distance metric
between unobserved and observed hyperparameter settings to help regulate
exploration. We demonstrate empirically that the new method is able to
outperform some state-of-the art techniques across a reasonable sized set of
classification problems.
