Searching in the Forest for Local Bayesian Optimization
  Because of its sample efficiency, Bayesian optimization (BO) has become a
popular approach dealing with expensive black-box optimization problems, such
as hyperparameter optimization (HPO). Recent empirical experiments showed that
the loss landscapes of HPO problems tend to be more benign than previously
assumed, i.e. in the best case uni-modal and convex, such that a BO framework
could be more efficient if it can focus on those promising local regions. In
this paper, we propose BOinG, a two-stage approach that is tailored toward
mid-sized configuration spaces, as one encounters in many HPO problems. In the
first stage, we build a scalable global surrogate model with a random forest to
describe the overall landscape structure. Further, we choose a promising
subregion via a bottom-up approach on the upper-level tree structure. In the
second stage, a local model in this subregion is utilized to suggest the point
to be evaluated next. Empirical experiments show that BOinG is able to exploit
the structure of typical HPO problems and performs particularly well on
mid-sized problems from synthetic functions and HPO.
