Calibration Improves Bayesian Optimization
  Bayesian optimization is a procedure that allows obtaining the global optimum
of black-box functions and that is useful in applications such as
hyper-parameter optimization. Uncertainty estimates over the shape of the
objective function are instrumental in guiding the optimization process.
However, these estimates can be inaccurate if the objective function violates
assumptions made within the underlying model (e.g., Gaussianity). We propose a
simple algorithm to calibrate the uncertainty of posterior distributions over
the objective function as part of the Bayesian optimization process. We show
that by improving the uncertainty estimates of the posterior distribution with
calibration, Bayesian optimization makes better decisions and arrives at the
global optimum in fewer steps. We show that this technique improves the
performance of Bayesian optimization on standard benchmark functions and
hyperparameter optimization tasks.
