Modulating Surrogates for Bayesian Optimization
  Bayesian optimization (BO) methods often rely on the assumption that the
objective function is well-behaved, but in practice, this is seldom true for
real-world objectives even if noise-free observations can be collected. Common
approaches, which try to model the objective as precisely as possible, often
fail to make progress by spending too many evaluations modeling irrelevant
details. We address this issue by proposing surrogate models that focus on the
well-behaved structure in the objective function, which is informative for
search, while ignoring detrimental structure that is challenging to model from
few observations. First, we demonstrate that surrogate models with appropriate
noise distributions can absorb challenging structures in the objective function
by treating them as irreducible uncertainty. Secondly, we show that a latent
Gaussian process is an excellent surrogate for this purpose, comparing with
Gaussian processes with standard noise distributions. We perform numerous
experiments on a range of BO benchmarks and find that our approach improves
reliability and performance when faced with challenging objective functions.
