Input Warping for Bayesian Optimization of Non-stationary Functions
  Bayesian optimization has proven to be a highly effective methodology for the
global optimization of unknown, expensive and multimodal functions. The ability
to accurately model distributions over functions is critical to the
effectiveness of Bayesian optimization. Although Gaussian processes provide a
flexible prior over functions which can be queried efficiently, there are
various classes of functions that remain difficult to model. One of the most
frequently occurring of these is the class of non-stationary functions. The
optimization of the hyperparameters of machine learning algorithms is a problem
domain in which parameters are often manually transformed a priori, for example
by optimizing in "log-space," to mitigate the effects of spatially-varying
length scale. We develop a methodology for automatically learning a wide family
of bijective transformations or warpings of the input space using the Beta
cumulative distribution function. We further extend the warping framework to
multi-task Bayesian optimization so that multiple tasks can be warped into a
jointly stationary space. On a set of challenging benchmark optimization tasks,
we observe that the inclusion of warping greatly improves on the
state-of-the-art, producing better results faster and more reliably.
