Bayesian Optimization for Policy Search via Online-Offline
  Experimentation
  Online field experiments are the gold-standard way of evaluating changes to
real-world interactive machine learning systems. Yet our ability to explore
complex, multi-dimensional policy spaces - such as those found in
recommendation and ranking problems - is often constrained by the limited
number of experiments that can be run simultaneously. To alleviate these
constraints, we augment online experiments with an offline simulator and apply
multi-task Bayesian optimization to tune live machine learning systems. We
describe practical issues that arise in these types of applications, including
biases that arise from using a simulator and assumptions for the multi-task
kernel. We measure empirical learning curves which show substantial gains from
including data from biased offline experiments, and show how these learning
curves are consistent with theoretical results for multi-task Gaussian process
generalization. We find that improved kernel inference is a significant driver
of multi-task generalization. Finally, we show several examples of Bayesian
optimization efficiently tuning a live machine learning system by combining
offline and online experiments.
