Efficient Hyperparameter Optimization under Multi-Source Covariate Shift
  A typical assumption in supervised machine learning is that the train
(source) and test (target) datasets follow completely the same distribution.
This assumption is, however, often violated in uncertain real-world
applications, which motivates the study of learning under covariate shift. In
this setting, the naive use of adaptive hyperparameter optimization methods
such as Bayesian optimization does not work as desired since it does not
address the distributional shift among different datasets. In this work, we
consider a novel hyperparameter optimization problem under the multi-source
covariate shift whose goal is to find the optimal hyperparameters for a target
task of interest using only unlabeled data in a target task and labeled data in
multiple source tasks. To conduct efficient hyperparameter optimization for the
target task, it is essential to estimate the target objective using only the
available information. To this end, we construct the variance reduced estimator
that unbiasedly approximates the target objective with a desirable variance
property. Building on the proposed estimator, we provide a general and
tractable hyperparameter optimization procedure, which works preferably in our
setting with a no-regret guarantee. The experiments demonstrate that the
proposed framework broadens the applications of automated hyperparameter
optimization.
