Optimally tackling covariate shift in RKHS-based nonparametric regression
We study the covariate shift problem in the context of nonparametric
regression over a reproducing kernel Hilbert space (RKHS). We focus on two
natural families of covariate shift problems defined using the likelihood
ratios between the source and target distributions. When the likelihood ratios
are uniformly bounded, we prove that the kernel ridge regression (KRR)
estimator with a carefully chosen regularization parameter is minimax
rate-optimal (up to a log factor) for a large family of RKHSs with regular
kernel eigenvalues. Interestingly, KRR does not require full knowledge of the
likelihood ratio apart from an upper bound on it. In striking contrast to the
standard statistical setting without covariate shift, we also demonstrate that
a na\"\i ve estimator, which minimizes the empirical risk over the function
class, is strictly suboptimal under covariate shift as compared to KRR. We then
address the larger class of covariate shift problems where likelihood ratio is
possibly unbounded yet has a finite second moment. Here, we show via careful
simulations that KRR fails to attain the optimal rate. Instead, we propose a
reweighted KRR estimator that weights samples based on a careful truncation of
the likelihood ratios. Again, we are able to show that this estimator is
minimax optimal, up to logarithmic factors.