Off-Policy Evaluation and Learning for External Validity under a
  Covariate Shift
  We consider evaluating and training a new policy for the evaluation data by
using the historical data obtained from a different policy. The goal of
off-policy evaluation (OPE) is to estimate the expected reward of a new policy
over the evaluation data, and that of off-policy learning (OPL) is to find a
new policy that maximizes the expected reward over the evaluation data.
Although the standard OPE and OPL assume the same distribution of covariate
between the historical and evaluation data, a covariate shift often exists,
i.e., the distribution of the covariate of the historical data is different
from that of the evaluation data. In this paper, we derive the efficiency bound
of OPE under a covariate shift. Then, we propose doubly robust and efficient
estimators for OPE and OPL under a covariate shift by using a nonparametric
estimator of the density ratio between the historical and evaluation data
distributions. We also discuss other possible estimators and compare their
theoretical properties. Finally, we confirm the effectiveness of the proposed
estimators through experiments.
