Confidence Interval for Off-Policy Evaluation from Dependent Samples via
  Bandit Algorithm: Approach from Standardized Martingales
  This study addresses the problem of off-policy evaluation (OPE) from
dependent samples obtained via the bandit algorithm. The goal of OPE is to
evaluate a new policy using historical data obtained from behavior policies
generated by the bandit algorithm. Because the bandit algorithm updates the
policy based on past observations, the samples are not independent and
identically distributed (i.i.d.). However, several existing methods for OPE do
not take this issue into account and are based on the assumption that samples
are i.i.d. In this study, we address this problem by constructing an estimator
from a standardized martingale difference sequence. To standardize the
sequence, we consider using evaluation data or sample splitting with a two-step
estimation. This technique produces an estimator with asymptotic normality
without restricting a class of behavior policies. In an experiment, the
proposed estimator performs better than existing methods, which assume that the
behavior policy converges to a time-invariant policy.
