Off-Policy Evaluation via Adaptive Weighting with Data from Contextual
  Bandits
  It has become increasingly common for data to be collected adaptively, for
example using contextual bandits. Historical data of this type can be used to
evaluate other treatment assignment policies to guide future innovation or
experiments. However, policy evaluation is challenging if the target policy
differs from the one used to collect data, and popular estimators, including
doubly robust (DR) estimators, can be plagued by bias, excessive variance, or
both. In particular, when the pattern of treatment assignment in the collected
data looks little like the pattern generated by the policy to be evaluated, the
importance weights used in DR estimators explode, leading to excessive
variance.
  In this paper, we improve the DR estimator by adaptively weighting
observations to control its variance. We show that a t-statistic based on our
improved estimator is asymptotically normal under certain conditions, allowing
us to form confidence intervals and test hypotheses. Using synthetic data and
public benchmarks, we provide empirical evidence for our estimator's improved
accuracy and inferential properties relative to existing alternatives.
