Confidence Intervals for Policy Evaluation in Adaptive Experiments
  Adaptive experiment designs can dramatically improve statistical efficiency
in randomized trials, but they also complicate statistical inference. For
example, it is now well known that the sample mean is biased in adaptive
trials. Inferential challenges are exacerbated when our parameter of interest
differs from the parameter the trial was designed to target, such as when we
are interested in estimating the value of a sub-optimal treatment after running
a trial to determine the optimal treatment using a stochastic bandit design. In
this context, typical estimators that use inverse propensity weighting to
eliminate sampling bias can be problematic: their distributions become skewed
and heavy-tailed as the propensity scores decay to zero. In this paper, we
present a class of estimators that overcome these issues. Our approach is to
adaptively reweight the terms of an augmented inverse propensity weighting
estimator to control the contribution of each term to the estimator's variance.
This adaptive weighting scheme prevents estimates from becoming heavy-tailed,
ensuring asymptotically correct coverage. It also reduces variance, allowing us
to test hypotheses with greater power - especially hypotheses that were not
targeted by the experimental design. We validate the accuracy of the resulting
estimates and their confidence intervals in numerical experiments and show our
methods compare favorably to existing alternatives in terms of RMSE and
coverage.
