Doubly Robust Interval Estimation for Optimal Policy Evaluation in
  Online Learning
  Evaluating the performance of an ongoing policy plays a vital role in many
areas such as medicine and economics, to provide crucial instruction on the
early-stop of the online experiment and timely feedback from the environment.
Policy evaluation in online learning thus attracts increasing attention by
inferring the mean outcome of the optimal policy (i.e., the value) in
real-time. Yet, such a problem is particularly challenging due to the dependent
data generated in the online environment, the unknown optimal policy, and the
complex exploration and exploitation trade-off in the adaptive experiment. In
this paper, we aim to overcome these difficulties in policy evaluation for
online learning. We explicitly derive the probability of exploration that
quantifies the probability of exploring the non-optimal actions under commonly
used bandit algorithms. We use this probability to conduct valid inference on
the online conditional mean estimator under each action and develop the doubly
robust interval estimation (DREAM) method to infer the value under the
estimated optimal policy in online learning. The proposed value estimator
provides double protection on the consistency and is asymptotically normal with
a Wald-type confidence interval provided. Extensive simulations and real data
applications are conducted to demonstrate the empirical validity of the
proposed DREAM method.
