Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
  Infinite horizon off-policy policy evaluation is a highly challenging task
due to the excessively large variance of typical importance sampling (IS)
estimators. Recently, Liu et al. (2018a) proposed an approach that
significantly reduces the variance of infinite-horizon off-policy evaluation by
estimating the stationary density ratio, but at the cost of introducing
potentially high biases due to the error in density ratio estimation. In this
paper, we develop a bias-reduced augmentation of their method, which can take
advantage of a learned value function to obtain higher accuracy. Our method is
doubly robust in that the bias vanishes when either the density ratio or the
value function estimation is perfect. In general, when either of them is
accurate, the bias can also be reduced. Both theoretical and empirical results
show that our method yields significant advantages over previous methods.
