Efficient Evaluation of Natural Stochastic Policies in Offline
  Reinforcement Learning
  We study the efficient off-policy evaluation of natural stochastic policies,
which are defined in terms of deviations from the behavior policy. This is a
departure from the literature on off-policy evaluation where most work consider
the evaluation of explicitly specified policies. Crucially, offline
reinforcement learning with natural stochastic policies can help alleviate
issues of weak overlap, lead to policies that build upon current practice, and
improve policies' implementability in practice. Compared with the classic case
of a pre-specified evaluation policy, when evaluating natural stochastic
policies, the efficiency bound, which measures the best-achievable estimation
error, is inflated since the evaluation policy itself is unknown. In this
paper, we derive the efficiency bounds of two major types of natural stochastic
policies: tilting policies and modified treatment policies. We then propose
efficient nonparametric estimators that attain the efficiency bounds under very
lax conditions. These also enjoy a (partial) double robustness property.
