Black-box Off-policy Estimation for Infinite-Horizon Reinforcement
  Learning
  Off-policy estimation for long-horizon problems is important in many
real-life applications such as healthcare and robotics, where high-fidelity
simulators may not be available and on-policy evaluation is expensive or
impossible. Recently, \cite{liu18breaking} proposed an approach that avoids the
\emph{curse of horizon} suffered by typical importance-sampling-based methods.
While showing promising results, this approach is limited in practice as it
requires data be drawn from the \emph{stationary distribution} of a
\emph{known} behavior policy. In this work, we propose a novel approach that
eliminates such limitations. In particular, we formulate the problem as solving
for the fixed point of a certain operator. Using tools from Reproducing Kernel
Hilbert Spaces (RKHSs), we develop a new estimator that computes importance
ratios of stationary distributions, without knowledge of how the off-policy
data are collected. We analyze its asymptotic consistency and finite-sample
generalization. Experiments on benchmarks verify the effectiveness of our
approach.
