DualDICE: Behavior-Agnostic Estimation of Discounted Stationary
  Distribution Corrections
  In many real-world reinforcement learning applications, access to the
environment is limited to a fixed dataset, instead of direct (online)
interaction with the environment. When using this data for either evaluation or
training of a new policy, accurate estimates of discounted stationary
distribution ratios -- correction terms which quantify the likelihood that the
new policy will experience a certain state-action pair normalized by the
probability with which the state-action pair appears in the dataset -- can
improve accuracy and performance. In this work, we propose an algorithm,
DualDICE, for estimating these quantities. In contrast to previous approaches,
our algorithm is agnostic to knowledge of the behavior policy (or policies)
used to generate the dataset. Furthermore, it eschews any direct use of
importance weights, thus avoiding potential optimization instabilities endemic
of previous methods. In addition to providing theoretical guarantees, we
present an empirical study of our algorithm applied to off-policy policy
evaluation and find that our algorithm significantly improves accuracy compared
to existing techniques.
