Improving the Efficiency of Off-Policy Reinforcement Learning by
  Accounting for Past Decisions
  Off-policy learning from multistep returns is crucial for sample-efficient
reinforcement learning, particularly in the experience replay setting now
commonly used with deep neural networks. Classically, off-policy estimation
bias is corrected in a per-decision manner: past temporal-difference errors are
re-weighted by the instantaneous Importance Sampling (IS) ratio (via
eligibility traces) after each action. Many important off-policy algorithms
such as Tree Backup and Retrace rely on this mechanism along with differing
protocols for truncating ("cutting") the ratios ("traces") to counteract the
excessive variance of the IS estimator. Unfortunately, cutting traces on a
per-decision basis is not necessarily efficient; once a trace has been cut
according to local information, the effect cannot be reversed later,
potentially resulting in the premature truncation of estimated returns and
slower learning. In the interest of motivating efficient off-policy algorithms,
we propose a multistep operator that permits arbitrary past-dependent traces.
We prove that our operator is convergent for policy evaluation, and for optimal
control when targeting greedy-in-the-limit policies. Our theorems establish the
first convergence guarantees for many existing algorithms including Truncated
IS, Non-Markov Retrace, and history-dependent TD($\lambda$). Our theoretical
results also provide guidance for the development of new algorithms that
jointly consider multiple past decisions for better credit assignment and
faster learning.
