Importance Resampling for Off-policy Prediction
  Importance sampling (IS) is a common reweighting strategy for off-policy
prediction in reinforcement learning. While it is consistent and unbiased, it
can result in high variance updates to the weights for the value function. In
this work, we explore a resampling strategy as an alternative to reweighting.
We propose Importance Resampling (IR) for off-policy prediction, which
resamples experience from a replay buffer and applies standard on-policy
updates. The approach avoids using importance sampling ratios in the update,
instead correcting the distribution before the update. We characterize the bias
and consistency of IR, particularly compared to Weighted IS (WIS). We
demonstrate in several microworlds that IR has improved sample efficiency and
lower variance updates, as compared to IS and several variance-reduced IS
strategies, including variants of WIS and V-trace which clips IS ratios. We
also provide a demonstration showing IR improves over IS for learning a value
function from images in a racing car simulator.
