Experience Replay with Likelihood-free Importance Weights
  The use of past experiences to accelerate temporal difference (TD) learning
of value functions, or experience replay, is a key component in deep
reinforcement learning. Prioritization or reweighting of important experiences
has shown to improve performance of TD learning algorithms.In this work, we
propose to reweight experiences based on their likelihood under the stationary
distribution of the current policy. Using the corresponding reweighted TD
objective, we implicitly encourage small approximation errors on the value
function over frequently encountered states. We use a likelihood-free density
ratio estimator over the replay buffer to assign the prioritization weights. We
apply the proposed approach empirically on two competitive methods, Soft Actor
Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over
a suite of OpenAI gym tasks and achieve superior sample complexity compared to
other baseline approaches.
