Off-Policy Policy Gradient Algorithms by Constraining the State
  Distribution Shift
  Off-policy deep reinforcement learning (RL) algorithms are incapable of
learning solely from batch offline data without online interactions with the
environment, due to the phenomenon known as \textit{extrapolation error}. This
is often due to past data available in the replay buffer that may be quite
different from the data distribution under the current policy. We argue that
most off-policy learning methods fundamentally suffer from a \textit{state
distribution shift} due to the mismatch between the state visitation
distribution of the data collected by the behavior and target policies. This
data distribution shift between current and past samples can significantly
impact the performance of most modern off-policy based policy optimization
algorithms. In this work, we first do a systematic analysis of state
distribution mismatch in off-policy learning, and then develop a novel
off-policy policy optimization method to constraint the state distribution
shift. To do this, we first estimate the state distribution based on features
of the state, using a density estimator and then develop a novel constrained
off-policy gradient objective that minimizes the state distribution shift. Our
experimental results on continuous control tasks show that minimizing this
distribution mismatch can significantly improve performance in most popular
practical off-policy policy gradient algorithms.
