Understanding and Mitigating the Limitations of Prioritized Experience
  Replay
  Prioritized Experience Replay (ER) has been empirically shown to improve
sample efficiency across many domains and attracted great attention; however,
there is little theoretical understanding of why such prioritized sampling
helps and its limitations. In this work, we take a deep look at the prioritized
ER. In a supervised learning setting, we show the equivalence between the
error-based prioritized sampling method for mean squared error and uniform
sampling for cubic power loss. We then provide theoretical insight into why it
improves convergence rate upon uniform sampling during early learning. Based on
the insight, we further point out two limitations of the prioritized ER method:
1) outdated priorities and 2) insufficient coverage of the sample space. To
mitigate the limitations, we propose our model-based stochastic gradient
Langevin dynamics sampling method. We show that our method does provide states
distributed close to an ideal prioritized sampling distribution estimated by
the brute-force method, which does not suffer from the two limitations. We
conduct experiments on both discrete and continuous control problems to show
our approach's efficacy and examine the practical implication of our method in
an autonomous driving application.
