Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
  This paper proposes a method for prioritizing the replay experience referred
to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight
Experience Replay (HER) that generates hindsight goals based on uniform
sampling. HGR samples with higher probability on the states visited in an
episode with larger temporal difference (TD) error, which is considered as a
proxy measure of the amount which the RL agent can learn from an experience.
The actual sampling for large TD error is performed in two steps: first, an
episode is sampled from the relay buffer according to the average TD error of
its experiences, and then, for the sampled episode, the hindsight goal leading
to larger TD error is sampled with higher probability from future visited
states. The proposed method combined with Deep Deterministic Policy Gradient
(DDPG), an off-policy model-free actor-critic algorithm, accelerates learning
significantly faster than that without any prioritization on four challenging
simulated robotic manipulation tasks. The empirical results show that HGR uses
samples more efficiently than previous methods across all tasks.
