Learning to Explore with Meta-Policy Gradient
  The performance of off-policy learning, including deep Q-learning and deep
deterministic policy gradient (DDPG), critically depends on the choice of the
exploration policy. Existing exploration methods are mostly based on adding
noise to the on-going actor policy and can only explore \emph{local} regions
close to what the actor policy dictates. In this work, we develop a simple
meta-policy gradient algorithm that allows us to adaptively learn the
exploration policy in DDPG. Our algorithm allows us to train flexible
exploration behaviors that are independent of the actor policy, yielding a
\emph{global exploration} that significantly speeds up the learning process.
With an extensive study, we show that our method significantly improves the
sample-efficiency of DDPG on a variety of reinforcement learning tasks.
