CEM-RL: Combining evolutionary and gradient-based methods for policy
  search
  Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are
two popular approaches to policy search. The former is widely applicable and
rather stable, but suffers from low sample efficiency. By contrast, the latter
is more sample efficient, but the most sample efficient variants are also
rather unstable and highly sensitive to hyper-parameter setting. So far, these
families of methods have mostly been compared as competing tools. However, an
emerging approach consists in combining them so as to get the best of both
worlds. Two previously existing combinations use either an ad hoc evolutionary
algorithm or a goal exploration process together with the Deep Deterministic
Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL
algorithm. In this paper, we propose a different combination scheme using the
simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy
gradient (td3), another off-policy deep RL algorithm which improves over ddpg.
We evaluate the resulting method, cem-rl, on a set of benchmarks classically
used in deep RL. We show that cem-rl benefits from several advantages over its
competitors and offers a satisfactory trade-off between performance and sample
efficiency.
