Data-Lean Evolutionary Reinforcement Learning by Multitasking with
  Importance Sampling
  Studies have shown evolution strategies (ES) to be a promising approach for
reinforcement learning (RL) with deep neural networks. However, the issue of
high sample complexity persists in applications of ES to deep RL. This paper is
the first to address the shortcoming of today's methods via a novel
neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer
information from a set of auxiliary tasks (of short episode length) to the
target (full length) RL task at hand. The auxiliary tasks, extracted from the
target, allow an agent to update and quickly evaluate policies on shorter time
horizons. The evolved skills are then transferred to guide the longer and
harder task towards an optimal policy. We demonstrate that the NuEMT algorithm
achieves data-lean evolutionary RL, reducing expensive agent-environment
interaction data requirements. Our key algorithmic contribution in this setting
is to introduce, for the first time, a multitask skills transfer mechanism
based on the statistical importance sampling technique. In addition, an
adaptive resource allocation strategy is utilized to assign computational
resources to auxiliary tasks based on their gleaned usefulness. Experiments on
a range of continuous control tasks from the OpenAI Gym confirm that our
proposed algorithm is efficient compared to recent ES baselines.
