Skill-based Meta-Reinforcement Learning
  While deep reinforcement learning methods have shown impressive results in
robot learning, their sample inefficiency makes the learning of complex,
long-horizon behaviors with real robot systems infeasible. To mitigate this
issue, meta-reinforcement learning methods aim to enable fast learning on novel
tasks by learning how to learn. Yet, the application has been limited to
short-horizon tasks with dense rewards. To enable learning long-horizon
behaviors, recent works have explored leveraging prior experience in the form
of offline datasets without reward or task annotations. While these approaches
yield improved sample efficiency, millions of interactions with environments
are still required to solve complex tasks. In this work, we devise a method
that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to
solve unseen target tasks with orders of magnitude fewer environment
interactions. Our core idea is to leverage prior experience extracted from
offline datasets during meta-learning. Specifically, we propose to (1) extract
reusable skills and a skill prior from offline datasets, (2) meta-train a
high-level policy that learns to efficiently compose learned skills into
long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve
an unseen target task. Experimental results on continuous control tasks in
navigation and manipulation demonstrate that the proposed method can
efficiently solve long-horizon novel target tasks by combining the strengths of
meta-learning and the usage of offline datasets, while prior approaches in RL,
meta-RL, and multi-task RL require substantially more environment interactions
to solve the tasks.
