Reset-Free Reinforcement Learning via Multi-Task Learning: Learning
  Dexterous Manipulation Behaviors without Human Intervention
  Reinforcement Learning (RL) algorithms can in principle acquire complex
robotic skills by learning from large amounts of data in the real world,
collected via trial and error. However, most RL algorithms use a carefully
engineered setup in order to collect data, requiring human supervision and
intervention to provide episodic resets. This is particularly evident in
challenging robotics problems, such as dexterous manipulation. To make data
collection scalable, such applications require reset-free algorithms that are
able to learn autonomously, without explicit instrumentation or human
intervention. Most prior work in this area handles single-task learning.
However, we might also want robots that can perform large repertoires of
skills. At first, this would appear to only make the problem harder. However,
the key observation we make in this work is that an appropriately chosen
multi-task RL setting actually alleviates the reset-free learning challenge,
with minimal additional machinery required. In effect, solving a multi-task
problem can directly solve the reset-free problem since different combinations
of tasks can serve to perform resets for other tasks. By learning multiple
tasks together and appropriately sequencing them, we can effectively learn all
of the tasks together reset-free. This type of multi-task learning can
effectively scale reset-free learning schemes to much more complex problems, as
we demonstrate in our experiments. We propose a simple scheme for multi-task
learning that tackles the reset-free learning problem, and show its
effectiveness at learning to solve complex dexterous manipulation tasks in both
hardware and simulation without any explicit resets. This work shows the
ability to learn dexterous manipulation behaviors in the real world with RL
without any human intervention.
