Autonomous Reinforcement Learning: Formalism and Benchmarking
  Reinforcement learning (RL) provides a naturalistic framing for learning
through trial and error, which is appealing both because of its simplicity and
effectiveness and because of its resemblance to how humans and animals acquire
skills through experience. However, real-world embodied learning, such as that
performed by humans and animals, is situated in a continual, non-episodic
world, whereas common benchmark tasks in RL are episodic, with the environment
resetting between trials to provide the agent with multiple attempts. This
discrepancy presents a major challenge when attempting to take RL algorithms
developed for episodic simulated environments and run them on real-world
platforms, such as robots. In this paper, we aim to address this discrepancy by
laying out a framework for Autonomous Reinforcement Learning (ARL):
reinforcement learning where the agent not only learns through its own
experience, but also contends with lack of human supervision to reset between
trials. We introduce a simulated benchmark EARL around this framework,
containing a set of diverse and challenging simulated tasks reflective of the
hurdles introduced to learning when only a minimal reliance on extrinsic
intervention can be assumed. We show that standard approaches to episodic RL
and existing approaches struggle as interventions are minimized, underscoring
the need for developing new algorithms for reinforcement learning with a
greater focus on autonomy.
