A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning
While reinforcement learning (RL) provides a framework for learning through
trial and error, translating RL algorithms into the real world has remained
challenging. A major hurdle to real-world application arises from the
development of algorithms in an episodic setting where the environment is reset
after every trial, in contrast with the continual and non-episodic nature of
the real-world encountered by embodied agents such as humans and robots. Prior
works have considered an alternating approach where a forward policy learns to
solve the task and the backward policy learns to reset the environment, but
what initial state distribution should the backward policy reset the agent to?
Assuming access to a few demonstrations, we propose a new method, MEDAL, that
trains the backward policy to match the state distribution in the provided
demonstrations. This keeps the agent close to the task-relevant states,
allowing for a mix of easy and difficult starting states for the forward
policy. Our experiments show that MEDAL matches or outperforms prior methods on
three sparse-reward continuous control tasks from the EARL benchmark, with 40%
gains on the hardest task, while making fewer assumptions than prior works.