Learning to Walk Autonomously via Reset-Free Quality-Diversity
  Quality-Diversity (QD) algorithms can discover large and complex behavioural
repertoires consisting of both diverse and high-performing skills. However, the
generation of behavioural repertoires has mainly been limited to simulation
environments instead of real-world learning. This is because existing QD
algorithms need large numbers of evaluations as well as episodic resets, which
require manual human supervision and interventions. This paper proposes
Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous
learning for robotics in open-ended environments. We build on Dynamics-Aware
Quality-Diversity (DA-QD) and introduce a behaviour selection policy that
leverages the diversity of the imagined repertoire and environmental
information to intelligently select of behaviours that can act as automatic
resets. We demonstrate this through a task of learning to walk within defined
training zones with obstacles. Our experiments show that we can learn full
repertoires of legged locomotion controllers autonomously without manual resets
with high sample efficiency in spite of harsh safety constraints. Finally,
using an ablation of different target objectives, we show that it is important
for RF-QD to have diverse types solutions available for the behaviour selection
policy over solutions optimised with a specific objective. Videos and code
available at https://sites.google.com/view/rf-qd.
