Bayesian Nonparametrics for Offline Skill Discovery
  Skills or low-level policies in reinforcement learning are temporally
extended actions that can speed up learning and enable complex behaviours.
Recent work in offline reinforcement learning and imitation learning has
proposed several techniques for skill discovery from a set of expert
trajectories. While these methods are promising, the number K of skills to
discover is always a fixed hyperparameter, which requires either prior
knowledge about the environment or an additional parameter search to tune it.
We first propose a method for offline learning of options (a particular skill
framework) exploiting advances in variational inference and continuous
relaxations. We then highlight an unexplored connection between Bayesian
nonparametrics and offline skill discovery, and show how to obtain a
nonparametric version of our model. This version is tractable thanks to a
carefully structured approximate posterior with a dynamically-changing number
of options, removing the need to specify K. We also show how our nonparametric
extension can be applied in other skill frameworks, and empirically demonstrate
that our method can outperform state-of-the-art offline skill learning
algorithms across a variety of environments. Our code is available at
https://github.com/layer6ai-labs/BNPO .
