Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via
  Latent Model Ensembles
  Learning complex robot behaviors through interaction requires structured
exploration. Planning should target interactions with the potential to optimize
long-term performance, while only reducing uncertainty where conducive to this
objective. This paper presents Latent Optimistic Value Exploration (LOVE), a
strategy that enables deep exploration through optimism in the face of
uncertain long-term rewards. We combine latent world models with value function
estimation to predict infinite-horizon returns and recover associated
uncertainty via ensembling. The policy is then trained on an upper confidence
bound (UCB) objective to identify and select the interactions most promising to
improve long-term performance. We apply LOVE to visual robot control tasks in
continuous action spaces and demonstrate on average more than 20% improved
sample efficiency in comparison to state-of-the-art and other exploration
objectives. In sparse and hard to explore environments we achieve an average
improvement of over 30%.
