Active World Model Learning with Progress Curiosity
  World models are self-supervised predictive models of how the world evolves.
Humans learn world models by curiously exploring their environment, in the
process acquiring compact abstractions of high bandwidth sensory inputs, the
ability to plan across long temporal horizons, and an understanding of the
behavioral patterns of other agents. In this work, we study how to design such
a curiosity-driven Active World Model Learning (AWML) system. To do so, we
construct a curious agent building world models while visually exploring a 3D
physical environment rich with distillations of representative real-world
agents. We propose an AWML system driven by $\gamma$-Progress: a scalable and
effective learning progress-based curiosity signal. We show that
$\gamma$-Progress naturally gives rise to an exploration policy that directs
attention to complex but learnable dynamics in a balanced manner, thus
overcoming the "white noise problem". As a result, our $\gamma$-Progress-driven
controller achieves significantly higher AWML performance than baseline
controllers equipped with state-of-the-art exploration strategies such as
Random Network Distillation and Model Disagreement.
