Improving Model-Based Reinforcement Learning with Internal State
  Representations through Self-Supervision
  Using a model of the environment, reinforcement learning agents can plan
their future moves and achieve superhuman performance in board games like
Chess, Shogi, and Go, while remaining relatively sample-efficient. As
demonstrated by the MuZero Algorithm, the environment model can even be learned
dynamically, generalizing the agent to many more tasks while at the same time
achieving state-of-the-art performance. Notably, MuZero uses internal state
representations derived from real environment states for its predictions. In
this paper, we bind the model's predicted internal state representation to the
environment state via two additional terms: a reconstruction model loss and a
simpler consistency loss, both of which work independently and unsupervised,
acting as constraints to stabilize the learning process. Our experiments show
that this new integration of reconstruction model loss and simpler consistency
loss provide a significant performance increase in OpenAI Gym environments. Our
modifications also enable self-supervised pretraining for MuZero, so the
algorithm can learn about environment dynamics before a goal is made available.
