Model-based Policy Optimization with Unsupervised Model Adaptation
  Model-based reinforcement learning methods learn a dynamics model with real
data sampled from the environment and leverage it to generate simulated data to
derive an agent. However, due to the potential distribution mismatch between
simulated data and real data, this could lead to degraded performance. Despite
much effort being devoted to reducing this distribution mismatch, existing
methods fail to solve it explicitly. In this paper, we investigate how to
bridge the gap between real and simulated data due to inaccurate model
estimation for better policy optimization. To begin with, we first derive a
lower bound of the expected return, which naturally inspires a bound
maximization algorithm by aligning the simulated and real data distributions.
To this end, we propose a novel model-based reinforcement learning framework
AMPO, which introduces unsupervised model adaptation to minimize the integral
probability metric (IPM) between feature distributions from real and simulated
data. Instantiating our framework with Wasserstein-1 distance gives a practical
model-based approach. Empirically, our approach achieves state-of-the-art
performance in terms of sample efficiency on a range of continuous control
benchmark tasks.
