Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization
  without Compounding Errors
  Model usage is the central challenge of model-based reinforcement learning.
Although dynamics model based on deep neural networks provide good
generalization for single step prediction, such ability is over exploited when
it is used to predict long horizon trajectories due to compounding errors. In
this work, we propose a Dyna-style model-based reinforcement learning
algorithm, which we called Maximum Entropy Model Rollouts (MEMR). To eliminate
the compounding errors, we only use our model to generate single-step rollouts.
Furthermore, we propose to generate \emph{diverse} model rollouts by
non-uniform sampling of the environment states such that the entropy of the
model rollouts is maximized. We mathematically derived the maximum entropy
sampling criteria for one data case under Gaussian prior. To accomplish this
criteria, we propose to utilize a prioritized experience replay. Our
preliminary experiments in challenging locomotion benchmarks show that our
approach achieves the same sample efficiency of the best model-based
algorithms, matches the asymptotic performance of the best model-free
algorithms, and significantly reduces the computation requirements of other
model-based methods.
