Deep Model-Based Reinforcement Learning via Estimated Uncertainty and
  Conservative Policy Optimization
  Model-based reinforcement learning algorithms tend to achieve higher sample
efficiency than model-free methods. However, due to the inevitable errors of
learned models, model-based methods struggle to achieve the same asymptotic
performance as model-free methods.
  In this paper, We propose a Policy Optimization method with Model-Based
Uncertainty (POMBU)---a novel model-based approach---that can effectively
improve the asymptotic performance using the uncertainty in Q-values. We derive
an upper bound of the uncertainty, based on which we can approximate the
uncertainty accurately and efficiently for model-based methods. We further
propose an uncertainty-aware policy optimization algorithm that optimizes the
policy conservatively to encourage performance improvement with high
probability. This can significantly alleviate the overfitting of policy to
inaccurate models.
  Experiments show POMBU can outperform existing state-of-the-art policy
optimization algorithms in terms of sample efficiency and asymptotic
performance. Moreover, the experiments demonstrate the excellent robustness of
POMBU compared to previous model-based approaches.
