Calibrated Model-Based Deep Reinforcement Learning
  Estimates of predictive uncertainty are important for accurate model-based
planning and reinforcement learning. However, predictive
uncertainties---especially ones derived from modern deep learning systems---can
be inaccurate and impose a bottleneck on performance. This paper explores which
uncertainties are needed for model-based reinforcement learning and argues that
good uncertainties must be calibrated, i.e. their probabilities should match
empirical frequencies of predicted events. We describe a simple way to augment
any model-based reinforcement learning agent with a calibrated model and show
that doing so consistently improves planning, sample complexity, and
exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves
state-of-the-art performance using 50\% fewer samples than the current leading
approach. Our findings suggest that calibration can improve the performance of
model-based reinforcement learning with minimal computational and
implementation overhead.
