Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning
  Accurate value estimates are important for off-policy reinforcement learning.
Algorithms based on temporal difference learning typically are prone to an
over- or underestimation bias building up over time. In this paper, we propose
a general method called Adaptively Calibrated Critics (ACC) that uses the most
recent high variance but unbiased on-policy rollouts to alleviate the bias of
the low variance temporal difference targets. We apply ACC to Truncated
Quantile Critics, which is an algorithm for continuous control that allows
regulation of the bias with a hyperparameter tuned per environment. The
resulting algorithm adaptively adjusts the parameter during training rendering
hyperparameter search unnecessary and sets a new state of the art on the OpenAI
gym continuous control benchmark among all algorithms that do not tune
hyperparameters for each environment. Additionally, we demonstrate that ACC is
quite general by further applying it to TD3 and showing an improved performance
also in this setting.
