Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
  In model-free deep reinforcement learning (RL) algorithms, using noisy value
estimates to supervise policy evaluation and optimization is detrimental to the
sample efficiency. As this noise is heteroscedastic, its effects can be
mitigated using uncertainty-based weights in the optimization process. Previous
methods rely on sampled ensembles, which do not capture all aspects of
uncertainty. We provide a systematic analysis of the sources of uncertainty in
the noisy supervision that occurs in RL, and introduce inverse-variance RL, a
Bayesian framework which combines probabilistic ensembles and Batch Inverse
Variance weighting. We propose a method whereby two complementary uncertainty
estimation methods account for both the Q-value and the environment
stochasticity to better mitigate the negative impacts of noisy supervision. Our
results show significant improvement in terms of sample efficiency on discrete
and continuous control tasks.
