SENTINEL: Taming Uncertainty with Ensemble-based Distributional
  Reinforcement Learning
  In this paper, we consider risk-sensitive sequential decision-making in
model-based Reinforcement Learning (RL). Our contributions are two-fold. First,
we introduce a novel and coherent quantification of risk, namely composite
risk, which quantifies joint effect of aleatory and epistemic risk during the
learning process. Existing works considered either aleatory or epistemic risk
individually, or an additive combination of the two. We prove that the additive
formulation is a particular case of the composite risk when the epistemic risk
measure is replaced with expectation. Thus, the composite risk provides an
estimate more sensitive to both aleatory and epistemic sources of uncertainties
than the individual and additive formulations. Following that, we propose to
use a bootstrapping method, SENTINEL-K, for performing distributional RL.
SENTINEL-K uses an ensemble of $K$ learners to estimate the return
distribution. We use the Follow The Regularised Leader (FTRL) to aggregate the
return distributions of $K$ learners and to estimate the composite risk. We
experimentally verify that SENTINEL-K estimates the return distribution better,
and while used with composite risk estimate, demonstrates better risk-sensitive
performance than state-of-the-art risk-sensitive and distributional RL
algorithms.
