Exploration with Multi-Sample Target Values for Distributional
  Reinforcement Learning
  Distributional reinforcement learning (RL) aims to learn a value-network that
predicts the full distribution of the returns for a given state, often modeled
via a quantile-based critic. This approach has been successfully integrated
into common RL methods for continuous control, giving rise to algorithms such
as Distributional Soft Actor-Critic (DSAC). In this paper, we introduce
multi-sample target values (MTV) for distributional RL, as a principled
replacement for single-sample target value estimation, as commonly employed in
current practice. The improved distributional estimates further lend themselves
to UCB-based exploration. These two ideas are combined to yield our
distributional RL algorithm, E2DC (Extra Exploration with Distributional
Critics). We evaluate our approach on a range of continuous control tasks and
demonstrate state-of-the-art model-free performance on difficult tasks such as
Humanoid control. We provide further insight into the method via visualization
and analysis of the learned distributions and their evolution during training.
