Statistics and Samples in Distributional Reinforcement Learning
  We present a unifying framework for designing and analysing distributional
reinforcement learning (DRL) algorithms in terms of recursively estimating
statistics of the return distribution. Our key insight is that DRL algorithms
can be decomposed as the combination of some statistical estimator and a method
for imputing a return distribution consistent with that set of statistics. With
this new understanding, we are able to provide improved analyses of existing
DRL algorithms as well as construct a new algorithm (EDRL) based upon
estimation of the expectiles of the return distribution. We compare EDRL with
existing methods on a variety of MDPs to illustrate concrete aspects of our
analysis, and develop a deep RL variant of the algorithm, ER-DQN, which we
evaluate on the Atari-57 suite of games.
