Distributional Reinforcement Learning with Unconstrained Monotonic
  Neural Networks
  The distributional reinforcement learning (RL) approach advocates for
representing the complete probability distribution of the random return instead
of only modelling its expectation. A distributional RL algorithm may be
characterised by two main components, namely the representation of the
distribution together with its parameterisation and the probability metric
defining the loss. The present research work considers the unconstrained
monotonic neural network (UMNN) architecture, a universal approximator of
continuous monotonic functions which is particularly well suited for modelling
different representations of a distribution (PDF, CDF, QF). This property
enables the efficient decoupling of the effect of the function approximator
class from that of the probability metric. The research paper firstly
introduces a methodology for learning different representations of the random
return distribution. Secondly, a novel distributional RL algorithm named
unconstrained monotonic deep Q-network (UMDQN) is presented. Lastly, in light
of this new algorithm, an empirical comparison is performed between three
probability quasimetrics, namely the Kullback-Leibler divergence, Cramer
distance, and Wasserstein distance. The results highlight the main strengths
and weaknesses associated with each probability metric together with an
important limitation of the Wasserstein distance. This research concludes by
calling for a reconsideration of all probability metrics in distributional RL,
contrasting with the clear dominance of the Wasserstein distance in recent
publications.
