Fully Parameterized Quantile Function for Distributional Reinforcement
  Learning
  Distributional Reinforcement Learning (RL) differs from traditional RL in
that, rather than the expectation of total returns, it estimates distributions
and has achieved state-of-the-art performance on Atari Games. The key challenge
in practical distributional RL algorithms lies in how to parameterize estimated
distributions so as to better approximate the true continuous distribution.
Existing distributional RL algorithms parameterize either the probability side
or the return value side of the distribution function, leaving the other side
uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper,
we propose fully parameterized quantile function that parameterizes both the
quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for
distributional RL. Our algorithm contains a fraction proposal network that
generates a discrete set of quantile fractions and a quantile value network
that gives corresponding quantile values. The two networks are jointly trained
to find the best approximation of the true distribution. Experiments on 55
Atari Games show that our algorithm significantly outperforms existing
distributional RL algorithms and creates a new record for the Atari Learning
Environment for non-distributed agents.
