Exploring the Training Robustness of Distributional Reinforcement
  Learning against Noisy State Observations
  In real scenarios, state observations that an agent observes may contain
measurement errors or adversarial noises, misleading the agent to take
suboptimal actions or even collapse while training. In this paper, we study the
training robustness of distributional Reinforcement Learning~(RL), a class of
state-of-the-art methods that estimate the whole distribution, as opposed to
only the expectation, of the total return. Firstly, we validate the contraction
of both expectation-based and distributional Bellman operators in the
State-Noisy Markov Decision Process~(SN-MDP), a typical tabular case that
incorporates both random and adversarial state observation noises. Beyond
SN-MDP, we then analyze the vulnerability of least squared loss in
expectation-based RL with either linear or nonlinear function approximation. By
contrast, we theoretically characterize the bounded gradient norm of
distributional RL loss based on the histogram density estimation. The resulting
stable gradients while the optimization in distributional RL accounts for its
better training robustness against state observation noises. Finally, extensive
experiments on the suite of games verified the convergence of both
expectation-based and distributional RL in the SN-MDP-like setting under
different strengths of state observation noises. More importantly, in noisy
settings beyond SN-MDP, distributional RL is less vulnerable against noisy
state observations compared with its expectation-based counterpart.
