Conservative Offline Distributional Reinforcement Learning
  Many reinforcement learning (RL) problems in practice are offline, learning
purely from observational data. A key challenge is how to ensure the learned
policy is safe, which requires quantifying the risk associated with different
actions. In the online setting, distributional RL algorithms do so by learning
the distribution over returns (i.e., cumulative rewards) instead of the
expected return; beyond quantifying risk, they have also been shown to learn
better representations for planning. We propose Conservative Offline
Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both
risk-neutral and risk-averse domains. CODAC adapts distributional RL to the
offline setting by penalizing the predicted quantiles of the return for
out-of-distribution actions. We prove that CODAC learns a conservative return
distribution -- in particular, for finite MDPs, CODAC converges to an uniform
lower bound on the quantiles of the return distribution; our proof relies on a
novel analysis of the distributional Bellman operator. In our experiments, on
two challenging robot navigation tasks, CODAC successfully learns risk-averse
policies using offline data collected purely from risk-neutral agents.
Furthermore, CODAC is state-of-the-art on the D4RL MuJoCo benchmark in terms of
both expected and risk-sensitive performance.
