Risk-Averse Offline Reinforcement Learning
  Training Reinforcement Learning (RL) agents in high-stakes applications might
be too prohibitive due to the risk associated to exploration. Thus, the agent
can only use data previously collected by safe policies. While previous work
considers optimizing the average performance using offline data, we focus on
optimizing a risk-averse criteria, namely the CVaR. In particular, we present
the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that
is able to learn risk-averse policies in a fully offline setting. We show that
O-RAAC learns policies with higher CVaR than risk-neutral approaches in
different robot control tasks. Furthermore, considering risk-averse criteria
guarantees distributional robustness of the average performance with respect to
particular distribution shifts. We demonstrate empirically that in the presence
of natural distribution-shifts, O-RAAC learns policies with good average
performance.
