ACReL: Adversarial Conditional value-at-risk Reinforcement Learning
  In the classical Reinforcement Learning (RL) setting, one aims to find a
policy that maximizes its expected return. This objective may be inappropriate
in safety-critical domains such as healthcare or autonomous driving, where
intrinsic uncertainties due to stochastic policies and environment variability
may lead to catastrophic failures. This can be addressed by using the
Conditional-Value-at-Risk (CVaR) objective to instill risk-aversion in learned
policies. In this paper, we propose Adversarial Cvar Reinforcement Learning
(ACReL), a novel adversarial meta-algorithm to optimize the CVaR objective in
RL. ACReL is based on a max-min between a policy player and a learned adversary
that perturbs the policy player's state transitions given a finite budget. We
prove that, the closer the players are to the game's equilibrium point, the
closer the learned policy is to the CVaR-optimal one with a risk tolerance
explicitly related to the adversary's budget. We provide a gradient-based
training procedure to solve the proposed game by formulating it as a
Stackelberg game, enabling the use of deep RL architectures and training
algorithms. Empirical experiments show that ACReL matches a CVaR RL
state-of-the-art baseline for retrieving CVaR optimal policies, while also
benefiting from theoretical guarantees.
