Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy
  While maximizing expected return is the goal in most reinforcement learning
approaches, risk-sensitive objectives such as conditional value at risk (CVaR)
are more suitable for many high-stakes applications. However, relatively little
is known about how to explore to quickly learn policies with good CVaR. In this
paper, we present the first algorithm for sample-efficient learning of
CVaR-optimal policies in Markov decision processes based on the optimism in the
face of uncertainty principle. This method relies on a novel optimistic version
of the distributional Bellman operator that moves probability mass from the
lower to the upper tail of the return distribution. We prove asymptotic
convergence and optimism of this operator for the tabular policy evaluation
case. We further demonstrate that our algorithm finds CVaR-optimal policies
substantially faster than existing baselines in several simulated environments
with discrete and continuous state spaces.
