Risk-Sensitive Reinforcement Learning via Policy Gradient Search
  The objective in a traditional reinforcement learning (RL) problem is to find
a policy that optimizes the expected value of a performance metric such as the
infinite-horizon cumulative discounted or long-run average cost/reward. In
practice, optimizing the expected value alone may not be satisfactory, in that
it may be desirable to incorporate the notion of risk into the optimization
problem formulation, either in the objective or as a constraint. Various risk
measures have been proposed in the literature, e.g., exponential utility,
variance, percentile performance, chance constraints, value at risk (quantile),
conditional value-at-risk, prospect theory and its later enhancement,
cumulative prospect theory. In this book, we consider risk-sensitive RL in two
settings: one where the goal is to find a policy that optimizes the usual
expected value objective while ensuring that a risk constraint is satisfied,
and the other where the risk measure is the objective. We survey some of the
recent work in this area specifically where policy gradient search is the
solution approach. In the first risk-sensitive RL setting, we cover popular
risk measures based on variance, conditional value-at-risk, and chance
constraints, and present a template for policy gradient-based risk-sensitive RL
algorithms using a Lagrangian formulation. For the setting where risk is
incorporated directly into the objective function, we consider an exponential
utility formulation, cumulative prospect theory, and coherent risk measures.
This non-exhaustive survey aims to give a flavor of the challenges involved in
solving risk-sensitive RL problems using policy gradient methods, as well as
outlining some potential future research directions.
