Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward
  Uncertainty
  We introduce a novel framework to account for sensitivity to rewards
uncertainty in sequential decision-making problems. While risk-sensitive
formulations for Markov decision processes studied so far focus on the
distribution of the cumulative reward as a whole, we aim at learning policies
sensitive to the uncertain/stochastic nature of the rewards, which has the
advantage of being conceptually more meaningful in some cases. To this end, we
present a new decomposition of the randomness contained in the cumulative
reward based on the Doob decomposition of a stochastic process, and introduce a
new conceptual tool - the \textit{chaotic variation} - which can rigorously be
interpreted as the risk measure of the martingale component associated to the
cumulative reward process. We innovate on the reinforcement learning side by
incorporating this new risk-sensitive approach into model-free algorithms, both
policy gradient and value function based, and illustrate its relevance on grid
world and portfolio optimization problems.
