Risk-Aware Transfer in Reinforcement Learning using Successor Features
  Sample efficiency and risk-awareness are central to the development of
practical reinforcement learning (RL) for complex decision-making. The former
can be addressed by transfer learning and the latter by optimizing some utility
function of the return. However, the problem of transferring skills in a
risk-aware manner is not well-understood. In this paper, we address the problem
of risk-aware policy transfer between tasks in a common domain that differ only
in their reward functions, in which risk is measured by the variance of reward
streams. Our approach begins by extending the idea of generalized policy
improvement to maximize entropic utilities, thus extending policy improvement
via dynamic programming to sets of policies and levels of risk-aversion. Next,
we extend the idea of successor features (SF), a value function representation
that decouples the environment dynamics from the rewards, to capture the
variance of returns. Our resulting risk-aware successor features (RaSF)
integrate seamlessly within the RL framework, inherit the superior task
generalization ability of SFs, and incorporate risk-awareness into the
decision-making. Experiments on a discrete navigation domain and control of a
simulated robotic arm demonstrate the ability of RaSFs to outperform
alternative methods including SFs, when taking the risk of the learned policies
into account.
