RAPTOR: End-to-end Risk-Aware MDP Planning and Policy Learning by
  Backpropagation
  Planning provides a framework for optimizing sequential decisions in complex
environments. Recent advances in efficient planning in deterministic or
stochastic high-dimensional domains with continuous action spaces leverage
backpropagation through a model of the environment to directly optimize
actions. However, existing methods typically not take risk into account when
optimizing in stochastic domains, which can be incorporated efficiently in MDPs
by optimizing the entropic utility of returns. We bridge this gap by
introducing Risk-Aware Planning using PyTorch (RAPTOR), a novel framework for
risk-sensitive planning through end-to-end optimization of the entropic utility
objective. A key technical difficulty of our approach lies in that direct
optimization of the entropic utility by backpropagation is impossible due to
the presence of environment stochasticity. The novelty of RAPTOR lies in the
reparameterization of the state distribution, which makes it possible to apply
stochastic backpropagatation through sufficient statistics of the entropic
utility computed from forward-sampled trajectories. The direct optimization of
this empirical objective in an end-to-end manner is called the risk-averse
straight-line plan, which commits to a sequence of actions in advance and can
be sub-optimal in highly stochastic domains. We address this shortcoming by
optimizing for risk-aware Deep Reactive Policies (RaDRP) in our framework. We
evaluate and compare these two forms of RAPTOR on three highly stochastic
do-mains, including nonlinear navigation, HVAC control, and linear reservoir
control, demonstrating the ability to manage risk in complex MDPs.
