Exponential Bellman Equation and Improved Regret Bounds for
  Risk-Sensitive Reinforcement Learning
  We study risk-sensitive reinforcement learning (RL) based on the entropic
risk measure. Although existing works have established non-asymptotic regret
guarantees for this problem, they leave open an exponential gap between the
upper and lower bounds. We identify the deficiencies in existing algorithms and
their analysis that result in such a gap. To remedy these deficiencies, we
investigate a simple transformation of the risk-sensitive Bellman equations,
which we call the exponential Bellman equation. The exponential Bellman
equation inspires us to develop a novel analysis of Bellman backup procedures
in risk-sensitive RL algorithms, and further motivates the design of a novel
exploration mechanism. We show that these analytic and algorithmic innovations
together lead to improved regret upper bounds over existing ones.
