An Alternative Softmax Operator for Reinforcement Learning
  A softmax operator applied to a set of values acts somewhat like the
maximization function and somewhat like an average. In sequential decision
making, softmax is often used in settings where it is necessary to maximize
utility but also to hedge against problems that arise from putting all of one's
weight behind a single maximum utility decision. The Boltzmann softmax operator
is the most commonly used softmax operator in this setting, but we show that
this operator is prone to misbehavior. In this work, we study a differentiable
softmax operator that, among other properties, is a non-expansion ensuring a
convergent behavior in learning and planning. We introduce a variant of SARSA
algorithm that, by utilizing the new operator, computes a Boltzmann policy with
a state-dependent temperature parameter. We show that the algorithm is
convergent and that it performs favorably in practice.
