If MaxEnt RL is the Answer, What is the Question?
  Experimentally, it has been observed that humans and animals often make
decisions that do not maximize their expected utility, but rather choose
outcomes randomly, with probability proportional to expected utility.
Probability matching, as this strategy is called, is equivalent to maximum
entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not
optimize expected utility. In this paper, we formally show that MaxEnt RL does
optimally solve certain classes of control problems with variability in the
reward function. In particular, we show (1) that MaxEnt RL can be used to solve
a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player
game where an adversary chooses the reward function. These results suggest a
deeper connection between MaxEnt RL, robust control, and POMDPs, and provide
insight for the types of problems for which we might expect MaxEnt RL to
produce effective solutions. Specifically, our results suggest that domains
with uncertainty in the task goal may be especially well-suited for MaxEnt RL
methods.
