Analysis of Agent Expertise in Ms. Pac-Man using
  Value-of-Information-based Policies
  Conventional reinforcement learning methods for Markov decision processes
rely on weakly-guided, stochastic searches to drive the learning process. It
can therefore be difficult to predict what agent behaviors might emerge. In
this paper, we consider an information-theoretic cost function for performing
constrained stochastic searches that promote the formation of risk-averse to
risk-favoring behaviors. This cost function is the value of information, which
provides the optimal trade-off between the expected return of a policy and the
policy's complexity; policy complexity is measured by number of bits and
controlled by a single hyperparameter on the cost function. As the policy
complexity is reduced, the agents will increasingly eschew risky actions. This
reduces the potential for high accrued rewards. As the policy complexity
increases, the agents will take actions, regardless of the risk, that can raise
the long-term rewards. The obtainable reward depends on a single, tunable
hyperparameter that regulates the degree of policy complexity.
  We evaluate the performance of value-of-information-based policies on a
stochastic version of Ms. Pac-Man. A major component of this paper is the
demonstration that ranges of policy complexity values yield different game-play
styles and explaining why this occurs. We also show that our
reinforcement-learning search mechanism is more efficient than the others we
utilize. This result implies that the value of information theory is
appropriate for framing the exploitation-exploration trade-off in reinforcement
learning.
