Maximizing Information Gain in Partially Observable Environments via
  Prediction Reward
  Information gathering in a partially observable environment can be formulated
as a reinforcement learning (RL), problem where the reward depends on the
agent's uncertainty. For example, the reward can be the negative entropy of the
agent's belief over an unknown (or hidden) variable. Typically, the rewards of
an RL agent are defined as a function of the state-action pairs and not as a
function of the belief of the agent; this hinders the direct application of
deep RL methods for such tasks. This paper tackles the challenge of using
belief-based rewards for a deep RL agent, by offering a simple insight that
maximizing any convex function of the belief of the agent can be approximated
by instead maximizing a prediction reward: a reward based on prediction
accuracy. In particular, we derive the exact error between negative entropy and
the expected prediction reward. This insight provides theoretical motivation
for several fields using prediction rewards---namely visual attention, question
answering systems, and intrinsic motivation---and highlights their connection
to the usually distinct fields of active perception, active sensing, and sensor
placement. Based on this insight we present deep anticipatory networks (DANs),
which enables an agent to take actions to reduce its uncertainty without
performing explicit belief inference. We present two applications of DANs:
building a sensor selection system for tracking people in a shopping mall and
learning discrete models of attention on fashion MNIST and MNIST digit
classification.
