Knowledge is reward: Learning optimal exploration by predictive reward
  cashing
  There is a strong link between the general concept of intelligence and the
ability to collect and use information. The theory of Bayes-adaptive
exploration offers an attractive optimality framework for training machines to
perform complex information gathering tasks. However, the computational
complexity of the resulting optimal control problem has limited the diffusion
of the theory to mainstream deep AI research. In this paper we exploit the
inherent mathematical structure of Bayes-adaptive problems in order to
dramatically simplify the problem by making the reward structure denser while
simultaneously decoupling the learning of exploitation and exploration
policies. The key to this simplification comes from the novel concept of
cross-value (i.e. the value of being in an environment while acting optimally
according to another), which we use to quantify the value of currently
available information. This results in a new denser reward structure that
"cashes in" all future rewards that can be predicted from the current
information state. In a set of experiments we show that the approach makes it
possible to learn challenging information gathering tasks without the use of
shaping and heuristic bonuses in situations where the standard RL algorithms
fail.
