Understanding the Origin of Information-Seeking Exploration in
  Probabilistic Objectives for Control
  The exploration-exploitation trade-off is central to the description of
adaptive behaviour in fields ranging from machine learning, to biology, to
economics. While many approaches have been taken, one approach to solving this
trade-off has been to equip or propose that agents possess an intrinsic
'exploratory drive' which is often implemented in terms of maximizing the
agents information gain about the world -- an approach which has been widely
studied in machine learning and cognitive science. In this paper we
mathematically investigate the nature and meaning of such approaches and
demonstrate that this combination of utility maximizing and information-seeking
behaviour arises from the minimization of an entirely difference class of
objectives we call divergence objectives. We propose a dichotomy in the
objective functions underlying adaptive behaviour between \emph{evidence}
objectives, which correspond to well-known reward or utility maximizing
objectives in the literature, and \emph{divergence} objectives which instead
seek to minimize the divergence between the agent's expected and desired
futures, and argue that this new class of divergence objectives could form the
mathematical foundation for a much richer understanding of the exploratory
components of adaptive and intelligent action, beyond simply greedy utility
maximization.
