Learning in Volatile Environments with the Bayes Factor Surprise
  Surprise-based learning allows agents to rapidly adapt to non-stationary
stochastic environments characterized by sudden changes. We show that exact
Bayesian inference in a hierarchical model gives rise to a surprise-modulated
trade-off between forgetting old observations and integrating them with the new
ones. The modulation depends on a probability ratio, which we call "Bayes
Factor Surprise", that tests the prior belief against the current belief. We
demonstrate that in several existing approximate algorithms the Bayes Factor
Surprise modulates the rate of adaptation to new observations. We derive three
novel surprised-based algorithms, one in the family of particle filters, one in
the family of variational learning, and the other in the family of message
passing, that have constant scaling in observation sequence length and
particularly simple update dynamics for any distribution in the exponential
family. Empirical results show that these surprise-based algorithms estimate
parameters better than alternative approximate approaches and reach levels of
performance comparable to computationally more expensive algorithms. The Bayes
Factor Surprise is related to but different from Shannon Surprise. In two
hypothetical experiments, we make testable predictions for physiological
indicators that dissociate the Bayes Factor Surprise from Shannon Surprise. The
theoretical insight of casting various approaches as surprise-based learning,
as well as the proposed online algorithms, may be applied to the analysis of
animal and human behavior, and to reinforcement learning in non-stationary
environments.
