An empirical evaluation of active inference in multi-armed bandits
  A key feature of sequential decision making under uncertainty is a need to
balance between exploiting--choosing the best action according to the current
knowledge, and exploring--obtaining information about values of other actions.
The multi-armed bandit problem, a classical task that captures this trade-off,
served as a vehicle in machine learning for developing bandit algorithms that
proved to be useful in numerous industrial applications. The active inference
framework, an approach to sequential decision making recently developed in
neuroscience for understanding human and animal behaviour, is distinguished by
its sophisticated strategy for resolving the exploration-exploitation
trade-off. This makes active inference an exciting alternative to already
established bandit algorithms. Here we derive an efficient and scalable
approximate active inference algorithm and compare it to two state-of-the-art
bandit algorithms: Bayesian upper confidence bound and optimistic Thompson
sampling. This comparison is done on two types of bandit problems: a stationary
and a dynamic switching bandit. Our empirical evaluation shows that the active
inference algorithm does not produce efficient long-term behaviour in
stationary bandits. However, in the more challenging switching bandit problem
active inference performs substantially better than the two state-of-the-art
bandit algorithms. The results open exciting venues for further research in
theoretical and applied machine learning, as well as lend additional
credibility to active inference as a general framework for studying human and
animal behaviour.
