Online reinforcement learning with sparse rewards through an active
  inference capsule
  Intelligent agents must pursue their goals in complex environments with
partial information and often limited computational capacity. Reinforcement
learning methods have achieved great success by creating agents that optimize
engineered reward functions, but which often struggle to learn in sparse-reward
environments, generally require many environmental interactions to perform
well, and are typically computationally very expensive. Active inference is a
model-based approach that directs agents to explore uncertain states while
adhering to a prior model of their goal behaviour. This paper introduces an
active inference agent which minimizes the novel free energy of the expected
future. Our model is capable of solving sparse-reward problems with a very high
sample efficiency due to its objective function, which encourages directed
exploration of uncertain states. Moreover, our model is computationally very
light and can operate in a fully online manner while achieving comparable
performance to offline RL methods. We showcase the capabilities of our model by
solving the mountain car problem, where we demonstrate its superior exploration
properties and its robustness to observation noise, which in fact improves
performance. We also introduce a novel method for approximating the prior model
from the reward function, which simplifies the expression of complex objectives
and improves performance over previous active inference approaches.
