Probabilistic Successor Representations with Kalman Temporal Differences
  The effectiveness of Reinforcement Learning (RL) depends on an animal's
ability to assign credit for rewards to the appropriate preceding stimuli. One
aspect of understanding the neural underpinnings of this process involves
understanding what sorts of stimulus representations support generalisation.
The Successor Representation (SR), which enforces generalisation over states
that predict similar outcomes, has become an increasingly popular model in this
space of inquiries. Another dimension of credit assignment involves
understanding how animals handle uncertainty about learned associations, using
probabilistic methods such as Kalman Temporal Differences (KTD). Combining
these approaches, we propose using KTD to estimate a distribution over the SR.
KTD-SR captures uncertainty about the estimated SR as well as covariances
between different long-term predictions. We show that because of this, KTD-SR
exhibits partial transition revaluation as humans do in this experiment without
additional replay, unlike the standard TD-SR algorithm. We conclude by
discussing future applications of the KTD-SR as a model of the interaction
between predictive and probabilistic animal reasoning.
