AKF-SR: Adaptive Kalman Filtering-based Successor Representation
  Recent studies in neuroscience suggest that Successor Representation
(SR)-based models provide adaptation to changes in the goal locations or reward
function faster than model-free algorithms, together with lower computational
cost compared to that of model-based algorithms. However, it is not known how
such representation might help animals to manage uncertainty in their
decision-making. Existing methods for SR learning do not capture uncertainty
about the estimated SR. In order to address this issue, the paper presents a
Kalman filter-based SR framework, referred to as Adaptive Kalman
Filtering-based Successor Representation (AKF-SR). First, Kalman temporal
difference approach, which is a combination of the Kalman filter and the
temporal difference method, is used within the AKF-SR framework to cast the SR
learning procedure into a filtering problem to benefit from the uncertainty
estimation of the SR, and also decreases in memory requirement and sensitivity
to model's parameters in comparison to deep neural network-based algorithms. An
adaptive Kalman filtering approach is then applied within the proposed AKF-SR
framework in order to tune the measurement noise covariance and measurement
mapping function of Kalman filter as the most important parameters affecting
the filter's performance. Moreover, an active learning method that exploits the
estimated uncertainty of the SR to form the behaviour policy leading to more
visits to less certain values is proposed to improve the overall performance of
an agent in terms of received rewards while interacting with its environment.
