Selective Particle Attention: Visual Feature-Based Attention in Deep
  Reinforcement Learning
  The human brain uses selective attention to filter perceptual input so that
only the components that are useful for behaviour are processed using its
limited computational resources. We focus on one particular form of visual
attention known as feature-based attention, which is concerned with identifying
features of the visual input that are important for the current task regardless
of their spatial location. Visual feature-based attention has been proposed to
improve the efficiency of Reinforcement Learning (RL) by reducing the
dimensionality of state representations and guiding learning towards relevant
features. Despite achieving human level performance in complex perceptual-motor
tasks, Deep RL algorithms have been consistently criticised for their poor
efficiency and lack of flexibility. Visual feature-based attention therefore
represents one option for addressing these criticisms. Nevertheless, it is
still an open question how the brain is able to learn which features to attend
to during RL. To help answer this question we propose a novel algorithm, termed
Selective Particle Attention (SPA), which imbues a Deep RL agent with the
ability to perform selective feature-based attention. SPA learns which
combinations of features to attend to based on their bottom-up saliency and how
accurately they predict future reward. We evaluate SPA on a multiple choice
task and a 2D video game that both involve raw pixel input and dynamic changes
to the task structure. We show various benefits of SPA over approaches that
naively attend to either all or random subsets of features. Our results
demonstrate (1) how visual feature-based attention in Deep RL models can
improve their learning efficiency and ability to deal with sudden changes in
task structure and (2) that particle filters may represent a viable
computational account of how visual feature-based attention occurs in the
brain.
