The Costs and Benefits of Goal-Directed Attention in Deep Convolutional
  Neural Networks
  People deploy top-down, goal-directed attention to accomplish tasks, such as
finding lost keys. By tuning the visual system to relevant information sources,
object recognition can become more efficient (a benefit) and more biased toward
the target (a potential cost). Motivated by selective attention in
categorisation models, we developed a goal-directed attention mechanism that
can process naturalistic (photographic) stimuli. Our attention mechanism can be
incorporated into any existing deep convolutional neural network (DCNNs). The
processing stages in DCNNs have been related to ventral visual stream. In that
light, our attentional mechanism incorporates top-down influences from
prefrontal cortex (PFC) to support goal-directed behaviour. Akin to how
attention weights in categorisation models warp representational spaces, we
introduce a layer of attention weights to the mid-level of a DCNN that amplify
or attenuate activity to further a goal. We evaluated the attentional mechanism
using photographic stimuli, varying the attentional target. We found that
increasing goal-directed attention has benefits (increasing hit rates) and
costs (increasing false alarm rates). At a moderate level, attention improves
sensitivity (i.e., increases $d^\prime$) at only a moderate increase in bias
for tasks involving standard images, blended images, and natural adversarial
images chosen to fool DCNNs. These results suggest that goal-directed attention
can reconfigure general-purpose DCNNs to better suit the current task goal,
much like PFC modulates activity along the ventral stream. In addition to being
more parsimonious and brain consistent, the mid-level attention approach
performed better than a standard machine learning approach for transfer
learning, namely retraining the final network layer to accommodate the new
task.
