Toward Improving the Evaluation of Visual Attention Models: a
  Crowdsourcing Approach
  Human visual attention is a complex phenomenon. A computational modeling of
this phenomenon must take into account where people look in order to evaluate
which are the salient locations (spatial distribution of the fixations), when
they look in those locations to understand the temporal development of the
exploration (temporal order of the fixations), and how they move from one
location to another with respect to the dynamics of the scene and the mechanics
of the eyes (dynamics). State-of-the-art models focus on learning saliency maps
from human data, a process that only takes into account the spatial component
of the phenomenon and ignore its temporal and dynamical counterparts. In this
work we focus on the evaluation methodology of models of human visual
attention. We underline the limits of the current metrics for saliency
prediction and scanpath similarity, and we introduce a statistical measure for
the evaluation of the dynamics of the simulated eye movements. While deep
learning models achieve astonishing performance in saliency prediction, our
analysis shows their limitations in capturing the dynamics of the process. We
find that unsupervised gravitational models, despite of their simplicity,
outperform all competitors. Finally, exploiting a crowd-sourcing platform, we
present a study aimed at evaluating how strongly the scanpaths generated with
the unsupervised gravitational models appear plausible to naive and expert
human observers.
