Interpretable Spatio-temporal Attention for Video Action Recognition
  Inspired by the observation that humans are able to process videos
efficiently by only paying attention where and when it is needed, we propose an
interpretable and easy plug-in spatial-temporal attention mechanism for video
action recognition. For spatial attention, we learn a saliency mask to allow
the model to focus on the most salient parts of the feature maps. For temporal
attention, we employ a convolutional LSTM based attention mechanism to identify
the most relevant frames from an input video. Further, we propose a set of
regularizers to ensure that our attention mechanism attends to coherent regions
in space and time. Our model not only improves video action recognition
accuracy, but also localizes discriminative regions both spatially and
temporally, despite being trained in a weakly-supervised manner with only
classification labels (no bounding box labels or time frame temporal labels).
We evaluate our approach on several public video action recognition datasets
with ablation studies. Furthermore, we quantitatively and qualitatively
evaluate our model's ability to localize discriminative regions spatially and
critical frames temporally. Experimental results demonstrate the efficacy of
our approach, showing superior or comparable accuracy with the state-of-the-art
methods while increasing model interpretability.
