Only Time Can Tell: Discovering Temporal Data for Temporal Modeling
  Understanding temporal information and how the visual world changes over time
is a fundamental ability of intelligent systems. In video understanding,
temporal information is at the core of many current challenges, including
compression, efficient inference, motion estimation or summarization. However,
in current video datasets it has been observed that action classes can often be
recognized without any temporal information from a single frame of video. As a
result, both benchmarking and training in these datasets may give an
unintentional advantage to models with strong image understanding capabilities,
as opposed to those with strong temporal understanding. In this paper we
address this problem head on by identifying action classes where temporal
information is actually necessary to recognize them and call these "temporal
classes". Selecting temporal classes using a computational method would bias
the process. Instead, we propose a methodology based on a simple and effective
human annotation experiment. We remove just the temporal information by
shuffling frames in time and measure if the action can still be recognized.
Classes that cannot be recognized when frames are not in order are included in
the temporal Dataset. We observe that this set is statistically different from
other static classes, and that performance in it correlates with a network's
ability to capture temporal information. Thus we use it as a benchmark on
current popular networks, which reveals a series of interesting facts. We also
explore the effect of training on the temporal dataset, and observe that this
leads to better generalization in unseen classes, demonstrating the need for
more temporal data. We hope that the proposed dataset of temporal categories
will help guide future research in temporal modeling for better video
understanding.
