Dense-Caption Matching and Frame-Selection Gating for Temporal
  Localization in VideoQA
  Videos convey rich information. Dynamic spatio-temporal relationships between
people/objects, and diverse multimodal events are present in a video clip.
Hence, it is important to develop automated models that can accurately extract
such information from videos. Answering questions on videos is one of the tasks
which can evaluate such AI abilities. In this paper, we propose a video
question answering model which effectively integrates multi-modal input sources
and finds the temporally relevant information to answer questions.
Specifically, we first employ dense image captions to help identify objects and
their detailed salient regions and actions, and hence give the model useful
extra information (in explicit textual format to allow easier matching) for
answering questions. Moreover, our model is also comprised of dual-level
attention (word/object and frame level), multi-head self/cross-integration for
different sources (video and dense captions), and gates which pass more
relevant information to the classifier. Finally, we also cast the frame
selection problem as a multi-label classification task and introduce two loss
functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary
Cross-Entropy (BBCE), to better supervise the model with human importance
annotations. We evaluate our model on the challenging TVQA dataset, where each
of our model components provides significant gains, and our overall model
outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We
also present several word, object, and frame level visualization studies. Our
code is publicly available at:
https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020
