Collaborative Transformers for Grounded Situation Recognition
  Grounded situation recognition is the task of predicting the main activity,
entities playing certain roles within the activity, and bounding-box groundings
of the entities in the given image. To effectively deal with this challenging
task, we introduce a novel approach where the two processes for activity
classification and entity estimation are interactive and complementary. To
implement this idea, we propose Collaborative Glance-Gaze TransFormer
(CoFormer) that consists of two modules: Glance transformer for activity
classification and Gaze transformer for entity estimation. Glance transformer
predicts the main activity with the help of Gaze transformer that analyzes
entities and their relations, while Gaze transformer estimates the grounded
entities by focusing only on the entities relevant to the activity predicted by
Glance transformer. Our CoFormer achieves the state of the art in all
evaluation metrics on the SWiG dataset. Training code and model weights are
available at https://github.com/jhcho99/CoFormer.
