Classification by Attention: Scene Graph Classification with Prior
  Knowledge
  A major challenge in scene graph classification is that the appearance of
objects and relations can be significantly different from one image to another.
Previous works have addressed this by relational reasoning over all objects in
an image or incorporating prior knowledge into classification. Unlike previous
works, we do not consider separate models for perception and prior knowledge.
Instead, we take a multi-task learning approach, where we implement the
classification as an attention layer. This allows for the prior knowledge to
emerge and propagate within the perception model. By enforcing the model also
to represent the prior, we achieve a strong inductive bias. We show that our
model can accurately generate commonsense knowledge and that the iterative
injection of this knowledge to scene representations leads to significantly
higher classification performance. Additionally, our model can be fine-tuned on
external knowledge given as triples. When combined with self-supervised
learning and with 1% of annotated images only, this gives more than 3%
improvement in object classification, 26% in scene graph classification, and
36% in predicate prediction accuracy.
