Graph Density-Aware Losses for Novel Compositions in Scene Graph
  Generation
  Scene graph generation (SGG) aims to predict graph-structured descriptions of
input images, in the form of objects and relationships between them. This task
is becoming increasingly useful for progress at the interface of vision and
language. Here, it is important - yet challenging - to perform well on novel
(zero-shot) or rare (few-shot) compositions of objects and relationships. In
this paper, we identify two key issues that limit such generalization. Firstly,
we show that the standard loss used in this task is unintentionally a function
of scene graph density. This leads to the neglect of individual edges in large
sparse graphs during training, even though these contain diverse few-shot
examples that are important for generalization. Secondly, the frequency of
relationships can create a strong bias in this task, such that a blind model
predicting the most frequent relationship achieves good performance.
Consequently, some state-of-the-art models exploit this bias to improve
results. We show that such models can suffer the most in their ability to
generalize to rare compositions, evaluating two different models on the Visual
Genome dataset and its more recent, improved version, GQA. To address these
issues, we introduce a density-normalized edge loss, which provides more than a
two-fold improvement in certain generalization metrics. Compared to other works
in this direction, our enhancements require only a few lines of code and no
added computational cost. We also highlight the difficulty of accurately
evaluating models using existing metrics, especially on zero/few shots, and
introduce a novel weighted metric.
