Efficient Two-Stage Detection of Human-Object Interactions with a Novel
  Unary-Pairwise Transformer
  Recent developments in transformer models for visual data have led to
significant improvements in recognition and detection tasks. In particular,
using learnable queries in place of region proposals has given rise to a new
class of one-stage detection models, spearheaded by the Detection Transformer
(DETR). Variations on this one-stage approach have since dominated human-object
interaction (HOI) detection. However, the success of such one-stage HOI
detectors can largely be attributed to the representation power of
transformers. We discovered that when equipped with the same transformer, their
two-stage counterparts can be more performant and memory-efficient, while
taking a fraction of the time to train. In this work, we propose the
Unary-Pairwise Transformer, a two-stage detector that exploits unary and
pairwise representations for HOIs. We observe that the unary and pairwise parts
of our transformer network specialise, with the former preferentially
increasing the scores of positive examples and the latter decreasing the scores
of negative examples. We evaluate our method on the HICO-DET and V-COCO
datasets, and significantly outperform state-of-the-art approaches. At
inference time, our model with ResNet50 approaches real-time performance on a
single GPU.
