SGTR: End-to-end Scene Graph Generation with Transformer
  Scene Graph Generation (SGG) remains a challenging visual understanding task
due to its compositional property. Most previous works adopt a bottom-up
two-stage or a point-based one-stage approach, which often suffers from high
time complexity or sub-optimal designs. In this work, we propose a novel SGG
method to address the aforementioned issues, formulating the task as a
bipartite graph construction problem. To solve the problem, we develop a
transformer-based end-to-end framework that first generates the entity and
predicate proposal set, followed by inferring directed edges to form the
relation triplets. In particular, we develop a new entity-aware predicate
representation based on a structural predicate generator that leverages the
compositional property of relationships. Moreover, we design a graph assembling
module to infer the connectivity of the bipartite scene graph based on our
entity-aware structure, enabling us to generate the scene graph in an
end-to-end manner. Extensive experimental results show that our design is able
to achieve the state-of-the-art or comparable performance on two challenging
benchmarks, surpassing most of the existing approaches and enjoying higher
efficiency in inference. We hope our model can serve as a strong baseline for
the Transformer-based scene graph generation. Code is available:
https://github.com/Scarecrow0/SGTR
