TubeDETR: Spatio-Temporal Video Grounding with Transformers
  We consider the problem of localizing a spatio-temporal tube in a video
corresponding to a given text query. This is a challenging task that requires
the joint and efficient modeling of temporal, spatial and multi-modal
interactions. To address this task, we propose TubeDETR, a transformer-based
architecture inspired by the recent success of such models for text-conditioned
object detection. Our model notably includes: (i) an efficient video and text
encoder that models spatial multi-modal interactions over sparsely sampled
frames and (ii) a space-time decoder that jointly performs spatio-temporal
localization. We demonstrate the advantage of our proposed components through
an extensive ablation study. We also evaluate our full approach on the
spatio-temporal video grounding task and demonstrate improvements over the
state of the art on the challenging VidSTG and HC-STVG benchmarks. Code and
trained models are publicly available at
https://antoyang.github.io/tubedetr.html.
