Learning Future Object Prediction with a Spatiotemporal Detection
  Transformer
  We explore future object prediction -- a challenging problem where all
objects visible in a future video frame are to be predicted. We propose to
tackle this problem end-to-end by training a detection transformer to directly
output future objects. In order to make accurate predictions about the future,
it is necessary to capture the dynamics in the scene, both of other objects and
of the ego-camera. We extend existing detection transformers in two ways to
capture the scene dynamics. First, we experiment with three different
mechanisms that enable the model to spatiotemporally process multiple frames.
Second, we feed ego-motion information to the model via cross-attention. We
show that both of these cues substantially improve future object prediction
performance. Our final approach learns to capture the dynamics and make
predictions on par with an oracle for 100 ms prediction horizons, and
outperform baselines for longer prediction horizons.
