SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction
  We propose advances that address two key challenges in future trajectory
prediction: (i) multimodality in both training data and predictions and (ii)
constant time inference regardless of number of agents. Existing trajectory
predictions are fundamentally limited by lack of diversity in training data,
which is difficult to acquire with sufficient coverage of possible modes. Our
first contribution is an automatic method to simulate diverse trajectories in
the top-view. It uses pre-existing datasets and maps as initialization, mines
existing trajectories to represent realistic driving behaviors and uses a
multi-agent vehicle dynamics simulator to generate diverse new trajectories
that cover various modes and are consistent with scene layout constraints. Our
second contribution is a novel method that generates diverse predictions while
accounting for scene semantics and multi-agent interactions, with constant-time
inference independent of the number of agents. We propose a convLSTM with novel
state pooling operations and losses to predict scene-consistent states of
multiple agents in a single forward pass, along with a CVAE for diversity. We
validate our proposed multi-agent trajectory prediction approach by training
and testing on the proposed simulated dataset and existing real datasets of
traffic scenes. In both cases, our approach outperforms SOTA methods by a large
margin, highlighting the benefits of both our diverse dataset simulation and
constant-time diverse trajectory prediction methods.
